fix(orchestration): enforce nested worker depth instead of an accidental fence (#16668)

* fix(orchestration): enforce nested worker depth instead of an accidental fence

Orca documented that "dispatched workers cannot spawn their own sub-workers
(worker-start is coordinator-fenced)". No such check existed. What existed was a
single Run-binding check in the workerStart RPC: a worker's terminal is not bound
to a Run, so worker-start happened to fail. The rule was emergent, asserted by no
test, and written in no doc — and it leaked. A worker could run-create its own
Run, task-create, and worker-start: now bound, the check passed.

Replace it with a real, configurable depth cap.

Depth is derived from the caller's own active Dispatch rather than from Run
binding, which is what dissolves the run-create bypass: creating a Run does not
stop you being a worker. Enforcement lives in a single dispatch-row writer that
owns all three INSERTs that mint a live worker — the generic claim, the supervised
worker-start path (including every retry), and the remote attachment. Two of those
were missed by earlier drafts of this change, so `creator` and `maxDepth` are
required parameters: a new spawn path cannot compile without deciding, and a
boundary test refuses the SQL anywhere else.

Schema v30 adds depth to dispatch_contexts and remote_dispatch_attachments,
NOT NULL DEFAULT 1 and backfilled to 1 so an unstamped or pre-upgrade row fails
closed rather than reading as a root coordinator. The attachment pane indexes
widen to the five states in which a remote worker may still be running:
loss of contact is not evidence of process death, so an unverifiable worker still
counts as a nesting parent.

Also adds the caller-evidence assertion that workerStart was the only Run-scoped
verb to skip, so a declared --from cannot name another terminal's pane and inherit
its depth.

Default is 1, so behaviour is unchanged unless the new setting is raised. Two
limitations are deliberate and documented rather than papered over: this is a
guardrail and not a security boundary, since a caller whose launch evidence is
unverifiable (any ordinary restored terminal) can declare another handle; and it
is enforced at supervised dispatch creation, so a settled worker whose process is
still alive counts as a root again.

* fix(orchestration): share caller resolution and pin worker gaps

* refactor(orchestration): make the caller resolver's pane contract explicit

Overloads so requireStablePane callers get a non-null string instead of casting,
and rename the attestation opt-out to say what it means: the caller asserts it
itself. A flag called assertEvidence:false reads as "attestation optional",
which is the hole this helper exists to close.

* fix(orchestration): propagate dispatch depth to federated workers

* chore(cli): refresh bundled orchestration guide
This commit is contained in:
Brennan Benson
2026-08-26 13:22:09 -07:00
committed by GitHub
parent 256f23c7a0
commit 8a07bbd8cf
96 changed files with 1788 additions and 356 deletions
+27
View File
@@ -176,6 +176,33 @@ Dispatch rules:
- After 3 consecutive failures on one task, the dispatch context circuit-breaks and the task is marked failed.
- Use `task-list --brief --json` for coordinator sweeps; it collapses whitespace and caps each echoed spec at 160 characters (`spec_truncated` marks shortened rows). Omit `--brief` when the full spec is required, or when an older CLI rejects it as an unknown flag.
## How deep workers can nest
A dispatched worker normally cannot dispatch sub-workers. Attempting it fails with
`nested_worker_depth_exceeded` and a message telling the worker to complete the task
itself. Do that — do not try to route around it.
The limit is a number, not an on/off switch. `Settings -> Agents -> Nested worker depth`
sets how many generations are allowed:
- `1` (default): a coordinator dispatches workers; those workers do not dispatch.
- `2`: workers may dispatch one further generation.
Depth is counted from the terminal that issues the command, not from the Run. Creating a
new Run does not reset it — a worker that runs `run-create` then `worker-start` is still a
worker, and still counted. This is the part that changed: the old behaviour rejected
sub-dispatch only because a worker's terminal was not bound to a Run, so creating a Run was
enough to slip past it.
Two limits worth knowing:
- **It is a guardrail, not a security boundary.** A caller that declares another terminal's
handle while its own launch evidence is unverifiable (an ordinary restored terminal, for
example) can be counted as that terminal instead. Orca does not treat workers as hostile.
- **It applies while a Dispatch is active.** After `worker_done`, or after a coordinator
settles the task, the terminal is no longer a worker and is counted as a root again. The
process may still be alive; that is the documented boundary, not an accident.
## Preferred Supervised Worker Loop
Use `worker-start` for the normal supervised path. It composes the existing worktree, terminal, readiness, and dispatch primitives while returning exact created/reused effects. Agents still choose placement and concurrency; Orca does not schedule workers or infer conflicts.
File diff suppressed because one or more lines are too long
@@ -1,5 +1,6 @@
import type { GlobalSettings } from '../../../shared/global-settings-types'
import { normalizeDisabledTuiAgents } from '../../../shared/tui-agent-selection'
import { resolveNestedWorkerMaxDepth } from '../../../shared/nested-worker-depth'
import {
normalizeTuiAgentArgsRecord,
normalizeTuiAgentEnvRecord
@@ -73,6 +74,11 @@ export function updateSettings(
if ('agentSkillSharingEnabled' in updates) {
sanitizedUpdates.agentSkillSharingEnabled = updates.agentSkillSharingEnabled === true
}
if ('nestedWorkerMaxDepth' in updates) {
sanitizedUpdates.nestedWorkerMaxDepth = resolveNestedWorkerMaxDepth({
nestedWorkerMaxDepth: updates.nestedWorkerMaxDepth
})
}
if ('disabledTuiAgents' in updates) {
sanitizedUpdates.disabledTuiAgents = normalizeDisabledTuiAgents(updates.disabledTuiAgents)
}
@@ -7,6 +7,7 @@ import { join } from 'node:path'
import { OrcaRuntimeService } from './orca-runtime'
import { OrchestrationDb } from './orchestration/db'
import type { DispatchContextRow } from './orchestration/types'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
const TAB_ID = '11111111-1111-4111-8111-111111111111'
const LEAF_ID = '22222222-2222-4222-8222-222222222222'
@@ -71,7 +72,10 @@ function dispatchOnHandle(
coordinatorPaneKey: '99999999-9999-4999-8999-999999999999:88888888-8888-4888-8888-888888888888'
})
const task = db.createTask({ spec, runId: run.id })
return { ...db.createDispatchContext(task.id, HANDLE, PANE_KEY), runId: run.id }
return {
...createRootDispatch(db, task.id, HANDLE, PANE_KEY),
runId: run.id
}
}
/** Where a lightweight Run's coordinator actually reads its mail (STA-4604). */
+16 -6
View File
@@ -144,6 +144,7 @@ import {
} from './terminal-view-attribute-store'
import { clearConfiguredWorktreeSharedDirectoriesCacheForTests } from '../git/worktree-shared-directories'
import { setWorktreeWatcherRemoval } from '../ipc/worktree-watcher-removal'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
const ORIGINAL_PLATFORM = process.platform
const ORIGINAL_PLATFORM_DESCRIPTOR = Object.getOwnPropertyDescriptor(process, 'platform')
@@ -22004,6 +22005,8 @@ describe('OrcaRuntimeService', () => {
try {
const task = db.createTask({ spec: 'continue after missing worker recovery' })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: { topology: 'current', agent: 'codex' }
})
@@ -22109,6 +22112,8 @@ describe('OrcaRuntimeService', () => {
try {
const task = db.createTask({ spec: 'retry missing worker recovery' })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: { topology: 'current', agent: 'codex' }
})
@@ -43844,11 +43849,12 @@ describe('OrcaRuntimeService', () => {
['worker-folder', runB.id]
].map(([name, runId]) => {
const task = db.createTask({ spec: name, runId })
return [name, db.createDispatchContext(task.id, handles[name], paneKey(name))]
return [name, createRootDispatch(db, task.id, handles[name], paneKey(name))]
})
)
const legacyTask = db.createTask({ spec: 'legacy worker' })
const legacyDispatch = db.createDispatchContext(
const legacyDispatch = createRootDispatch(
db,
legacyTask.id,
handles['legacy-worker'],
paneKey('legacy-worker')
@@ -43976,7 +43982,8 @@ describe('OrcaRuntimeService', () => {
expect(creatorAuthority?.processIncarnation).toBeTruthy()
expect(coordinatorAuthority?.processIncarnation).toBeTruthy()
const creatorTask = db.createTask({ spec: 'create nested work', runId: runA.id })
db.createDispatchContext(
createRootDispatch(
db,
creatorTask.id,
handles.creator,
paneKey('creator'),
@@ -43991,7 +43998,8 @@ describe('OrcaRuntimeService', () => {
createdByProcessIncarnation: creatorAuthority?.processIncarnation ?? undefined,
createdByRunGeneration: runA.consumer_generation
})
const workerDispatch = db.createDispatchContext(
const workerDispatch = createRootDispatch(
db,
workerTask.id,
handles.worker,
paneKey('worker')
@@ -44004,7 +44012,8 @@ describe('OrcaRuntimeService', () => {
createdByProcessIncarnation: coordinatorAuthority?.processIncarnation ?? undefined,
createdByRunGeneration: runA.consumer_generation
})
const coordinatorCreatedDispatch = db.createDispatchContext(
const coordinatorCreatedDispatch = createRootDispatch(
db,
coordinatorCreatedTask.id,
handles['coordinator-created-worker'],
paneKey('coordinator-created-worker')
@@ -44092,7 +44101,8 @@ describe('OrcaRuntimeService', () => {
coordinatorPaneKey: makePaneKey(terminals[99].tabId, terminals[99].leafId)
})
const task = db.createTask({ spec: 'one dispatched terminal', runId: run.id })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
handles[0],
makePaneKey(terminals[0].tabId, terminals[0].leafId)
+7
View File
@@ -18,6 +18,7 @@ import {
assertAgentSkillSharingAllowed,
isAgentSkillSharingEnabled
} from '../../shared/agent-skill-sharing-gate'
import { resolveNestedWorkerMaxDepth } from '../../shared/nested-worker-depth'
import { sortDirEntries } from '../../shared/file-name-sort'
import { isServerDriveListRequest, listWindowsDrives } from './windows-drive-listing'
import { extractLastOsc7Uri, extractOscScanTail } from '../daemon/osc7-uri-extraction'
@@ -1407,6 +1408,7 @@ type RuntimeStore = {
prBotAuthorOverrides?: GlobalSettings['prBotAuthorOverrides']
artifactSharingEnabled?: GlobalSettings['artifactSharingEnabled']
agentSkillSharingEnabled?: GlobalSettings['agentSkillSharingEnabled']
nestedWorkerMaxDepth?: GlobalSettings['nestedWorkerMaxDepth']
terminalQuickCommands?: GlobalSettings['terminalQuickCommands']
gitlabProjects?: GlobalSettings['gitlabProjects']
mobileAutoRestoreFitMs?: number | null
@@ -5283,6 +5285,11 @@ export class OrcaRuntimeService {
assertAgentSkillSharingAllowed(() => isAgentSkillSharingEnabled(this.store?.getSettings()))
}
/** Renderer-owned; read here because dispatch enforcement lives in main. */
getNestedWorkerMaxDepth(): number {
return resolveNestedWorkerMaxDepth(this.store?.getSettings())
}
async publishDiscoveredSkillsFromAgent(
request: AgentSkillShareRequest,
discoveredSkills: readonly DiscoveredSkill[],
@@ -16,6 +16,7 @@ import {
temporaryDirectories,
TERMINAL_HANDLE
} from './orchestration-mailbox-notification-test-harness'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
vi.mock('electron', () => ({
app: { getPath: vi.fn(() => tmpdir()), isPackaged: false },
@@ -43,7 +44,7 @@ describe('orchestration detached mailbox routing', () => {
'33333333-3333-4333-8333-333333333333:44444444-4444-4444-8444-444444444444'
})
const task = db.createTask({ spec: 'Worker task', runId: run.id })
const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
await driveToLiveIdle(harness.runtime)
const message = db.insertMessage({
from: 'term_coordinator',
@@ -118,7 +119,8 @@ describe('orchestration detached mailbox routing', () => {
'55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666'
})
const task = db.createTask({ spec: 'Worker task', runId: workerRun.id })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
'term_mailbox_before_remint',
`99999999-9999-4999-8999-999999999999:${LEAF_ID}`
@@ -356,7 +358,7 @@ describe('orchestration detached mailbox routing', () => {
'55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666'
})
const task = db.createTask({ spec: 'Reminted worker', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_before_remint', PANE_KEY)
const dispatch = createRootDispatch(db, task.id, 'term_before_remint', PANE_KEY)
const waiting = harness.runtime.waitForMessage(`dispatch:${dispatch.id}`, {
typeFilter: ['dispatch'],
timeoutMs: 5_000
@@ -430,7 +432,7 @@ describe('orchestration detached mailbox routing', () => {
coordinatorPaneKey: PANE_KEY
})
const task = db.createTask({ spec: 'Same-handle worker', runId: run.id })
db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
const message = db.insertMessage({
from: 'term_sender',
to: TERMINAL_HANDLE,
@@ -459,7 +461,7 @@ describe('orchestration detached mailbox routing', () => {
coordinatorPaneKey: PANE_KEY
})
const task = db.createTask({ spec: 'Same-handle worker', runId: run.id })
const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
const message = db.insertMessage({
from: 'term_sender',
to: TERMINAL_HANDLE,
@@ -501,7 +503,7 @@ describe('orchestration detached mailbox routing', () => {
.prepare('UPDATE messages SET to_handle = ? WHERE id = ?')
.run(TERMINAL_HANDLE, directMessage.id)
const task = db.createTask({ spec: 'Dispatch migration', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_dispatch', PANE_KEY)
const dispatch = createRootDispatch(db, task.id, 'term_dispatch', PANE_KEY)
db.insertMessage({
from: 'term_sender',
to: `dispatch:${dispatch.id}`,
@@ -28,6 +28,7 @@ import {
} from './orchestration-mailbox-notification-test-harness'
import { RpcDispatcher } from './rpc/dispatcher'
import { ORCHESTRATION_METHODS } from './rpc/methods/orchestration'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
vi.mock('electron', () => ({
app: { getPath: vi.fn(() => tmpdir()), isPackaged: false },
@@ -750,7 +751,7 @@ describe('orchestration notification mailbox consistency', () => {
'55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666'
})
const task = db.createTask({ spec: 'Worker task', runId: run.id })
const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
for (let index = 0; index < 50; index += 1) {
insertDirectRunMessage(db, run.id, `Worker status ${index}`)
}
@@ -19,6 +19,7 @@ import {
temporaryDirectories,
TERMINAL_HANDLE
} from './orchestration-mailbox-notification-test-harness'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
vi.mock('electron', () => ({
app: { getPath: vi.fn(() => tmpdir()), isPackaged: false },
@@ -69,7 +70,7 @@ describe('orchestration mailbox routing races', () => {
coordinatorPaneKey: SECOND_PANE_KEY
})
const task = db.createTask({ spec: 'Worker task', runId: run.id })
const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
for (let index = 0; index < 151; index += 1) {
insertDirectRunMessage(db, run.id, `Before completion ${index}`)
}
@@ -152,7 +153,7 @@ describe('orchestration mailbox routing races', () => {
coordinatorPaneKey: SECOND_PANE_KEY
})
const task = db.createTask({ spec: 'Waiting worker', runId: run.id })
const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
const status = insertDirectRunMessage(db, run.id, 'Filtered-out status')
const controller = new AbortController()
const waiting = dispatchMailboxCheck(harness.runtime, {
@@ -201,7 +202,7 @@ describe('orchestration mailbox routing races', () => {
coordinatorPaneKey: SECOND_PANE_KEY
})
const task = db.createTask({ spec: 'Cancelled worker check', runId: run.id })
const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY)
const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY)
for (let index = 0; index < 151; index += 1) {
insertDirectRunMessage(db, run.id, `Before cancelled migration ${index}`)
}
@@ -296,7 +297,7 @@ describe('orchestration mailbox routing races', () => {
'55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666'
})
const task = db.createTask({ spec: 'Worker task', runId: run.id })
db.createDispatchContext(task.id, 'term_previous_worker', PANE_KEY)
createRootDispatch(db, task.id, 'term_previous_worker', PANE_KEY)
const current = insertDirectRunMessage(db, run.id, 'Current worker handle')
const previous = db.insertMessage({
from: 'term_coordinator',
@@ -421,9 +422,9 @@ describe('orchestration mailbox routing races', () => {
'55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666'
})
const task = db.createTask({ spec: 'Valid worker', runId: run.id })
const valid = db.createDispatchContext(task.id, 'term_old', PANE_KEY)
const valid = createRootDispatch(db, task.id, 'term_old', PANE_KEY)
const collisionTask = db.createTask({ spec: 'Malformed collision', runId: run.id })
db.createDispatchContext(collisionTask.id, 'term_collision', `:${LEAF_ID}`)
createRootDispatch(db, collisionTask.id, 'term_collision', `:${LEAF_ID}`)
expect(db.getActiveDispatchForIdentity('term_reminted', PANE_KEY)?.id).toBe(valid.id)
const plan = sqliteFor(db)
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it } from 'vitest'
import { openDecisionGateFromMessage } from './coordinator-decision-gates'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('coordinator decision-gate authority', () => {
let db: OrchestrationDb
@@ -12,7 +13,7 @@ describe('coordinator decision-gate authority', () => {
it('opens a gate only for the sender-owned active Dispatch', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'owned gate target' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', 'tab_owner:leaf_owner')
const dispatch = createRootDispatch(db, task.id, 'term_owner', 'tab_owner:leaf_owner')
const logs: string[] = []
openDecisionGateFromMessage(
@@ -41,13 +42,14 @@ describe('coordinator decision-gate authority', () => {
it('rejects a gate targeting another active Dispatch without mutating either Task', () => {
db = new OrchestrationDb(':memory:')
const attackerTask = db.createTask({ spec: 'attacker assignment' })
const attacker = db.createDispatchContext(
const attacker = createRootDispatch(
db,
attackerTask.id,
'term_attacker',
'tab_attacker:leaf_attacker'
)
const victimTask = db.createTask({ spec: 'victim assignment' })
const victim = db.createDispatchContext(victimTask.id, 'term_victim', 'tab_victim:leaf_victim')
const victim = createRootDispatch(db, victimTask.id, 'term_victim', 'tab_victim:leaf_victim')
const logs: string[] = []
openDecisionGateFromMessage(
@@ -78,7 +80,7 @@ describe('coordinator decision-gate authority', () => {
it('accepts the canonical sender of an imported federated Dispatch', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'remote gate target' })
const dispatch = db.createDispatchContext(task.id, 'remote-worker')
const dispatch = createRootDispatch(db, task.id, 'remote-worker')
openDecisionGateFromMessage(
db,
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it } from 'vitest'
import { applyEscalationToDispatch } from './coordinator-escalation-triage'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('coordinator escalation authority', () => {
let db: OrchestrationDb
@@ -12,13 +13,14 @@ describe('coordinator escalation authority', () => {
it('rejects an escalation targeting another active Dispatch', () => {
db = new OrchestrationDb(':memory:')
const attackerTask = db.createTask({ spec: 'attacker assignment' })
const attacker = db.createDispatchContext(
const attacker = createRootDispatch(
db,
attackerTask.id,
'term_attacker',
'tab_attacker:leaf_attacker'
)
const victimTask = db.createTask({ spec: 'victim assignment' })
const victim = db.createDispatchContext(victimTask.id, 'term_victim')
const victim = createRootDispatch(db, victimTask.id, 'term_victim')
const logs: string[] = []
applyEscalationToDispatch(
@@ -42,7 +44,7 @@ describe('coordinator escalation authority', () => {
it('accepts the canonical sender of an imported federated Dispatch', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'remote escalation target' })
const dispatch = db.createDispatchContext(task.id, 'remote-worker')
const dispatch = createRootDispatch(db, task.id, 'remote-worker')
applyEscalationToDispatch(
db,
@@ -26,6 +26,8 @@ export type CoordinatorRuntime = {
probeWorktreeDrift(worktreeSelector: string): Promise<WorktreeDrift>
// Why: pane-only fallback preserves reservation identity for lightweight runtime fakes.
getTerminalPaneKey?(handle: string): string | null
// Why optional: lightweight fakes omit it and get the fail-closed default.
getNestedWorkerMaxDepth?(): number
// Why: automatic dispatch persists the same authenticated pane/process tuple as manual dispatch.
getOrchestrationDispatchAuthority?(handle: string): {
paneKey: string | null
@@ -68,6 +68,7 @@ export async function dispatchTaskToWorker(params: {
onLog: (msg: string) => void
// Why: the coordinator owns the failed-task list, so a circuit break is reported back instead of mutated here.
onCircuitBroken: (taskId: string) => void
nestedWorkerMaxDepth: number
}): Promise<TaskDispatchResult> {
const { db, runtime, task, targetHandle, baseDrift, onLog } = params
// Why (§3.1): drift check runs before createDispatchContext so a refusal doesn't bump failure_count (carried forward as MAX in db.ts:301-306) and burn the circuit-breaker budget; the task stays `ready` and retries next tick.
@@ -95,13 +96,17 @@ export async function dispatchTaskToWorker(params: {
dispatchAuthority?.paneKey && dispatchAuthority.processIncarnation
? dispatchAuthority.processIncarnation
: undefined
const dispatch = db.createDispatchContext(
task.id,
targetHandle,
const dispatch = db.createDispatchContext({
taskId: task.id,
assigneeHandle: targetHandle,
assigneePaneKey,
dispatchAuthority?.launchTokenHash ?? undefined,
processIncarnation
)
launchTokenHash: dispatchAuthority?.launchTokenHash ?? undefined,
processIncarnation,
// Why system: the automatic loop is host-local Orca code driven by
// coordinator_runs, not a CLI caller, so it is a root by construction.
creator: { kind: 'system' },
maxDepth: params.nestedWorkerMaxDepth
})
// Why: dispatched agents use orca-dev in dev mode to reach the dev runtime's socket, not production (Section 6.4).
const preamble = buildDispatchPreamble({
@@ -7,6 +7,7 @@ import {
DISPATCH_STALE_THRESHOLD,
parseAllowStaleBaseFromSpec
} from './coordinator-stale-base-flag'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
type DriftResult = {
base: string
@@ -223,7 +224,7 @@ describe('Coordinator', () => {
const runtime = createMockRuntime()
const task = db.createTask({ spec: 'send-driven completion' })
const dispatch = db.createDispatchContext(task.id, 'term_a')
const dispatch = createRootDispatch(db, task.id, 'term_a')
const msg = db.insertMessage({
from: 'term_a',
to: 'coord',
@@ -250,7 +251,7 @@ describe('Coordinator', () => {
const runtime = createMockRuntime()
const task = db.createTask({ spec: 'duplicate completion' })
const dispatch = db.createDispatchContext(task.id, 'term_a')
const dispatch = createRootDispatch(db, task.id, 'term_a')
const payload = JSON.stringify({
taskId: task.id,
dispatchId: dispatch.id,
@@ -530,7 +531,7 @@ describe('Coordinator', () => {
// No terminals available so dispatchReadyTasks creates one and we can
// drive the stale-scan deterministically via SQL backdating.
const task = db.createTask({ spec: 'work' })
const ctx = db.createDispatchContext(task.id, 'term_stale')
const ctx = createRootDispatch(db, task.id, 'term_stale')
// Backdate dispatched_at and last_heartbeat_at beyond the 10-min threshold
// so getStaleDispatches returns this row on the first tick.
@@ -569,7 +570,7 @@ describe('Coordinator', () => {
runtime.terminals = [{ handle: 'term_a', worktreeId: 'wt1', connected: true, writable: true }]
const task = db.createTask({ spec: 'work' })
const ctx = db.createDispatchContext(task.id, 'term_a')
const ctx = createRootDispatch(db, task.id, 'term_a')
const coordinator = new Coordinator(db, runtime, {
spec: 'go',
@@ -606,9 +607,9 @@ describe('Coordinator', () => {
const logs: string[] = []
const task = db.createTask({ spec: 'retry-sensitive work' })
const staleCtx = db.createDispatchContext(task.id, 'term_old')
const staleCtx = createRootDispatch(db, task.id, 'term_old')
db.failDispatch(staleCtx.id, 'retry elsewhere')
const activeCtx = db.createDispatchContext(task.id, 'term_current')
const activeCtx = createRootDispatch(db, task.id, 'term_current')
db.insertMessage({
from: 'term_old',
@@ -664,7 +665,7 @@ describe('Coordinator', () => {
const task = db.createTask({ spec: 'owned work' })
const leafId = '11111111-1111-4111-8111-111111111111'
const ctx = db.createDispatchContext(task.id, 'term_owner', `tab_before:${leafId}`)
const ctx = createRootDispatch(db, task.id, 'term_owner', `tab_before:${leafId}`)
db.insertMessage({
from: 'term_reminted',
@@ -13,6 +13,7 @@ import {
listAvailableWorkerTerminals,
warnStaleDispatches
} from './coordinator-task-dispatch'
import { NESTED_WORKER_MAX_DEPTH_DEFAULT } from '../../../shared/nested-worker-depth'
export type CoordinatorOptions = {
spec: string
@@ -283,6 +284,8 @@ export class Coordinator {
baseDrift,
coordinatorHandle: this.opts.coordinatorHandle,
worktree: this.opts.worktree,
nestedWorkerMaxDepth:
this.runtime.getNestedWorkerMaxDepth?.() ?? NESTED_WORKER_MAX_DEPTH_DEFAULT,
onLog: this.opts.onLog,
onCircuitBroken: (taskId) => this.state.failedTasks.push(taskId)
})
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
// Why: buildAgentOrchestrationByPaneKey issues 2 dispatch lookups per terminal
// on EVERY 16ms graph publish. For users who never orchestrate, every one of
@@ -64,7 +65,7 @@ describe('orchestration empty-dispatch short-circuit (benchmark)', () => {
it('still runs the fan-out once a dispatch exists (correctness preserved)', () => {
const db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
db.createDispatchContext(task.id, 'term_5')
createRootDispatch(db, task.id, 'term_5')
const handles = Array.from({ length: 10 }, (_, i) => `term_${i}`)
const contexts = simulateGraphPublish(db, handles)
@@ -75,7 +76,7 @@ describe('orchestration empty-dispatch short-circuit (benchmark)', () => {
it('predicate lifecycle: false when empty, true after dispatch (even completed), false after reset', () => {
const db = new OrchestrationDb(':memory:')
expect(db.hasAnyDispatchContexts()).toBe(false)
const ctx = db.createDispatchContext(db.createTask({ spec: 'work' }).id, 'term_worker')
const ctx = createRootDispatch(db, db.createTask({ spec: 'work' }).id, 'term_worker')
expect(db.hasAnyDispatchContexts()).toBe(true)
// Completed rows still count — recent-completed lookups must stay valid.
db.completeDispatch(ctx.id)
@@ -1,5 +1,6 @@
import { afterEach, describe, expect, it } from 'vitest'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
let db: OrchestrationDb | undefined
@@ -13,7 +14,7 @@ function seedHeartbeatedDispatch(): { d: OrchestrationDb; dispatchId: string } {
const d = new OrchestrationDb(':memory:')
db = d
const task = d.createTask({ spec: 'work' })
const dispatch = d.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(d, task.id, 'term_worker')
d.recordHeartbeat(dispatch.id, '2026-05-03T00:00:00.000Z')
return { d, dispatchId: dispatch.id }
}
@@ -4,6 +4,7 @@ import { join } from 'node:path'
import { afterEach, describe, expect, it, vi } from 'vitest'
import type Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
type DatabaseHarness = {
db: OrchestrationDb
@@ -31,7 +32,7 @@ describe('Task/Dispatch invariant transactions', () => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'atomic work' })
const dependent = db.createTask({ spec: 'dependent work', deps: [task.id] })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const capability = db.mintDispatchCapability({
dispatchId: dispatch.id,
paneKey: 'tab_worker:leaf_worker',
@@ -74,7 +75,7 @@ describe('Task/Dispatch invariant transactions', () => {
it('does not commit a caller-owned transaction', () => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'outer transaction work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const sqlite = sqliteFor(db)
sqlite.exec('BEGIN IMMEDIATE')
@@ -101,7 +102,7 @@ describe('Task/Dispatch invariant transactions', () => {
const sqlite = sqliteFor(db)
sqlite.exec('BEGIN IMMEDIATE')
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
expect(db.getTask(task.id)?.status).toBe('dispatched')
expect(db.getDispatchContextById(dispatch.id)?.status).toBe('dispatched')
sqlite.exec('ROLLBACK')
@@ -115,9 +116,9 @@ describe('Task/Dispatch invariant transactions', () => {
(status) => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'legacy split work' })
const first = db.createDispatchContext(task.id, 'term_first')
const first = createRootDispatch(db, task.id, 'term_first')
sqliteFor(db).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const second = db.createDispatchContext(task.id, 'term_second')
const second = createRootDispatch(db, task.id, 'term_second')
db.updateTaskStatus(task.id, status, 'terminal result')
@@ -131,10 +132,10 @@ describe('Task/Dispatch invariant transactions', () => {
expect(db.getActiveDispatchForTerminal('term_first')).toBeUndefined()
expect(db.getActiveDispatchForTerminal('term_second')).toBeUndefined()
expect(() =>
db.createDispatchContext(db.createTask({ spec: 'first later work' }).id, 'term_first')
createRootDispatch(db, db.createTask({ spec: 'first later work' }).id, 'term_first')
).not.toThrow()
expect(() =>
db.createDispatchContext(db.createTask({ spec: 'second later work' }).id, 'term_second')
createRootDispatch(db, db.createTask({ spec: 'second later work' }).id, 'term_second')
).not.toThrow()
}
)
@@ -142,9 +143,9 @@ describe('Task/Dispatch invariant transactions', () => {
it('does not requeue a legacy split Task while another Dispatch remains active', () => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'legacy split retry' })
const first = db.createDispatchContext(task.id, 'term_first')
const first = createRootDispatch(db, task.id, 'term_first')
sqliteFor(db).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const second = db.createDispatchContext(task.id, 'term_second')
const second = createRootDispatch(db, task.id, 'term_second')
db.failDispatch(second.id, 'targeted failure')
@@ -156,9 +157,9 @@ describe('Task/Dispatch invariant transactions', () => {
it('does not block a legacy split Task while another Dispatch remains active', () => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'legacy split release' })
const first = db.createDispatchContext(task.id, 'term_first')
const first = createRootDispatch(db, task.id, 'term_first')
sqliteFor(db).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const second = db.createDispatchContext(task.id, 'term_second')
const second = createRootDispatch(db, task.id, 'term_second')
expect(db.beginWorkerStop(second.id, 'runtime_test')).toMatchObject({
disposition: 'context_only',
@@ -174,7 +175,7 @@ describe('Task/Dispatch invariant transactions', () => {
(status) => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'guarded work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
expect(() => db.updateTaskStatus(task.id, status, 'must not persist')).toThrowError(
expect.objectContaining({
@@ -215,7 +216,7 @@ describe('Task/Dispatch invariant transactions', () => {
return prepare(sql)
})
expect(() => first.db.createDispatchContext(task.id, 'term_worker')).toThrow(
expect(() => createRootDispatch(first.db, task.id, 'term_worker')).toThrow(
`Task ${task.id} is failed; only ready tasks can be dispatched`
)
expect(injected).toBe(true)
@@ -233,7 +234,8 @@ describe('Task/Dispatch invariant transactions', () => {
let winnerId: string | undefined
vi.spyOn(sqlite, 'prepare').mockImplementation((sql) => {
if (!winnerId && sql.includes('INSERT INTO dispatch_contexts')) {
winnerId = concurrent.db.createDispatchContext(
winnerId = createRootDispatch(
concurrent.db,
secondTask.id,
'term_reminted',
'tab_new:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
@@ -243,7 +245,8 @@ describe('Task/Dispatch invariant transactions', () => {
})
expect(() =>
first.db.createDispatchContext(
createRootDispatch(
first.db,
firstTask.id,
'term_worker',
'tab_old:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
@@ -264,13 +267,16 @@ describe('Task/Dispatch invariant transactions', () => {
it('rejects worker authority when another Dispatch owns the pane', () => {
const { db } = createDatabase()
const ownerTask = db.createTask({ spec: 'current pane owner' })
const owner = db.createDispatchContext(
const owner = createRootDispatch(
db,
ownerTask.id,
'term_owner',
'tab_old:cccccccc-cccc-4ccc-8ccc-cccccccccccc'
)
const workerTask = db.createTask({ spec: 'competing supervised worker' })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: workerTask.id,
startOptions: {}
})
@@ -304,7 +310,12 @@ describe('Task/Dispatch invariant transactions', () => {
(status) => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'supervised lifecycle' })
const started = db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
const capability = db.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_worker',
@@ -348,6 +359,8 @@ describe('Task/Dispatch invariant transactions', () => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'federated lifecycle' })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
federation: {
@@ -4,6 +4,7 @@ import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import type Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
type WorkerFixture = {
dispatchId: string
@@ -55,7 +56,7 @@ describe('Task/Dispatch lifecycle guards', () => {
(outcome) => {
const database = createDatabase()
const task = database.createTask({ spec: 'legacy mixed split' })
const contextOnly = database.createDispatchContext(task.id, 'term_context')
const contextOnly = createRootDispatch(database, task.id, 'term_context')
sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const worker = startWorker(database, task.id, 'reporter')
@@ -75,7 +76,8 @@ describe('Task/Dispatch lifecycle guards', () => {
})
expect(database.getActiveDispatchForTerminal('term_context')).toBeUndefined()
expect(() =>
database.createDispatchContext(
createRootDispatch(
database,
database.createTask({ spec: 'later context work' }).id,
'term_context'
)
@@ -88,7 +90,7 @@ describe('Task/Dispatch lifecycle guards', () => {
const task = database.createTask({ spec: 'reversed legacy mixed split' })
const worker = startWorker(database, task.id, 'reversed_reporter')
sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const contextOnly = database.createDispatchContext(task.id, 'term_reversed_context')
const contextOnly = createRootDispatch(database, task.id, 'term_reversed_context')
expect(
database.settleWorkerReport({
@@ -175,6 +177,8 @@ describe('Task/Dispatch lifecycle guards', () => {
const database = createDatabase()
const task = database.createTask({ spec: `${kind} split start failure` })
const failed = database.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
...(kind === 'federated'
@@ -219,9 +223,14 @@ describe('Task/Dispatch lifecycle guards', () => {
(operation) => {
const database = createDatabase()
const task = database.createTask({ spec: `${operation} historical sibling` })
const contextOnly = database.createDispatchContext(task.id, `term_${operation}`)
const contextOnly = createRootDispatch(database, task.id, `term_${operation}`)
sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const failed = database.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const failed = database.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
database.failWorkerStart(failed.dispatch.id, 'start_failed', 'worker failed to start')
expect(database.getTask(task.id)?.status).toBe('dispatched')
@@ -239,7 +248,8 @@ describe('Task/Dispatch lifecycle guards', () => {
expect(database.getDispatchContextById(contextOnly.id)?.status).toBe('failed')
expect(database.getActiveDispatchForTerminal(`term_${operation}`)).toBeUndefined()
expect(() =>
database.createDispatchContext(
createRootDispatch(
database,
database.createTask({ spec: `${operation} later work` }).id,
`term_${operation}`
)
@@ -305,7 +315,12 @@ describe('Task/Dispatch lifecycle guards', () => {
const task = database.createTask({ spec: 'uncertain legacy worker split' })
const live = startWorker(database, task.id, 'uncertain_live')
sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const uncertain = database.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const uncertain = database.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
database.markWorkerStartUnknown(uncertain.dispatch.id, 'agent_readiness', 'outcome unknown')
expect(database.getTask(task.id)?.status).toBe('blocked')
@@ -331,7 +346,12 @@ describe('Task/Dispatch lifecycle guards', () => {
const task = database.createTask({ spec: `${recovery} uncertain sibling` })
const live = startWorker(database, task.id, `${recovery}_live`)
sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const uncertain = database.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const uncertain = database.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
database.markWorkerStartUnknown(uncertain.dispatch.id, 'agent_readiness', 'outcome unknown')
if (recovery === 'federated-reconcile') {
@@ -381,7 +401,7 @@ describe('Task/Dispatch lifecycle guards', () => {
const task = database.createTask({ spec: 'corrupt gated task' })
const gate = database.createGate({ taskId: task.id, question: 'Proceed?' })
sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id)
const dispatch = database.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(database, task.id, 'term_worker')
sqliteFor(database).prepare("UPDATE tasks SET status = 'blocked' WHERE id = ?").run(task.id)
expect(() => database.resolveGate(gate.id, 'yes')).toThrowError(
@@ -404,7 +424,12 @@ function createDatabase(): OrchestrationDb {
}
function startWorker(database: OrchestrationDb, taskId: string, name: string): WorkerFixture {
const started = database.createStartingWorkerDispatch({ taskId, startOptions: {} })
const started = database.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId,
startOptions: {}
})
const paneSuffix = name.length.toString(16).padStart(12, '0')
const paneKey = `tab_${name}:aaaaaaaa-aaaa-4aaa-8aaa-${paneSuffix}`
const processIncarnation = `${name}:1`
@@ -4,6 +4,7 @@ import { join } from 'node:path'
import { afterEach, describe, expect, it, vi } from 'vitest'
import type Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
type DatabaseHarness = {
db: OrchestrationDb
@@ -28,7 +29,7 @@ describe('Task/Dispatch concurrency', () => {
it('rolls back Dispatch failure when Task requeue fails', () => {
const { db } = createDatabase()
const task = db.createTask({ spec: 'atomic retry failure' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
sqliteFor(db).exec(`
CREATE TRIGGER reject_task_requeue
BEFORE UPDATE OF status ON tasks
@@ -55,7 +56,12 @@ describe('Task/Dispatch concurrency', () => {
const first = createDatabase()
const concurrent = createDatabase(first.path)
const task = first.db.createTask({ spec: 'worker completion wins' })
const started = first.db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = first.db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
const capability = first.db.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_worker',
@@ -117,10 +123,14 @@ describe('Task/Dispatch concurrency', () => {
const losingTask = first.db.createTask({ spec: 'losing worker' })
const winningTask = first.db.createTask({ spec: 'winning worker' })
const loser = first.db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: losingTask.id,
startOptions: {}
})
const winner = concurrent.db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: winningTask.id,
startOptions: {}
})
+43 -42
View File
@@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest'
import Database from '../../sqlite/sync-database'
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
import type { MessageType } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
// Overwrites the datetime('now')-seeded timestamps with explicit fixture values
// so stale-detection assertions stay deterministic (no wall clock).
@@ -255,7 +256,7 @@ describe('OrchestrationDb', () => {
it('completing a task frees its active dispatch context', () => {
const d = createDb()
const task = d.createTask({ spec: 'do it' })
d.createDispatchContext(task.id, 'term_a')
createRootDispatch(d, task.id, 'term_a')
d.updateTaskStatus(task.id, 'completed')
@@ -285,7 +286,7 @@ describe('OrchestrationDb', () => {
const d = createDb()
const ready = d.createTask({ spec: 'ready task' })
const dispatched = d.createTask({ spec: 'active task' })
const ctx = d.createDispatchContext(dispatched.id, 'term_worker')
const ctx = createRootDispatch(d, dispatched.id, 'term_worker')
const rows = d.listTasksWithDispatch()
const readyRow = rows.find((r) => r.id === ready.id)
@@ -300,7 +301,7 @@ describe('OrchestrationDb', () => {
it('listTasksWithDispatch does not surface completed dispatches', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
d.createDispatchContext(task.id, 'term_worker')
createRootDispatch(d, task.id, 'term_worker')
d.updateTaskStatus(task.id, 'completed')
const rows = d.listTasksWithDispatch()
@@ -323,7 +324,7 @@ describe('OrchestrationDb', () => {
it('creates a dispatch context and marks task as dispatched', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
const ctx = d.createDispatchContext(task.id, 'term_worker')
const ctx = createRootDispatch(d, task.id, 'term_worker')
expect(ctx.id).toMatch(/^ctx_/)
expect(ctx.task_id).toBe(task.id)
@@ -337,7 +338,7 @@ describe('OrchestrationDb', () => {
const parent = d.createTask({ spec: 'parent' })
const child = d.createTask({ spec: 'child', deps: [parent.id] })
expect(() => d.createDispatchContext(child.id, 'term_worker')).toThrow(
expect(() => createRootDispatch(d, child.id, 'term_worker')).toThrow(
/only ready tasks can be dispatched/
)
})
@@ -346,9 +347,9 @@ describe('OrchestrationDb', () => {
const d = createDb()
const t1 = d.createTask({ spec: 'first' })
const t2 = d.createTask({ spec: 'second' })
d.createDispatchContext(t1.id, 'term_worker')
createRootDispatch(d, t1.id, 'term_worker')
expect(() => d.createDispatchContext(t2.id, 'term_worker')).toThrow(
expect(() => createRootDispatch(d, t2.id, 'term_worker')).toThrow(
/already has an active dispatch/
)
})
@@ -362,9 +363,9 @@ describe('OrchestrationDb', () => {
const d = createDb()
const t1 = d.createTask({ spec: 'first' })
const t2 = d.createTask({ spec: 'second' })
d.createDispatchContext(t1.id, 'term_old', `tab_1:${LEAF_A}`)
createRootDispatch(d, t1.id, 'term_old', `tab_1:${LEAF_A}`)
expect(() => d.createDispatchContext(t2.id, 'term_new', `tab_1:${LEAF_A}`)).toThrow(
expect(() => createRootDispatch(d, t2.id, 'term_new', `tab_1:${LEAF_A}`)).toThrow(
/already has an active dispatch/
)
})
@@ -373,9 +374,9 @@ describe('OrchestrationDb', () => {
const d = createDb()
const t1 = d.createTask({ spec: 'first' })
const t2 = d.createTask({ spec: 'second' })
d.createDispatchContext(t1.id, 'term_old', `tab_1:${LEAF_A}`)
createRootDispatch(d, t1.id, 'term_old', `tab_1:${LEAF_A}`)
expect(() => d.createDispatchContext(t2.id, 'term_new', `tab_2:${LEAF_A}`)).toThrow(
expect(() => createRootDispatch(d, t2.id, 'term_new', `tab_2:${LEAF_A}`)).toThrow(
/already has an active dispatch/
)
})
@@ -384,37 +385,37 @@ describe('OrchestrationDb', () => {
const d = createDb()
const t1 = d.createTask({ spec: 'first' })
const t2 = d.createTask({ spec: 'second' })
d.createDispatchContext(t1.id, 'term_a', `tab_1:${LEAF_A}`)
createRootDispatch(d, t1.id, 'term_a', `tab_1:${LEAF_A}`)
expect(() => d.createDispatchContext(t2.id, 'term_b', `tab_1:${LEAF_B}`)).not.toThrow()
expect(() => createRootDispatch(d, t2.id, 'term_b', `tab_1:${LEAF_B}`)).not.toThrow()
})
it('falls back to handle lock when pane keys are missing', () => {
const d = createDb()
const t1 = d.createTask({ spec: 'first' })
const t2 = d.createTask({ spec: 'second' })
d.createDispatchContext(t1.id, 'term_worker')
createRootDispatch(d, t1.id, 'term_worker')
// New dispatch has a pane key but the active row is legacy (no pane key):
// only handle identity can lock; a different handle is free.
expect(() => d.createDispatchContext(t2.id, 'term_other', `tab_1:${LEAF_A}`)).not.toThrow()
expect(() => createRootDispatch(d, t2.id, 'term_other', `tab_1:${LEAF_A}`)).not.toThrow()
})
it('allows dispatch to a terminal after previous dispatch completes', () => {
const d = createDb()
const t1 = d.createTask({ spec: 'first' })
const t2 = d.createTask({ spec: 'second' })
const ctx1 = d.createDispatchContext(t1.id, 'term_worker')
const ctx1 = createRootDispatch(d, t1.id, 'term_worker')
d.completeDispatch(ctx1.id)
expect(() => d.createDispatchContext(t2.id, 'term_worker')).not.toThrow()
expect(() => createRootDispatch(d, t2.id, 'term_worker')).not.toThrow()
})
it('getDispatchContext returns latest for a task', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
const ctx = d.createDispatchContext(task.id, 'term_a')
const ctx = createRootDispatch(d, task.id, 'term_a')
const found = d.getDispatchContext(task.id)
expect(found?.id).toBe(ctx.id)
})
@@ -422,9 +423,9 @@ describe('OrchestrationDb', () => {
it('getDispatchContext uses insertion order when timestamps tie', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
const ctx1 = d.createDispatchContext(task.id, 'term_a')
const ctx1 = createRootDispatch(d, task.id, 'term_a')
d.failDispatch(ctx1.id, 'retry')
const ctx2 = d.createDispatchContext(task.id, 'term_a')
const ctx2 = createRootDispatch(d, task.id, 'term_a')
expect(d.getDispatchContext(task.id)?.id).toBe(ctx2.id)
})
@@ -432,7 +433,7 @@ describe('OrchestrationDb', () => {
it('getActiveDispatchForTerminal returns active dispatch', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
d.createDispatchContext(task.id, 'term_a')
createRootDispatch(d, task.id, 'term_a')
const active = d.getActiveDispatchForTerminal('term_a')
expect(active?.task_id).toBe(task.id)
@@ -442,10 +443,10 @@ describe('OrchestrationDb', () => {
it('getLatestDispatchForTerminal returns the most recent completed dispatch', () => {
const d = createDb()
const firstTask = d.createTask({ spec: 'first' })
const first = d.createDispatchContext(firstTask.id, 'term_a')
const first = createRootDispatch(d, firstTask.id, 'term_a')
d.completeDispatch(first.id)
const secondTask = d.createTask({ spec: 'second' })
const second = d.createDispatchContext(secondTask.id, 'term_a')
const second = createRootDispatch(d, secondTask.id, 'term_a')
d.completeDispatch(second.id)
const latest = d.getLatestDispatchForTerminal('term_a')
@@ -457,19 +458,19 @@ describe('OrchestrationDb', () => {
it('circuit breaker trips after 3 failures', () => {
const d = createDb()
const task = d.createTask({ spec: 'flaky' })
const ctx = d.createDispatchContext(task.id, 'term_a')
const ctx = createRootDispatch(d, task.id, 'term_a')
const after1 = d.failDispatch(ctx.id, 'timeout')
expect(after1?.failure_count).toBe(1)
expect(after1?.status).toBe('failed')
expect(d.getTask(task.id)?.status).toBe('ready')
const ctx2 = d.createDispatchContext(task.id, 'term_a')
const ctx2 = createRootDispatch(d, task.id, 'term_a')
const after2 = d.failDispatch(ctx2.id, 'timeout')
expect(after2?.failure_count).toBe(2)
expect(after2?.status).toBe('failed')
const ctx3 = d.createDispatchContext(task.id, 'term_a')
const ctx3 = createRootDispatch(d, task.id, 'term_a')
const after3 = d.failDispatch(ctx3.id, 'timeout')
expect(after3?.failure_count).toBe(3)
expect(after3?.status).toBe('circuit_broken')
@@ -480,7 +481,7 @@ describe('OrchestrationDb', () => {
it('completeDispatch sets completed_at', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
const ctx = d.createDispatchContext(task.id, 'term_a')
const ctx = createRootDispatch(d, task.id, 'term_a')
d.completeDispatch(ctx.id)
const updated = d.getDispatchContext(task.id)
@@ -493,7 +494,7 @@ describe('OrchestrationDb', () => {
it('creates a gate and blocks the task', () => {
const d = createDb()
const task = d.createTask({ spec: 'needs approval' })
d.createDispatchContext(task.id, 'term_a')
createRootDispatch(d, task.id, 'term_a')
const gate = d.createGate({
taskId: task.id,
question: 'Proceed?',
@@ -622,7 +623,7 @@ describe('OrchestrationDb', () => {
const d = createDb()
d.insertMessage({ from: 'a', to: 'b', subject: 'test' })
const task = d.createTask({ spec: 'work' })
d.createDispatchContext(task.id, 'term_a')
createRootDispatch(d, task.id, 'term_a')
d.resetTasks()
@@ -647,7 +648,7 @@ describe('OrchestrationDb', () => {
it('recordHeartbeat updates last_heartbeat_at on dispatched rows', () => {
const d = createDb()
const task = d.createTask({ spec: 'work' })
const ctx = d.createDispatchContext(task.id, 'term_a')
const ctx = createRootDispatch(d, task.id, 'term_a')
d.recordHeartbeat(ctx.id, '2026-05-04T00:00:00.000Z')
const after = d.getDispatchContext(task.id)
@@ -665,10 +666,10 @@ describe('OrchestrationDb', () => {
const taskB = d.createTask({ spec: 'b' })
const taskC = d.createTask({ spec: 'c' })
const taskD = d.createTask({ spec: 'd' })
const ctxA = d.createDispatchContext(taskA.id, 'term_a')
const ctxB = d.createDispatchContext(taskB.id, 'term_b')
const ctxC = d.createDispatchContext(taskC.id, 'term_c')
const ctxD = d.createDispatchContext(taskD.id, 'term_d')
const ctxA = createRootDispatch(d, taskA.id, 'term_a')
const ctxB = createRootDispatch(d, taskB.id, 'term_b')
const ctxC = createRootDispatch(d, taskC.id, 'term_c')
const ctxD = createRootDispatch(d, taskD.id, 'term_d')
d.completeDispatch(ctxD.id)
const now = Date.now()
@@ -709,15 +710,15 @@ describe('OrchestrationDb', () => {
// Fresh worker: dispatched 12:00, heartbeat 12:05 (space-format), both
// after the 11:55 threshold → NOT stale.
const fresh = d.createDispatchContext(d.createTask({ spec: 'fresh' }).id, 'term_fresh')
const fresh = createRootDispatch(d, d.createTask({ spec: 'fresh' }).id, 'term_fresh')
setDispatchTimes(d, fresh.id, '2026-07-12 12:00:00', '2026-07-12 12:05:00')
// Legacy ISO-format fresh row (mixed-format table) stays fresh too.
const legacy = d.createDispatchContext(d.createTask({ spec: 'legacy' }).id, 'term_legacy')
const legacy = createRootDispatch(d, d.createTask({ spec: 'legacy' }).id, 'term_legacy')
setDispatchTimes(d, legacy.id, '2026-07-12T12:00:00.000Z', '2026-07-12T12:05:00.000Z')
// Genuinely hung: dispatched + heartbeated at 10:00, ~2h before threshold.
const hung = d.createDispatchContext(d.createTask({ spec: 'hung' }).id, 'term_hung')
const hung = createRootDispatch(d, d.createTask({ spec: 'hung' }).id, 'term_hung')
setDispatchTimes(d, hung.id, '2026-07-12 10:00:00', '2026-07-12 10:00:00')
const stale = d.getStaleDispatches('2026-07-12T11:55:00.000Z')
@@ -729,7 +730,7 @@ describe('OrchestrationDb', () => {
// Space-format dispatched_at one minute after the threshold, no heartbeat
// yet → still inside the grace window, must not be flagged.
const ctx = d.createDispatchContext(d.createTask({ spec: 'x' }).id, 'term_x')
const ctx = createRootDispatch(d, d.createTask({ spec: 'x' }).id, 'term_x')
setDispatchTimes(d, ctx.id, '2026-07-12 12:00:00')
const stale = d.getStaleDispatches('2026-07-12T11:59:00.000Z')
@@ -741,7 +742,7 @@ describe('OrchestrationDb', () => {
it('getStaleDispatches keeps a fresh row just after a UTC-midnight threshold (#8452)', () => {
const d = createDb()
const ctx = d.createDispatchContext(d.createTask({ spec: 'midnight' }).id, 'term_midnight')
const ctx = createRootDispatch(d, d.createTask({ spec: 'midnight' }).id, 'term_midnight')
setDispatchTimes(d, ctx.id, '2026-05-04 00:04:00')
const stale = d.getStaleDispatches('2026-05-04T00:00:00.000Z')
@@ -754,7 +755,7 @@ describe('OrchestrationDb', () => {
it('getStaleDispatches keeps a live worker with a fresh space-format heartbeat (#8452)', () => {
const d = createDb()
const ctx = d.createDispatchContext(d.createTask({ spec: 'live' }).id, 'term_live')
const ctx = createRootDispatch(d, d.createTask({ spec: 'live' }).id, 'term_live')
setDispatchTimes(d, ctx.id, '2026-07-12 10:00:00', '2026-07-12 11:59:00')
const stale = d.getStaleDispatches('2026-07-12T11:55:00.000Z')
@@ -911,7 +912,7 @@ describe('OrchestrationDb', () => {
// (b) last_heartbeat_at column exists on dispatch_contexts
const task = d.createTask({ spec: 'work' })
const ctx = d.createDispatchContext(task.id, 'term_a')
const ctx = createRootDispatch(d, task.id, 'term_a')
d.recordHeartbeat(ctx.id, '2026-05-04T00:00:00.000Z')
expect(d.getDispatchContext(task.id)?.last_heartbeat_at).toBe('2026-05-04T00:00:00.000Z')
expect(d.getTask(task.id)?.task_title).toBe('work')
@@ -942,7 +943,7 @@ describe('OrchestrationDb', () => {
db = d
const task = d.createTask({ spec: 'work' })
const ctx = d.createDispatchContext(task.id, 'term_a', 'tab_1:leaf_1')
const ctx = createRootDispatch(d, task.id, 'term_a', 'tab_1:leaf_1')
expect(d.getDispatchContextById(ctx.id)?.assignee_pane_key).toBe('tab_1:leaf_1')
const msg = d.insertMessage({
+1 -1
View File
@@ -6,7 +6,7 @@ export {
} from './db/contract-constants'
export type { RunListPage, TaskRuntimeLineageRow } from './db/run-list-page'
export { ORCHESTRATION_DELIVERY_BATCH_LIMIT } from './db/messages/mailbox-routing-page'
export { DISPATCH_CONTEXT_CLAIM_SQL } from './db/dispatch-context/dispatch-context-store'
export { DISPATCH_CONTEXT_CLAIM_SQL } from './db/dispatch-row-writer'
export type {
ForeignDirectMailboxRoutingPage,
MailboxRoutingPage
@@ -4,6 +4,7 @@ import { attachDispatchCapability } from './dispatch-context/dispatch-capability
import { attachDispatchCompletion } from './dispatch-context/dispatch-completion'
import { attachDispatchContextStore } from './dispatch-context/dispatch-context-store'
import { attachDispatchLookup } from './dispatch-context/dispatch-lookup'
import { attachDispatchDepth } from './dispatch-depth'
import { attachWorkerReportSettlement } from './dispatch-context/worker-report-settlement'
import { attachFederatedDispatchStore } from './federation/federated-dispatch-store'
import { attachFederationRelayAck } from './federation/federation-relay-ack'
@@ -115,6 +116,7 @@ export function attachOrchestrationDbMethods(ctor: { prototype: object }): void
attachDispatchContextStore(ctor)
attachDispatchCapability(ctor)
attachDispatchLookup(ctor)
attachDispatchDepth(ctor)
attachDispatchCompletion(ctor)
attachWorkerReportSettlement(ctor)
attachDecisionGateStore(ctor)
@@ -7,4 +7,4 @@ export const LEGACY_CONTRACT_VERSION = 0
export const CURRENT_CONTRACT_VERSION = ORCHESTRATION_CONTRACT_VERSION
// Schema versions: v2 'heartbeat'+last_heartbeat_at, v3 delivered_at, v4 task-creator terminal, v5 task_title/display_name, v6 pane identity, v7 lightweight Runs, v8 crash-safe Run deliveries, v9 durable question threads, v10 Dispatch capabilities, v11 durable mutation receipts, v12 composed worker state, v18 post-v6 version-skew repair, v19 adopted legacy Runs and compatibility receipts, v20 legacy question backfill, v21 legacy scheduler-loss provenance, v22 dispatch assignee lookup, v23 worker terminal resource ownership, v24 creator-incarnation authority, v25 active Dispatch handle lookup, v26 indexed mutation receipt capacity, v27 durable federation acknowledgments, v28 durable local mutation caller identity.
export const SCHEMA_VERSION = 29
export const SCHEMA_VERSION = 30
@@ -3,48 +3,27 @@ import { OrchestrationError } from '../../orchestration-error'
import { parsePaneKey } from '../../../../../shared/stable-pane-id'
import { CURRENT_CONTRACT_VERSION } from '../contract-constants'
import { generateId } from '../generated-id'
import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL, paneKeyMatchSuffix } from '../pane-key-match'
import { paneKeyMatchSuffix } from '../pane-key-match'
import { claimDispatchContextRow } from '../dispatch-row-writer'
import type { DispatchCreator } from '../dispatch-depth'
import type { OrchestrationDb } from '../orchestration-db'
export const DISPATCH_CONTEXT_CLAIM_SQL = `INSERT INTO dispatch_contexts (
id, run_id, task_id, contract_version, launch_token_hash,
assignee_handle, assignee_pane_key, process_incarnation,
status, failure_count, dispatched_at
)
SELECT ?, run_id, id, ?, ?, ?, ?, ?, 'dispatched', ?, datetime('now')
FROM tasks
WHERE id = ? AND status = 'ready'
AND NOT EXISTS (
SELECT 1 FROM dispatch_contexts active
WHERE active.assignee_handle = ?
AND active.status IN ('pending', 'dispatched')
)
AND (
? IS NULL OR NOT EXISTS (
SELECT 1 FROM dispatch_contexts active
WHERE active.assignee_pane_key = ?
AND active.status IN ('pending', 'dispatched')
)
)
AND (
? IS NULL OR NOT EXISTS (
SELECT 1 FROM dispatch_contexts active
WHERE active.assignee_pane_key IS NOT NULL
AND active.status IN ('pending', 'dispatched')
AND instr(active.assignee_pane_key, ':') > 1
AND ${DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL} = ?
)
)`
export function createDispatchContext(
this: OrchestrationDb,
taskId: string,
assigneeHandle: string,
// Why: pane key is the remint-stable identity behind the handle — lets worker_done ownership survive handle reissue.
assigneePaneKey?: string,
launchTokenHash?: string,
processIncarnation?: string
params: {
taskId: string
assigneeHandle: string
// Why: pane key is the remint-stable identity behind the handle — lets worker_done ownership survive handle reissue.
assigneePaneKey?: string
launchTokenHash?: string
processIncarnation?: string
/** Who is dispatching, for nesting depth. Required so a new caller must decide. */
creator: DispatchCreator
maxDepth: number
}
): DispatchContextRow {
const { taskId, assigneeHandle, assigneePaneKey, launchTokenHash, processIncarnation } = params
const depth = this.resolveChildDispatchDepth(params.creator, params.maxDepth)
const task = this.getTask(taskId)
if (!task) {
throw new Error(`Task not found: ${taskId}`)
@@ -73,23 +52,18 @@ export function createDispatchContext(
const id = generateId('ctx')
this.db.exec('SAVEPOINT create_dispatch_context')
try {
const inserted = this.db
.prepare(DISPATCH_CONTEXT_CLAIM_SQL)
.run(
id,
CURRENT_CONTRACT_VERSION,
launchTokenHash ?? null,
assigneeHandle,
assigneePaneKey ?? null,
processIncarnation ?? null,
priorFailures,
taskId,
assigneeHandle,
assigneePaneKey ?? null,
assigneePaneKey ?? null,
paneSuffix,
paneSuffix
)
const inserted = claimDispatchContextRow(this.db, {
id,
contractVersion: CURRENT_CONTRACT_VERSION,
launchTokenHash: launchTokenHash ?? null,
assigneeHandle,
assigneePaneKey: assigneePaneKey ?? null,
processIncarnation: processIncarnation ?? null,
priorFailures,
depth,
taskId,
paneSuffix
})
if (inserted.changes !== 1) {
const current = this.getTask(taskId)
const occupied = this.findActiveDispatchForAssignee(assigneeHandle, assigneePaneKey)
@@ -0,0 +1,271 @@
import { afterEach, describe, expect, it } from 'vitest'
import { OrchestrationDb } from '../db'
import { AmbiguousDispatchParentError } from './dispatch-depth'
/**
* These pin the fence Orca documented but never enforced: before this feature a
* dispatched worker could create its own Run and dispatch sub-workers freely.
* Every rejection case here passes on the pre-change tree.
*/
describe('nested worker depth', () => {
let db: OrchestrationDb
const SYSTEM = { kind: 'system' } as const
const UNCAPPED = Number.MAX_SAFE_INTEGER
afterEach(() => db?.close())
function coordinatorDispatchesWorker(maxDepth = UNCAPPED) {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'root task' })
const worker = db.createDispatchContext({
taskId: task.id,
assigneeHandle: 'term_worker',
assigneePaneKey: 'tab_worker:leaf_worker',
creator: SYSTEM,
maxDepth
})
return worker
}
it('stamps a root coordinator dispatch at depth 1', () => {
expect(coordinatorDispatchesWorker().depth).toBe(1)
})
it('refuses a worker dispatching a sub-worker at the default cap', () => {
coordinatorDispatchesWorker()
const nested = db.createTask({ spec: 'nested task' })
expect(() =>
db.createDispatchContext({
taskId: nested.id,
assigneeHandle: 'term_sub',
assigneePaneKey: 'tab_sub:leaf_sub',
creator: {
kind: 'terminal',
handle: 'term_worker',
paneKey: 'tab_worker:leaf_worker'
},
maxDepth: 1
})
).toThrow(/depth 2 \(max 1\)/)
})
it('tells the refused worker to complete the task itself', () => {
coordinatorDispatchesWorker()
const nested = db.createTask({ spec: 'nested task' })
expect(() =>
db.createDispatchContext({
taskId: nested.id,
assigneeHandle: 'term_sub',
creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' },
maxDepth: 1
})
).toThrow(/Complete this task yourself/)
})
it('permits one more generation when the cap is raised, and records depth 2', () => {
coordinatorDispatchesWorker()
const nested = db.createTask({ spec: 'nested task' })
const sub = db.createDispatchContext({
taskId: nested.id,
assigneeHandle: 'term_sub',
assigneePaneKey: 'tab_sub:leaf_sub',
creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' },
maxDepth: 2
})
expect(sub.depth).toBe(2)
})
it('closes the run-create bypass: a fresh Run does not reset the creator depth', () => {
// The old fence keyed off Run binding, so a worker that created its own Run
// walked straight through. Depth comes from the creator's dispatch instead.
coordinatorDispatchesWorker()
const ownRun = db.createRun({
objective: 'worker-owned run',
coordinatorHandle: 'term_worker',
coordinatorPaneKey: 'tab_worker:leaf_worker'
})
const nested = db.createTask({ spec: 'nested task', runId: ownRun.id })
expect(() =>
db.createDispatchContext({
taskId: nested.id,
assigneeHandle: 'term_sub',
creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' },
maxDepth: 1
})
).toThrow(/depth 2 \(max 1\)/)
})
it('treats the in-process coordinator loop as a root even from a worker pane', () => {
coordinatorDispatchesWorker()
expect(db.resolveCreatorDepth({ kind: 'system' })).toBe(0)
})
it('resolves an unknown terminal to root depth', () => {
db = new OrchestrationDb(':memory:')
expect(db.resolveCreatorDepth({ kind: 'terminal', handle: 'term_nobody' })).toBe(0)
})
describe('remote attachments as parents', () => {
const PANE = 'tab_remote:leaf_remote'
const INCARNATION = 'inc-1'
function attachRemoteWorker(state: string, depth: number, paneKey = PANE, inc = INCARNATION) {
db.db
.prepare(
`INSERT INTO remote_dispatch_attachments
(dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch,
pane_key, process_incarnation, state, depth)
VALUES (?, ?, 'peer', 1, 'epoch', ?, ?, ?, ?)`
)
.run(`ctx_${state}_${depth}_${paneKey}_${inc}`, 'task_remote', paneKey, inc, state, depth)
}
// Loss of contact is never evidence of process death: an unverifiable remote
// worker must still block nesting. See docs/reference/ssh-execution-boundary.md.
for (const state of ['starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown']) {
it(`counts a '${state}' attachment as a live parent`, () => {
db = new OrchestrationDb(':memory:')
attachRemoteWorker(state, 1)
expect(
db.resolveCreatorDepth({
kind: 'terminal',
handle: 'term_remote',
paneKey: PANE,
processIncarnation: INCARNATION
})
).toBe(1)
})
}
for (const state of ['succeeded', 'failed', 'stopped', 'abandoned']) {
it(`does not count a settled '${state}' attachment`, () => {
db = new OrchestrationDb(':memory:')
attachRemoteWorker(state, 1)
expect(
db.resolveCreatorDepth({
kind: 'terminal',
handle: 'term_remote',
paneKey: PANE,
processIncarnation: INCARNATION
})
).toBe(0)
})
}
it('ignores an attachment whose pane was reused by a new process', () => {
db = new OrchestrationDb(':memory:')
attachRemoteWorker('ready', 2)
expect(
db.resolveCreatorDepth({
kind: 'terminal',
handle: 'term_remote',
paneKey: PANE,
processIncarnation: 'inc-2'
})
).toBe(0)
})
it('fails closed when one identity matches two live attachments', () => {
db = new OrchestrationDb(':memory:')
attachRemoteWorker('ready', 1)
attachRemoteWorker('starting', 2)
expect(() =>
db.resolveCreatorDepth({
kind: 'terminal',
handle: 'term_remote',
paneKey: PANE,
processIncarnation: INCARNATION
})
).toThrow(AmbiguousDispatchParentError)
})
it('takes the maximum when a process holds both a local and a remote role', () => {
// Query order must not decide the answer: the deeper role governs.
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'local role' })
db.createDispatchContext({
taskId: task.id,
assigneeHandle: 'term_both',
assigneePaneKey: PANE,
creator: { kind: 'system' },
maxDepth: UNCAPPED
})
attachRemoteWorker('ready', 3)
expect(
db.resolveCreatorDepth({
kind: 'terminal',
handle: 'term_both',
paneKey: PANE,
processIncarnation: INCARNATION
})
).toBe(3)
})
})
describe('the supervised worker-start path', () => {
// r2 put enforcement in createDispatchContext and missed this entirely:
// worker-start has its own insert, and so does every retry through it.
function startWorker(
taskId: string,
creator: Parameters<OrchestrationDb['resolveCreatorDepth']>[0],
maxDepth: number
) {
return db.createStartingWorkerDispatch({
taskId,
startOptions: {},
creator,
maxDepth
})
}
it('stamps depth 1 for a root coordinator', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'root work' })
expect(startWorker(task.id, SYSTEM, UNCAPPED).dispatch.depth).toBe(1)
})
it('refuses a worker starting a sub-worker at the default cap', () => {
coordinatorDispatchesWorker()
const nested = db.createTask({ spec: 'nested work' })
expect(() =>
startWorker(
nested.id,
{ kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' },
1
)
).toThrow(/depth 2 \(max 1\)/)
})
it('refuses a worker retrying into a sub-worker at the default cap', () => {
coordinatorDispatchesWorker()
const nested = db.createTask({ spec: 'nested retry work' })
const first = startWorker(nested.id, SYSTEM, UNCAPPED)
db.failWorkerStart(first.dispatch.id, 'accepted', 'first attempt failed')
expect(() =>
db.createStartingWorkerDispatch({
taskId: nested.id,
startOptions: {},
retryOf: first.dispatch.id,
creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' },
maxDepth: 1
})
).toThrow(/depth 2 \(max 1\)/)
})
})
it('keeps a local row with a null process incarnation eligible as a parent', () => {
// Context-only dispatch stores null on purpose; requiring an incarnation
// locally would silently drop real parents and fail open.
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'context only' })
const row = db.createDispatchContext({
taskId: task.id,
assigneeHandle: 'term_ctx',
assigneePaneKey: 'tab_ctx:leaf_ctx',
creator: { kind: 'system' },
maxDepth: UNCAPPED
})
expect(row.process_incarnation).toBeNull()
expect(db.resolveCreatorDepth({ kind: 'terminal', handle: 'term_ctx' })).toBe(1)
})
})
@@ -0,0 +1,156 @@
import {
NESTED_WORKER_DEPTH_EXCEEDED_CODE,
NESTED_WORKER_DEPTH_EXCEEDED_NEXT_STEPS,
ROOT_DISPATCH_DEPTH,
nestedWorkerDepthExceededMessage
} from '../../../../shared/nested-worker-depth'
import { OrchestrationError } from '../orchestration-error'
import { isEquivalentPaneKey } from './pane-key-match'
import type { OrchestrationDb } from './orchestration-db'
import type { DispatchContextRow, RemoteDispatchAttachmentRow } from '../types'
/**
* Who is creating a dispatch row, for nesting-depth purposes.
*
* `system` is Orca's own in-process coordinator loop, which is host-local code
* rather than a CLI caller and is a root by construction. It is an internal
* discriminated branch on purpose — never a caller-supplied value, or a worker
* could claim to be the loop.
*/
export type DispatchCreator =
| { kind: 'system' }
| {
kind: 'terminal'
handle: string
paneKey?: string
/** Remote attachment matching requires the exact incarnation; local rows do not. */
processIncarnation?: string
}
/**
* Attachment states in which the worker may still be running.
*
* `start_unknown` means prompt delivery may have succeeded; `stopping` and
* `stop_unknown` do not establish that the process exited. Loss of contact is
* never evidence of process death — see docs/reference/ssh-execution-boundary.md.
* An `unverifiable` worker must still count as a nesting parent.
*/
const POTENTIALLY_LIVE_ATTACHMENT_STATES = [
'starting',
'ready',
'start_unknown',
'stopping',
'stop_unknown'
] as const
export class AmbiguousDispatchParentError extends Error {
constructor(message: string) {
super(message)
this.name = 'AmbiguousDispatchParentError'
}
}
/**
* Depth of the deepest live role this caller currently holds.
*
* Why the maximum rather than the first match: one terminal process can hold a
* local dispatch and a remote attachment at the same time, and the command
* cannot say which role motivated it. Taking the maximum cannot undercount, so
* it cannot let a deep worker pass as a shallow one.
*/
export function resolveCreatorDepth(this: OrchestrationDb, creator: DispatchCreator): number {
if (creator.kind === 'system') {
return ROOT_DISPATCH_DEPTH
}
const depths: number[] = []
// Local rows match on handle/pane as they always have. process_incarnation is
// nullable here and context-only dispatch stores null deliberately, so
// requiring it would drop real parents.
const local = this.findActiveDispatchForAssignee(creator.handle, creator.paneKey) as
| DispatchContextRow
| undefined
if (local) {
depths.push(local.depth)
}
for (const attachment of findPotentiallyLiveAttachmentsForCreator.call(this, creator)) {
depths.push(attachment.depth)
}
return depths.length > 0 ? Math.max(...depths) : ROOT_DISPATCH_DEPTH
}
/**
* Remote attachments matching this caller's pane AND exact process incarnation.
*
* Handle is deliberately not compared: identity survives handle remint and
* nothing updates `remote_dispatch_attachments.terminal_handle` when it happens,
* so a stored-handle predicate would reject a live federated parent. Pane
* equivalence plus exact incarnation is what remote authority already uses.
*/
function findPotentiallyLiveAttachmentsForCreator(
this: OrchestrationDb,
creator: Extract<DispatchCreator, { kind: 'terminal' }>
): RemoteDispatchAttachmentRow[] {
if (!creator.paneKey || !creator.processIncarnation) {
return []
}
const placeholders = POTENTIALLY_LIVE_ATTACHMENT_STATES.map(() => '?').join(', ')
const rows = this.db
.prepare(
`SELECT * FROM remote_dispatch_attachments
WHERE process_incarnation = ?
AND pane_key IS NOT NULL
AND state IN (${placeholders})`
)
.all(
creator.processIncarnation,
...POTENTIALLY_LIVE_ATTACHMENT_STATES
) as RemoteDispatchAttachmentRow[]
const matches = rows.filter(
(row) => row.pane_key !== null && isEquivalentPaneKey(row.pane_key, creator.paneKey as string)
)
// Two live attachments for one identity is an anomaly, not a depth question.
// Surface it rather than silently picking one.
if (matches.length > 1) {
throw new AmbiguousDispatchParentError(
`Terminal ${creator.handle} matches ${matches.length} live remote attachments; cannot establish nesting depth.`
)
}
return matches
}
/**
* Depth to stamp on a row this creator is about to make, rejecting over-cap.
*
* Every path that mints a live worker goes through here, so the cap cannot be
* skipped by adding a new spawn verb.
*/
export function resolveChildDispatchDepth(
this: OrchestrationDb,
creator: DispatchCreator,
maxDepth: number
): number {
const childDepth = this.resolveCreatorDepth(creator) + 1
if (childDepth > maxDepth) {
throw new OrchestrationError(
NESTED_WORKER_DEPTH_EXCEEDED_CODE,
nestedWorkerDepthExceededMessage(childDepth, maxDepth),
{ effectsApplied: false, nextSteps: [...NESTED_WORKER_DEPTH_EXCEEDED_NEXT_STEPS] }
)
}
return childDepth
}
export type DispatchDepthMethods = {
resolveCreatorDepth: typeof resolveCreatorDepth
resolveChildDispatchDepth: typeof resolveChildDispatchDepth
}
export function attachDispatchDepth(ctor: { prototype: object }): void {
Object.assign(ctor.prototype, { resolveCreatorDepth, resolveChildDispatchDepth })
}
@@ -0,0 +1,139 @@
import { readFileSync, readdirSync, statSync } from 'node:fs'
import { join, relative, resolve } from 'node:path'
import { describe, expect, it } from 'vitest'
/**
* Guard the live-worker row chokepoint at the tree level rather than per call site.
*
* Nesting depth has to be stamped on every row that represents a live supervised
* worker. Three separate modules used to own their own INSERT, and three review
* rounds each found one more spawn path than the previous round believed existed.
* `dispatch-row-writer.ts` owns the statements once; this test is what stops the
* fourth path from owning one again.
*
* Known limit, recorded rather than assumed away: this scans SQL string literals.
* SQL assembled from a shared table-name constant, split template fragments, or a
* query builder would evade it — see the detector cases below.
*/
const WRITER_MODULE = 'src/main/runtime/orchestration/db/dispatch-row-writer.ts'
const GUARDED_TABLES = ['dispatch_contexts', 'remote_dispatch_attachments'] as const
/** `INSERT ... INTO <table>`, tolerating OR-clauses and newlines between the words. */
const insertPattern = (table: string): RegExp =>
new RegExp(String.raw`INSERT\b[\s\S]{0,40}?\bINTO\s+${table}\b`, 'i')
/**
* Schema DDL, migrations, and reset all legitimately name these tables. They
* create, alter, and delete rows — they never mint a live worker.
*/
const EXEMPT_PATH_FRAGMENTS = ['/db/schema/', '/db/reset/', '/orchestration-schema-version-skew']
const SCANNED_EXTENSIONS = ['.ts', '.tsx']
const IGNORED_DIRECTORIES = new Set(['node_modules', 'dist', 'out', 'build', '.git'])
function isTestFile(path: string): boolean {
return (
/\.(?:test|spec)\.tsx?$/.test(path) ||
/(?:test-harness|test-utils|test-setup|test-fixture)/.test(path) ||
path.includes('/__tests__/') ||
path.includes('/__fixtures__/')
)
}
function collectSourceFiles(root: string): string[] {
const found: string[] = []
let entries: string[]
try {
entries = readdirSync(root)
} catch {
return found
}
for (const entry of entries) {
if (IGNORED_DIRECTORIES.has(entry)) {
continue
}
const full = join(root, entry)
if (statSync(full).isDirectory()) {
found.push(...collectSourceFiles(full))
} else if (SCANNED_EXTENSIONS.some((ext) => entry.endsWith(ext))) {
found.push(full)
}
}
return found
}
describe('live-worker row insert boundary', () => {
const repoRoot = resolve(__dirname, '../../../../..')
const srcRoot = join(repoRoot, 'src')
it('inserts guarded tables only from dispatch-row-writer.ts', () => {
const offenders: string[] = []
for (const file of collectSourceFiles(srcRoot)) {
const rel = relative(repoRoot, file).split('\\').join('/')
if (rel === WRITER_MODULE || isTestFile(rel)) {
continue
}
if (EXEMPT_PATH_FRAGMENTS.some((fragment) => rel.includes(fragment))) {
continue
}
const contents = readFileSync(file, 'utf8')
for (const table of GUARDED_TABLES) {
if (insertPattern(table).test(contents)) {
offenders.push(`${rel} inserts ${table}`)
}
}
}
expect(offenders).toEqual([])
})
it('the writer module actually owns an insert for every guarded table', () => {
const contents = readFileSync(join(repoRoot, WRITER_MODULE), 'utf8')
for (const table of GUARDED_TABLES) {
expect(insertPattern(table).test(contents)).toBe(true)
}
})
it('does not fire on schema DDL, migration DDL, or reset SQL', () => {
// Why explicit: a naive identifier scan flags all three, which is how the
// first two drafts of this ratchet failed against their own tree.
const exempt = [
'src/main/runtime/orchestration/db/schema/create-graph-tables-sql.ts',
'src/main/runtime/orchestration/db/schema/migrate-v13-v30.ts',
'src/main/runtime/orchestration/db/reset/orchestration-reset.ts'
]
for (const rel of exempt) {
expect(
EXEMPT_PATH_FRAGMENTS.some((fragment) => rel.includes(fragment)),
`${rel} must be exempt`
).toBe(true)
}
})
it('detects the insert forms it claims to detect', () => {
expect(insertPattern('dispatch_contexts').test('INSERT INTO dispatch_contexts (id)')).toBe(true)
expect(
insertPattern('dispatch_contexts').test('INSERT OR REPLACE INTO dispatch_contexts (id)')
).toBe(true)
expect(insertPattern('dispatch_contexts').test('INSERT\n INTO dispatch_contexts')).toBe(true)
expect(insertPattern('dispatch_contexts').test('SELECT * FROM dispatch_contexts')).toBe(false)
expect(insertPattern('dispatch_contexts').test('DELETE FROM dispatch_contexts')).toBe(false)
// Guards against matching the longer sibling table name by prefix.
expect(insertPattern('dispatch_contexts').test('INSERT INTO dispatch_contexts_archive')).toBe(
false
)
})
it('records the evasions this scanner cannot catch', () => {
// Why asserted rather than commented: these are the scanner's known blind
// spots. Centralization is the convention; this test only guards the common
// form. If any of these ever becomes reachable in production SQL, the
// boundary needs an AST-level check instead.
const dynamicTable = 'const t = "dispatch_contexts"; db.prepare(`INSERT INTO ${t} (id)`)'
const splitLiteral = 'db.prepare("INSERT INTO " + "dispatch_contexts (id)")'
const queryBuilder = 'db.insertInto("dispatch_contexts").values({ id })'
expect(insertPattern('dispatch_contexts').test(dynamicTable)).toBe(false)
expect(insertPattern('dispatch_contexts').test(splitLiteral)).toBe(false)
expect(insertPattern('dispatch_contexts').test(queryBuilder)).toBe(false)
})
})
@@ -0,0 +1,144 @@
import type Database from '../../../sqlite/sync-database'
import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL } from './pane-key-match'
/**
* The only place that inserts rows representing a live supervised worker.
*
* Why centralized: nesting depth must be stamped on every such row, and three
* separate modules used to own their own INSERT. A boundary test forbids these
* statements anywhere else, so a new spawn path cannot skip the stamp.
*
* Transaction-neutral on purpose — each caller keeps its own BEGIN IMMEDIATE or
* SAVEPOINT, mutation-receipt write, and companion inserts.
*/
export const DISPATCH_CONTEXT_CLAIM_SQL = `INSERT INTO dispatch_contexts (
id, run_id, task_id, contract_version, launch_token_hash,
assignee_handle, assignee_pane_key, process_incarnation,
status, failure_count, depth, dispatched_at
)
SELECT ?, run_id, id, ?, ?, ?, ?, ?, 'dispatched', ?, ?, datetime('now')
FROM tasks
WHERE id = ? AND status = 'ready'
AND NOT EXISTS (
SELECT 1 FROM dispatch_contexts active
WHERE active.assignee_handle = ?
AND active.status IN ('pending', 'dispatched')
)
AND (
? IS NULL OR NOT EXISTS (
SELECT 1 FROM dispatch_contexts active
WHERE active.assignee_pane_key = ?
AND active.status IN ('pending', 'dispatched')
)
)
AND (
? IS NULL OR NOT EXISTS (
SELECT 1 FROM dispatch_contexts active
WHERE active.assignee_pane_key IS NOT NULL
AND active.status IN ('pending', 'dispatched')
AND instr(active.assignee_pane_key, ':') > 1
AND ${DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL} = ?
)
)`
const STARTING_DISPATCH_CONTEXT_SQL = `INSERT INTO dispatch_contexts (
id, run_id, task_id, contract_version, launch_token_hash, depth, status, dispatched_at
) VALUES (?, ?, ?, ?, ?, ?, 'pending', datetime('now'))`
const REMOTE_DISPATCH_ATTACHMENT_SQL = `INSERT INTO remote_dispatch_attachments (
dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch, depth
) VALUES (?, ?, ?, ?, ?, ?)`
/** Last line of defence: a row that reached here unstamped would read as a root. */
function assertStampedDepth(depth: number): void {
if (!Number.isInteger(depth) || depth < 1) {
throw new Error(
`Refusing to write a live-worker row with depth ${depth}; expected an integer >= 1.`
)
}
}
/** Adopts an existing agent terminal, claiming a ready task atomically. */
export function claimDispatchContextRow(
db: Database.Database,
params: {
id: string
contractVersion: number
launchTokenHash: string | null
assigneeHandle: string
assigneePaneKey: string | null
processIncarnation: string | null
priorFailures: number
depth: number
taskId: string
paneSuffix: string | null
}
): { changes: number | bigint } {
assertStampedDepth(params.depth)
return db
.prepare(DISPATCH_CONTEXT_CLAIM_SQL)
.run(
params.id,
params.contractVersion,
params.launchTokenHash,
params.assigneeHandle,
params.assigneePaneKey,
params.processIncarnation,
params.priorFailures,
params.depth,
params.taskId,
params.assigneeHandle,
params.assigneePaneKey,
params.assigneePaneKey,
params.paneSuffix,
params.paneSuffix
)
}
/** Supervised `worker-start`, including every retry and the federated home side. */
export function insertStartingDispatchContextRow(
db: Database.Database,
params: {
id: string
runId: string
taskId: string
contractVersion: number
launchTokenHash: string | null
depth: number
}
): void {
assertStampedDepth(params.depth)
db.prepare(STARTING_DISPATCH_CONTEXT_SQL).run(
params.id,
params.runId,
params.taskId,
params.contractVersion,
params.launchTokenHash,
params.depth
)
}
/** The worker host's record of a live worker driven by a remote Run home. */
export function insertRemoteDispatchAttachmentRow(
db: Database.Database,
params: {
dispatchId: string
taskId: string
homePeerFingerprint: string
protocolVersion: number
runtimeEpoch: string
/** Propagated from the Run home, not computed here; absent (old client) = 1. */
depth: number
}
): void {
assertStampedDepth(params.depth)
db.prepare(REMOTE_DISPATCH_ATTACHMENT_SQL).run(
params.dispatchId,
params.taskId,
params.homePeerFingerprint,
params.protocolVersion,
params.runtimeEpoch,
params.depth
)
}
@@ -2,6 +2,7 @@ import type { WorkerDispatchState, RemoteDispatchAttachmentRow } from '../../typ
import { OrchestrationError } from '../../orchestration-error'
import { ensureMutationReceiptCapacity } from '../../mutation-receipt-capacity'
import type { OrchestrationDb } from '../orchestration-db'
import { insertRemoteDispatchAttachmentRow } from '../dispatch-row-writer'
export function createRemoteDispatchAttachment(
this: OrchestrationDb,
@@ -11,6 +12,8 @@ export function createRemoteDispatchAttachment(
homePeerFingerprint: string
protocolVersion: number
runtimeEpoch: string
/** Child depth computed by the Run home; absent from an old client = 1 (fails closed). */
depth?: number
mutationReceipt: {
callerFingerprint: string
requestId: string
@@ -54,19 +57,14 @@ export function createRemoteDispatchAttachment(
params.mutationReceipt.payloadHash,
JSON.stringify({ accepted: { dispatchId: params.dispatchId } })
)
this.db
.prepare(
`INSERT INTO remote_dispatch_attachments (
dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch
) VALUES (?, ?, ?, ?, ?)`
)
.run(
params.dispatchId,
params.taskId,
params.homePeerFingerprint,
params.protocolVersion,
params.runtimeEpoch
)
insertRemoteDispatchAttachmentRow(this.db, {
dispatchId: params.dispatchId,
taskId: params.taskId,
homePeerFingerprint: params.homePeerFingerprint,
protocolVersion: params.protocolVersion,
runtimeEpoch: params.runtimeEpoch,
depth: params.depth ?? 1
})
this.db.exec('COMMIT')
return this.getRemoteDispatchAttachment(params.dispatchId) as RemoteDispatchAttachmentRow
} catch (error) {
@@ -4,6 +4,7 @@ import type { DispatchCapabilityMethods } from './dispatch-context/dispatch-capa
import type { DispatchCompletionMethods } from './dispatch-context/dispatch-completion'
import type { DispatchContextStoreMethods } from './dispatch-context/dispatch-context-store'
import type { DispatchLookupMethods } from './dispatch-context/dispatch-lookup'
import type { DispatchDepthMethods } from './dispatch-depth'
import type { WorkerReportSettlementMethods } from './dispatch-context/worker-report-settlement'
import type { FederatedDispatchStoreMethods } from './federation/federated-dispatch-store'
import type { FederationRelayAckMethods } from './federation/federation-relay-ack'
@@ -114,6 +115,7 @@ export type OrchestrationDbMethods = CreateTablesMethods &
DispatchContextStoreMethods &
DispatchCapabilityMethods &
DispatchLookupMethods &
DispatchDepthMethods &
DispatchCompletionMethods &
WorkerReportSettlementMethods &
DecisionGateStoreMethods &
@@ -0,0 +1,28 @@
import type { DispatchContextRow } from '../types'
import type { OrchestrationDb } from './orchestration-db'
/**
* Dispatch as a root coordinator with no nesting cap.
*
* Tests that predate nesting depth care about dispatch behaviour, not the cap;
* this keeps them at their original call shape instead of repeating the same
* creator/maxDepth pair at every site.
*/
export function createRootDispatch(
db: OrchestrationDb,
taskId: string,
assigneeHandle: string,
assigneePaneKey?: string,
launchTokenHash?: string,
processIncarnation?: string
): DispatchContextRow {
return db.createDispatchContext({
taskId,
assigneeHandle,
assigneePaneKey,
launchTokenHash,
processIncarnation,
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER
})
}
@@ -42,17 +42,24 @@ CREATE TABLE IF NOT EXISTS remote_dispatch_attachments (
effects TEXT NOT NULL DEFAULT '[]',
residual_resources TEXT NOT NULL DEFAULT '[]',
to_worker_imported_sequence INTEGER NOT NULL DEFAULT 0,
-- Nesting depth of the worker this attachment represents. Propagated from the
-- Run home; absent from an old client means 1, which fails closed.
depth INTEGER NOT NULL DEFAULT 1,
last_error TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
updated_at TEXT NOT NULL DEFAULT (datetime('now'))
);
-- Why five states: 'start_unknown', 'stopping', and 'stop_unknown' do not
-- establish process exit, and a potentially-live worker must still count as a
-- nesting parent. See docs/reference/ssh-execution-boundary.md.
CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane
ON remote_dispatch_attachments(pane_key)
WHERE state IN ('starting', 'ready');
WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown');
CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane_suffix
ON remote_dispatch_attachments(${REMOTE_ATTACHMENT_PANE_KEY_MATCH_SUFFIX_SQL})
WHERE state IN ('starting', 'ready') AND pane_key IS NOT NULL;
WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown')
AND pane_key IS NOT NULL;
CREATE TABLE IF NOT EXISTS federation_relay_items (
dispatch_id TEXT NOT NULL,
@@ -127,6 +134,9 @@ CREATE TABLE IF NOT EXISTS dispatch_contexts (
last_failure TEXT,
-- Why the process is gone, when Orca could establish it. See TerminalExitCause.
termination_reason TEXT,
-- Nesting depth: a root coordinator's worker is 1, its worker's worker is 2.
-- Defaults to 1 so an unstamped row fails closed rather than reading as a root.
depth INTEGER NOT NULL DEFAULT 1,
dispatched_at TEXT,
completed_at TEXT,
created_at TEXT NOT NULL DEFAULT (datetime('now')),
@@ -1,8 +1,11 @@
import { migrateMutationReceiptCapacity } from '../../mutation-receipt-capacity'
import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL } from '../pane-key-match'
import {
DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL,
REMOTE_ATTACHMENT_PANE_KEY_MATCH_SUFFIX_SQL
} from '../pane-key-match'
import type { OrchestrationDb } from '../orchestration-db'
export function applySchemaMigrationsV13ToV29(this: OrchestrationDb, current: number): void {
export function applySchemaMigrationsV13ToV30(this: OrchestrationDb, current: number): void {
if (current < 13 && !this.hasColumn('worker_dispatches', 'runtime_epoch')) {
this.db.exec('ALTER TABLE worker_dispatches ADD COLUMN runtime_epoch TEXT')
}
@@ -157,6 +160,29 @@ export function applySchemaMigrationsV13ToV29(this: OrchestrationDb, current: nu
if (current < 29 && !this.hasColumn('dispatch_contexts', 'termination_reason')) {
this.db.exec('ALTER TABLE dispatch_contexts ADD COLUMN termination_reason TEXT')
}
if (current < 30) {
if (!this.hasColumn('dispatch_contexts', 'depth')) {
this.db.exec('ALTER TABLE dispatch_contexts ADD COLUMN depth INTEGER NOT NULL DEFAULT 1')
}
if (!this.hasColumn('remote_dispatch_attachments', 'depth')) {
this.db.exec(
'ALTER TABLE remote_dispatch_attachments ADD COLUMN depth INTEGER NOT NULL DEFAULT 1'
)
}
// Why drop first: CREATE INDEX IF NOT EXISTS cannot widen an existing
// partial index predicate, and these two covered only starting/ready.
this.db.exec(`
DROP INDEX IF EXISTS idx_remote_dispatch_attachments_active_pane;
DROP INDEX IF EXISTS idx_remote_dispatch_attachments_active_pane_suffix;
CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane
ON remote_dispatch_attachments(pane_key)
WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown');
CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane_suffix
ON remote_dispatch_attachments(${REMOTE_ATTACHMENT_PANE_KEY_MATCH_SUFFIX_SQL})
WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown')
AND pane_key IS NOT NULL;
`)
}
this.db.exec(`
CREATE INDEX IF NOT EXISTS idx_dispatch_assignee_pane_leaf
ON dispatch_contexts(${DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL})
@@ -1,7 +1,7 @@
import { resolveOrchestrationMigrationStartVersion } from '../../orchestration-schema-version-skew'
import { SCHEMA_VERSION } from '../contract-constants'
import type { OrchestrationDb } from '../orchestration-db'
import { applySchemaMigrationsV13ToV29 } from './migrate-v13-v29'
import { applySchemaMigrationsV13ToV30 } from './migrate-v13-v30'
import { applySchemaMigrationsV2ToV12 } from './migrate-v2-v12'
// Why: CREATE TABLE IF NOT EXISTS won't alter existing DBs; migrate in a txn that bumps user_version only on success (atomic all-or-nothing).
@@ -15,7 +15,7 @@ export function migrate(this: OrchestrationDb): void {
this.db.exec('BEGIN IMMEDIATE')
try {
applySchemaMigrationsV2ToV12.call(this, current)
applySchemaMigrationsV13ToV29.call(this, current)
applySchemaMigrationsV13ToV30.call(this, current)
this.db.pragma(`user_version = ${SCHEMA_VERSION}`)
this.db.exec('COMMIT')
} catch (err) {
@@ -4,6 +4,8 @@ import { ensureMutationReceiptCapacity } from '../../mutation-receipt-capacity'
import { CURRENT_CONTRACT_VERSION } from '../contract-constants'
import { generateId } from '../generated-id'
import type { OrchestrationDb } from '../orchestration-db'
import { insertStartingDispatchContextRow } from '../dispatch-row-writer'
import type { DispatchCreator } from '../dispatch-depth'
export function createStartingWorkerDispatch(
this: OrchestrationDb,
@@ -25,6 +27,9 @@ export function createStartingWorkerDispatch(
method: string
payloadHash: string
}
/** Who is dispatching, for nesting depth. Required so a new caller must decide. */
creator: DispatchCreator
maxDepth: number
}
): { dispatch: DispatchContextRow; worker: WorkerDispatchRow } {
this.db.exec('BEGIN IMMEDIATE')
@@ -95,13 +100,14 @@ export function createStartingWorkerDispatch(
params.mutationReceipt.requestId
)
}
this.db
.prepare(
`INSERT INTO dispatch_contexts (
id, run_id, task_id, contract_version, launch_token_hash, status, dispatched_at
) VALUES (?, ?, ?, ?, ?, 'pending', datetime('now'))`
)
.run(id, task.run_id, task.id, CURRENT_CONTRACT_VERSION, params.launchTokenHash ?? null)
insertStartingDispatchContextRow(this.db, {
id,
runId: task.run_id,
taskId: task.id,
contractVersion: CURRENT_CONTRACT_VERSION,
launchTokenHash: params.launchTokenHash ?? null,
depth: this.resolveChildDispatchDepth(params.creator, params.maxDepth)
})
this.db
.prepare(
`INSERT INTO worker_dispatches (
@@ -1,12 +1,13 @@
import { describe, expect, it } from 'vitest'
import type Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('dispatch failure idempotency', () => {
it('counts an active dispatch failure only once', () => {
const db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
expect(db.failDispatch(dispatch.id, 'exit')?.failure_count).toBe(1)
const duplicate = db.failDispatch(dispatch.id, 'duplicate escalation')
@@ -19,7 +20,7 @@ describe('dispatch failure idempotency', () => {
it('does not overwrite a completed dispatch', () => {
const db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
db.completeDispatch(dispatch.id)
const lateFailure = db.failDispatch(dispatch.id, 'late exit')
@@ -33,7 +34,7 @@ describe('dispatch failure idempotency', () => {
const db = new OrchestrationDb(':memory:')
const sqlite = (db as unknown as { db: Database.Database }).db
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
sqlite.exec(`
CREATE TRIGGER reject_task_failure_update
BEFORE UPDATE ON tasks WHEN OLD.id = '${task.id}'
@@ -128,6 +128,8 @@ describe('federation relay parsing', () => {
})
const task = db.createTask({ spec: 'Remote work', runId: run.id })
const { dispatch } = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
federation: {
@@ -196,6 +198,8 @@ describe('federation relay acknowledgments', () => {
})
const task = db.createTask({ spec: 'Remote work', runId: run.id })
const { dispatch } = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
federation: {
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it } from 'vitest'
import { OrchestrationDb } from './db'
import { reconcileLifecycleMessage } from './lifecycle-reconciliation'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('lifecycle reconciliation', () => {
let db: OrchestrationDb
@@ -10,7 +11,7 @@ describe('lifecycle reconciliation', () => {
it('rejects handle churn when neither side has stable pane identity', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_before_restart')
const dispatch = createRootDispatch(db, task.id, 'term_before_restart')
const logs: string[] = []
const message = db.insertMessage({
from: 'term_after_restart',
@@ -37,7 +38,7 @@ describe('lifecycle reconciliation', () => {
it('completes worker_done from the dispatched pane after a handle remint', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_before_restart', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_before_restart', `tab_w:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_after_restart',
to: 'term_coordinator',
@@ -54,7 +55,7 @@ describe('lifecycle reconciliation', () => {
it('fails both the dispatch and task from an authenticated failed worker report', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_worker',
to: 'term_coordinator',
@@ -87,7 +88,7 @@ describe('lifecycle reconciliation', () => {
it('replays an identical terminal outcome without mutating settled state', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const makeMessage = () =>
db.insertMessage({
from: 'term_worker',
@@ -144,7 +145,7 @@ describe('lifecycle reconciliation', () => {
const task = db.createTask({ spec: 'work' })
// Dispatch recorded the post-break-out pane key; the worker shell still
// holds the spawn-time key with the old tab id.
const dispatch = db.createDispatchContext(task.id, 'term_before_restart', `tab_new:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_before_restart', `tab_new:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_after_restart',
to: 'term_coordinator',
@@ -161,7 +162,7 @@ describe('lifecycle reconciliation', () => {
it('rejects mismatched opaque pane keys instead of treating them as legacy', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_reminted',
to: 'term_coordinator',
@@ -178,7 +179,7 @@ describe('lifecycle reconciliation', () => {
it('rejects worker_done from a foreign pane that claims the assignee handle', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_owner',
to: 'term_coordinator',
@@ -221,7 +222,7 @@ describe('lifecycle reconciliation', () => {
it('does not let a caller-supplied rejection marker turn completion into success', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_worker',
to: 'term_coordinator',
@@ -250,7 +251,7 @@ describe('lifecycle reconciliation', () => {
it('rejects a coordinator completion for a pane-bound dispatch', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`)
const message = db.insertMessage({
from: 'term_coordinator',
to: 'term_coordinator',
@@ -269,7 +270,7 @@ describe('lifecycle reconciliation', () => {
it('uses exact handle equality only for a legacy dispatch without a pane key', () => {
db = new OrchestrationDb(':memory:')
const acceptedTask = db.createTask({ spec: 'legacy work' })
const acceptedDispatch = db.createDispatchContext(acceptedTask.id, 'term_legacy')
const acceptedDispatch = createRootDispatch(db, acceptedTask.id, 'term_legacy')
const accepted = db.insertMessage({
from: 'term_legacy',
to: 'term_coordinator',
@@ -284,7 +285,7 @@ describe('lifecycle reconciliation', () => {
expect(reconcileLifecycleMessage(db, accepted).action).toBe('completed')
const rejectedTask = db.createTask({ spec: 'other legacy work' })
const rejectedDispatch = db.createDispatchContext(rejectedTask.id, 'term_other_legacy')
const rejectedDispatch = createRootDispatch(db, rejectedTask.id, 'term_other_legacy')
const rejected = db.insertMessage({
from: 'term_foreign',
to: 'term_coordinator',
@@ -307,7 +308,7 @@ describe('lifecycle reconciliation', () => {
db = new OrchestrationDb(':memory:')
const parent = db.createTask({ spec: 'parent' })
const child = db.createTask({ spec: 'child', deps: [parent.id] })
const dispatch = db.createDispatchContext(parent.id, 'term_worker', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, parent.id, 'term_worker', `tab_w:${LEAF_A}`)
const payload = JSON.stringify({
taskId: parent.id,
dispatchId: dispatch.id,
@@ -343,7 +344,7 @@ describe('lifecycle reconciliation', () => {
it('does not let a foreign replay overwrite an authorized completion', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`)
const payload = JSON.stringify({
taskId: task.id,
dispatchId: dispatch.id,
@@ -378,7 +379,7 @@ describe('lifecycle reconciliation', () => {
it('surfaces worker_done sent from a different pane as rejected', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const logs: string[] = []
const message = db.insertMessage({
from: 'term_other_worker',
@@ -401,7 +402,7 @@ describe('lifecycle reconciliation', () => {
it('surfaces a heartbeat sent from a different pane without recording liveness', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const heartbeat = db.insertMessage({
from: 'term_other_worker',
to: 'term_coordinator',
@@ -435,7 +436,7 @@ describe('lifecycle reconciliation', () => {
it('surfaces a foreign heartbeat that claims the assignee handle', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`)
const heartbeat = db.insertMessage({
from: 'term_owner',
to: 'term_coordinator',
@@ -455,7 +456,7 @@ describe('lifecycle reconciliation', () => {
it('records a heartbeat whose pane key drifted only in the tab half', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_new:${LEAF_A}`)
const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_new:${LEAF_A}`)
const heartbeat = db.insertMessage({
from: 'term_owner',
to: 'term_coordinator',
@@ -475,9 +476,9 @@ describe('lifecycle reconciliation', () => {
it('suppresses same-dispatch heartbeats once worker_done is reconciled', () => {
db = new OrchestrationDb(':memory:')
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const otherTask = db.createTask({ spec: 'other work' })
const otherDispatch = db.createDispatchContext(otherTask.id, 'term_other')
const otherDispatch = createRootDispatch(db, otherTask.id, 'term_other')
const insertHeartbeat = (dispatchId: string, from: string) =>
db.insertMessage({
from,
@@ -5,6 +5,7 @@ import { DISPATCH_CIRCUIT_BREAK_FAILURES } from './db/dispatch-context/dispatch-
import { makePaneKey } from '../../../shared/stable-pane-id'
import { getDefaultWorkspaceSession } from '../../../shared/constants'
import type { WorkspaceSessionState } from '../../../shared/workspace-session-state-types'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
// STA-4604: failActiveDispatchOnExit fails the dispatch on worker PTY exit but used to
// gate the "Agent exited unexpectedly" escalation on the legacy coordinator_runs table.
@@ -130,7 +131,7 @@ async function gradeWorkerExit(
db.createCoordinatorRun({ spec: 'legacy coordinator loop', coordinatorHandle })
}
const task = db.createTask({ spec: 'do the work', runId })
const dispatch = db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY)
const dispatch = createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY)
runtime.setOrchestrationDb(db as never)
runtime.onPtyExit(WORKER_PTY_ID, 137)
@@ -239,7 +240,7 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => {
coordinatorPaneKey: COORDINATOR_PANE_KEY
})
const task = db.createTask({ spec: 'do the work', runId: run.id })
db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY)
createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY)
runtime.setOrchestrationDb(db as never)
const waiting = runtime.waitForMessage(`run:${run.id}`, {
@@ -281,7 +282,7 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => {
coordinatorPaneKey: makePaneKey('tab-other', '33333333-3333-4333-8333-333333333333')
})
const task = db.createTask({ spec: 'owned work', runId: ownRun.id })
db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY)
createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY)
runtime.setOrchestrationDb(db as never)
runtime.onPtyExit(WORKER_PTY_ID, 137)
@@ -306,7 +307,12 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => {
coordinatorPaneKey: COORDINATOR_PANE_KEY
})
const task = db.createTask({ spec: 'supervised work', runId: run.id })
const started = db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
db.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: workerHandle,
@@ -350,10 +356,10 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => {
const task = db.createTask({ spec: 'repeatedly failing work', runId: run.id })
// Burn the breaker down to its last life so this exit is the one that trips it.
for (let attempt = 1; attempt < DISPATCH_CIRCUIT_BREAK_FAILURES; attempt += 1) {
const previous = db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY)
const previous = createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY)
db.failDispatch(previous.id, `attempt ${attempt}`, { workerProcessExited: true })
}
const dispatch = db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY)
const dispatch = createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY)
runtime.setOrchestrationDb(db as never)
runtime.onPtyExit(WORKER_PTY_ID, 137)
@@ -414,7 +420,7 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => {
coordinatorPaneKey: COORDINATOR_PANE_KEY
})
const task = db.createTask({ spec: 'work outliving its coordinator', runId: run.id })
db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY)
createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY)
// Rebinding the pane to a newer Run clears the old Run's coordinator_handle.
db.createRun({
objective: 'newer run on the same coordinator pane',
@@ -0,0 +1,114 @@
import { mkdtempSync, rmSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { resolveOrchestrationMigrationStartVersion } from './orchestration-schema-version-skew'
import { SCHEMA_VERSION } from './db/contract-constants'
/**
* Backfilling to 1 rather than 0 is the whole point: every pre-v30 row belongs to
* a worker that was already dispatched, so reading it as a root coordinator would
* hand every in-flight worker a free generation of sub-workers at upgrade.
*/
describe('nested worker depth migration (v30)', () => {
let db: OrchestrationDb | undefined
let tempDir: string | undefined
afterEach(() => {
db?.close()
db = undefined
if (tempDir) {
rmSync(tempDir, { recursive: true, force: true })
tempDir = undefined
}
})
function createV29Database(): string {
tempDir = mkdtempSync(join(tmpdir(), 'orca-nested-depth-migration-'))
const dbPath = join(tempDir, 'orchestration.db')
const fresh = new OrchestrationDb(dbPath)
fresh.close()
const oldDb = new Database(dbPath)
oldDb.exec('ALTER TABLE dispatch_contexts DROP COLUMN depth')
oldDb.exec('ALTER TABLE remote_dispatch_attachments DROP COLUMN depth')
oldDb.pragma('user_version = 29')
oldDb
.prepare(
`INSERT INTO dispatch_contexts (id, run_id, task_id, contract_version, status)
VALUES ('ctx_inflight', 'run_legacy', 'task_legacy', 1, 'dispatched')`
)
.run()
oldDb
.prepare(
`INSERT INTO remote_dispatch_attachments
(dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch, state)
VALUES ('ctx_remote', 'task_remote', 'peer', 1, 'epoch', 'ready')`
)
.run()
oldDb.close()
return dbPath
}
it('backfills in-flight rows to depth 1, not 0', () => {
const dbPath = createV29Database()
db = new OrchestrationDb(dbPath)
const sqlite = (db as unknown as { db: Database.Database }).db
expect(sqlite.pragma('user_version', { simple: true })).toBe(SCHEMA_VERSION)
expect(
sqlite.prepare("SELECT depth FROM dispatch_contexts WHERE id = 'ctx_inflight'").get()
).toEqual({ depth: 1 })
expect(
sqlite
.prepare("SELECT depth FROM remote_dispatch_attachments WHERE dispatch_id = 'ctx_remote'")
.get()
).toEqual({ depth: 1 })
})
it('leaves an upgraded in-flight worker unable to spawn at the default cap', () => {
const dbPath = createV29Database()
db = new OrchestrationDb(dbPath)
const sqlite = (db as unknown as { db: Database.Database }).db
sqlite
.prepare(
"UPDATE dispatch_contexts SET assignee_handle = 'term_upgraded' WHERE id = 'ctx_inflight'"
)
.run()
const task = db.createTask({ spec: 'post-upgrade nesting attempt' })
expect(() =>
db!.createDispatchContext({
taskId: task.id,
assigneeHandle: 'term_sub',
creator: { kind: 'terminal', handle: 'term_upgraded' },
maxDepth: 1
})
).toThrow(/depth 2 \(max 1\)/)
})
it('does not mistake a real v29 database for a broken v30 one', () => {
// An unconditional column check here would report the schema incomplete and
// replay migrations from v6 instead of starting at 29.
const dbPath = createV29Database()
const oldDb = new Database(dbPath)
expect(resolveOrchestrationMigrationStartVersion(oldDb, 29, SCHEMA_VERSION)).toBe(29)
oldDb.close()
})
it('widens the attachment pane indexes to the potentially-live states', () => {
const dbPath = createV29Database()
db = new OrchestrationDb(dbPath)
const sqlite = (db as unknown as { db: Database.Database }).db
const sql = sqlite
.prepare(
"SELECT sql FROM sqlite_master WHERE type = 'index' AND name = 'idx_remote_dispatch_attachments_active_pane'"
)
.get() as { sql: string }
for (const state of ['start_unknown', 'stopping', 'stop_unknown']) {
expect(sql.sql).toContain(state)
}
})
})
@@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest'
import type Database from '../../sqlite/sync-database'
import SyncDatabase from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
const LEGACY_COORDINATOR_HANDLE = 'term_legacy_coord'
const LEGACY_COORDINATOR_PANE = 'tab_coord:44444444-4444-4444-8444-444444444444'
@@ -49,7 +50,7 @@ function createAdoptedFixture(options: { settleWork: boolean }): AdoptedFixture
spec: 'legacy assignment',
createdByTerminalHandle: LEGACY_COORDINATOR_HANDLE
})
const dispatch = before.createDispatchContext(task.id, LEGACY_WORKER_HANDLE, LEGACY_WORKER_PANE)
const dispatch = createRootDispatch(before, task.id, LEGACY_WORKER_HANDLE, LEGACY_WORKER_PANE)
const recovery = before.insertMessage({
from: LEGACY_WORKER_HANDLE,
to: LEGACY_COORDINATOR_HANDLE,
@@ -1,6 +1,7 @@
import { afterEach, describe, expect, it } from 'vitest'
import type Database from '../../sqlite/sync-database'
import { DISPATCH_CONTEXT_CLAIM_SQL, OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
const CREATOR_PANE = 'tab-creator:11111111-1111-4111-8111-111111111111'
const CREATOR_PROCESS = 'pty-creator:incarnation-a'
@@ -84,13 +85,7 @@ describe('creator authority lookup performance', () => {
coordinatorPaneKey: 'tab-coordinator:22222222-2222-4222-8222-222222222222'
})
const creatorTask = db.createTask({ spec: 'creator', runId: run.id })
db.createDispatchContext(
creatorTask.id,
'term-creator',
CREATOR_PANE,
undefined,
CREATOR_PROCESS
)
createRootDispatch(db, creatorTask.id, 'term-creator', CREATOR_PANE, undefined, CREATOR_PROCESS)
const workerTask = db.createTask({
spec: 'worker',
runId: run.id,
@@ -145,7 +140,8 @@ describe('creator authority lookup performance', () => {
)
.run(retainedDispatchCount, run.id)
const creatorTask = db.createTask({ spec: 'creator', runId: run.id })
const creatorDispatch = db.createDispatchContext(
const creatorDispatch = createRootDispatch(
db,
creatorTask.id,
'term-creator',
CREATOR_PANE,
@@ -173,9 +169,14 @@ describe('creator authority lookup performance', () => {
const elapsedMs = performance.now() - startedAt
const competingTask = db.createTask({ spec: 'competing creator', runId: run.id })
expect(() => db!.createDispatchContext(competingTask.id, 'term-creator')).toThrow(
`Terminal term-creator already has an active dispatch (${creatorDispatch.id}`
)
expect(() =>
db!.createDispatchContext({
taskId: competingTask.id,
assigneeHandle: 'term-creator',
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER
})
).toThrow(`Terminal term-creator already has an active dispatch (${creatorDispatch.id}`)
expect(elapsedMs).toBeLessThan(200)
}
)
@@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest'
import Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { SCHEMA_VERSION } from './db/contract-constants'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
const MUTATION_RECEIPT_MAX_ROWS = 10_000
@@ -155,6 +156,8 @@ describe('OrchestrationDb bounded mutation receipts', () => {
expect(() =>
db!.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
mutationReceipt: {
@@ -224,7 +227,7 @@ describe('OrchestrationDb dispatch assignee index migration', () => {
const dbPath = join(tempDir, 'orchestration.db')
db = new OrchestrationDb(dbPath)
const task = db.createTask({ spec: 'indexed lookup' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
db.close()
db = undefined
@@ -299,7 +302,7 @@ describe('OrchestrationDb dispatch assignee index migration', () => {
createdByProcessIncarnation: 'pty_creator:incarnation-a',
createdByRunGeneration: run.consumer_generation
})
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
db.close()
db = undefined
@@ -157,7 +157,13 @@ describe('OrchestrationDb legacy contract storage', () => {
createdByTerminalHandle: 'term_legacy_coord'
})
const dispatch = db!.createDispatchContext(task.id, 'term_legacy_coord', 'tab_mixed:leaf_mixed')
const dispatch = db!.createDispatchContext({
taskId: task.id,
assigneeHandle: 'term_legacy_coord',
assigneePaneKey: 'tab_mixed:leaf_mixed',
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER
})
expect(dispatch.contract_version).toBe(CURRENT_CONTRACT_VERSION)
expect(db!.getRunMailboxOwnerIdsForHandle('term_legacy_coord')).toEqual([])
@@ -749,7 +755,12 @@ describe('OrchestrationDb legacy contract storage', () => {
).toThrow(/different answer/)
const currentTask = db!.createTask({ runId: state.adoptedRunId, spec: 'current retry' })
const currentDispatch = db!.createDispatchContext(currentTask.id, 'term_current_retry')
const currentDispatch = db!.createDispatchContext({
taskId: currentTask.id,
assigneeHandle: 'term_current_retry',
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER
})
const currentQuestion = db!.createQuestion({
runId: state.adoptedRunId,
dispatchId: currentDispatch.id,
@@ -3,6 +3,7 @@ import { tmpdir } from 'node:os'
import { join } from 'node:path'
import Database from '../../sqlite/sync-database'
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
export type LegacyStorageCutoverFixture = {
dbPath: string
@@ -38,7 +39,8 @@ export function createLegacyStorageCutoverFixture(): {
coordinatorHandle: 'term_unrelated_coord',
coordinatorPaneKey: 'tab_unrelated:55555555-5555-4555-8555-555555555555'
})
const currentDispatch = first.createDispatchContext(
const currentDispatch = createRootDispatch(
first,
currentTask.id,
'term_current_worker',
'tab_current:22222222-2222-4222-9222-222222222222',
@@ -55,7 +57,8 @@ export function createLegacyStorageCutoverFixture(): {
spec: 'legacy',
createdByTerminalHandle: 'term_legacy_coord'
})
first.createDispatchContext(
createRootDispatch(
first,
legacyTask.id,
'term_legacy_worker',
'tab_legacy:33333333-3333-4333-8333-333333333333'
@@ -65,7 +68,8 @@ export function createLegacyStorageCutoverFixture(): {
question: 'Retained gate?'
})
first.resolveGate(legacyGate.id, 'continue')
const retryDispatch = first.createDispatchContext(
const retryDispatch = createRootDispatch(
first,
legacyTask.id,
'term_legacy_worker',
'tab_legacy:33333333-3333-4333-8333-333333333333'
@@ -1,5 +1,6 @@
import { afterEach, describe, expect, it } from 'vitest'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('OrchestrationDb mutation and question state', () => {
let db: OrchestrationDb | undefined
@@ -93,7 +94,7 @@ describe('OrchestrationDb mutation and question state', () => {
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
})
const task = d.createTask({ spec: 'ask', runId: run.id })
const dispatch = d.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(d, task.id, 'term_worker')
const created = d.createQuestion({
runId: run.id,
dispatchId: dispatch.id,
@@ -144,7 +145,7 @@ describe('OrchestrationDb mutation and question state', () => {
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
})
const task = d.createTask({ spec: 'ask', runId: run.id })
const dispatch = d.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(d, task.id, 'term_worker')
const created = d.createQuestion({
runId: run.id,
dispatchId: dispatch.id,
@@ -15,6 +15,8 @@ describe('OrchestrationDb reset scopes', () => {
})
const task = db.createTask({ spec: 'work', runId: run.id })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: { worktree: 'current' },
runtimeEpoch: 'runtime_1',
@@ -3,6 +3,7 @@ import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('OrchestrationDb Run state', () => {
let db: OrchestrationDb | undefined
@@ -172,7 +173,7 @@ describe('OrchestrationDb Run state', () => {
coordinatorPaneKey: 'tab_other:22222222-2222-4222-9222-222222222222'
})
const task = d.createTask({ spec: 'work', runId: runB.id })
const dispatch = d.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(d, task.id, 'term_worker')
const mismatched = d.insertMessage({
from: 'worker',
to: `dispatch:${dispatch.id}`,
@@ -285,7 +286,7 @@ describe('OrchestrationDb Run state', () => {
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
})
const task = d.createTask({ spec: 'work', runId: run.id })
const dispatch = d.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(d, task.id, 'term_worker')
const message = d.insertMessage({
runId: run.id,
from: 'term_worker',
@@ -26,7 +26,9 @@ const POST_V6_COLUMNS = [
] as const
const VERSIONED_POST_V6_COLUMNS = [
{ version: 27, table: 'federated_dispatches', column: 'to_home_acknowledged_sequence' }
{ version: 27, table: 'federated_dispatches', column: 'to_home_acknowledged_sequence' },
{ version: 30, table: 'dispatch_contexts', column: 'depth' },
{ version: 30, table: 'remote_dispatch_attachments', column: 'depth' }
] as const
const POST_V6_INDEXES = [
@@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest'
import Database from '../../sqlite/sync-database'
import { LEGACY_CONTRACT_VERSION, LEGACY_RUN_ID, OrchestrationDb } from './db'
import { resolveOrchestrationMigrationStartVersion } from './orchestration-schema-version-skew'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('OrchestrationDb version-skew migration', () => {
let db: OrchestrationDb | undefined
@@ -158,7 +159,7 @@ describe('OrchestrationDb version-skew migration', () => {
coordinatorPaneKey: 'tab_v2:leaf_coord'
})
const task = db.createTask({ spec: 'reply with ack', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_worker_v2', 'tab_v2:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker_v2', 'tab_v2:leaf_worker')
const question = db.createQuestion({
runId: run.id,
dispatchId: dispatch.id,
@@ -17,6 +17,8 @@ describe('OrchestrationDb worker Dispatch state', () => {
const d = createDb()
const task = d.createTask({ spec: 'worker' })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: { topology: 'current', agent: 'codex' }
})
@@ -57,7 +59,12 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('retains an active supervised worker terminal', () => {
const d = createDb()
const task = d.createTask({ spec: 'retain active worker' })
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_worker',
@@ -81,6 +88,8 @@ describe('OrchestrationDb worker Dispatch state', () => {
const d = createDb()
const task = d.createTask({ spec: 'recover missing worker' })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: { topology: 'current', agent: 'codex' }
})
@@ -125,6 +134,8 @@ describe('OrchestrationDb worker Dispatch state', () => {
}
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: { topology: 'current' },
mutationReceipt
@@ -146,6 +157,8 @@ describe('OrchestrationDb worker Dispatch state', () => {
expect(() =>
d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: 'task_missing',
startOptions: {},
mutationReceipt: {
@@ -162,7 +175,12 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('fails a composed start without losing residual resource receipts', () => {
const d = createDb()
const task = d.createTask({ spec: 'worker' })
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.recordWorkerStage({
dispatchId: started.dispatch.id,
stage: 'terminal_created',
@@ -182,9 +200,16 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('allows retry only from the Task current terminal Dispatch', () => {
const d = createDb()
const task = d.createTask({ spec: 'retry current' })
const first = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const first = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.failWorkerStart(first.dispatch.id, 'agent_readiness', 'first failed')
const second = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
retryOf: first.dispatch.id,
startOptions: {}
@@ -193,6 +218,8 @@ describe('OrchestrationDb worker Dispatch state', () => {
expect(() =>
d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
retryOf: first.dispatch.id,
startOptions: {}
@@ -200,6 +227,8 @@ describe('OrchestrationDb worker Dispatch state', () => {
).toThrow('cannot retry')
expect(
d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
retryOf: second.dispatch.id,
startOptions: {}
@@ -210,9 +239,16 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('treats abandon of a superseded Dispatch as a no-op', () => {
const d = createDb()
const task = d.createTask({ spec: 'stale abandon' })
const first = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const first = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.failWorkerStart(first.dispatch.id, 'agent_readiness', 'first failed')
const second = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
retryOf: first.dispatch.id,
startOptions: {}
@@ -248,7 +284,12 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('lets the stop fence win before a late worker completion', () => {
const d = createDb()
const task = d.createTask({ spec: 'race' })
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_worker',
@@ -276,7 +317,12 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('allows explicit stop recovery from uncertain local and remote starts', () => {
const d = createDb()
const task = d.createTask({ spec: 'uncertain local start' })
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.markWorkerStartUnknown(started.dispatch.id, 'agent_readiness', 'connection lost')
expect(d.beginWorkerStop(started.dispatch.id, 'runtime_test')).toMatchObject({
@@ -359,7 +405,12 @@ describe('OrchestrationDb worker Dispatch state', () => {
it('returns already-settled when completion wins before stop', () => {
const d = createDb()
const task = d.createTask({ spec: 'race' })
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = d.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
d.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_worker',
+4
View File
@@ -205,6 +205,8 @@ export type RemoteDispatchAttachmentRow = {
effects: string
residual_resources: string
to_worker_imported_sequence: number
/** Nesting depth propagated from the Run home; 1 when an old client omitted it. */
depth: number
last_error: string | null
created_at: string
updated_at: string
@@ -278,6 +280,8 @@ export type DispatchContextRow = {
/** Why the dispatch ended, when Orca could establish it — `operator_close`,
* `signaled`, `exited`, `unknown`. Null on rows written before STA-4603. */
termination_reason: TerminalExitCause['kind'] | null
/** Nesting depth; a root coordinator's worker is 1. Never 0 on a persisted row. */
depth: number
dispatched_at: string | null
completed_at: string | null
created_at: string
@@ -4,6 +4,7 @@ import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness'
import type { OrchestrationDb } from '../../orchestration/db'
import type { OrcaRuntimeService } from '../../orca-runtime'
import { ORCHESTRATION_ASK_MAX_TIMEOUT_MS } from '../../../../shared/orchestration-ask-timeout'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
describe('orchestration RPC methods', () => {
const h = createOrchestrationRpcHarness()
@@ -59,7 +60,7 @@ describe('orchestration RPC methods', () => {
it('records one idempotent answer from the current Run consumer', async () => {
setup()
const task = db.createTask({ spec: 'question work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const created = db.createQuestion({
runId: activeRunId!,
dispatchId: dispatch.id,
@@ -103,7 +104,7 @@ describe('orchestration RPC methods', () => {
describe('orchestration.ask', () => {
function createAskingDispatch(handle = 'term_worker') {
const task = db.createTask({ spec: 'question work' })
const dispatch = db.createDispatchContext(task.id, handle)
const dispatch = createRootDispatch(db, task.id, handle)
return { task, dispatch }
}
@@ -4,6 +4,7 @@ import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness'
import type { OrchestrationDb } from '../../orchestration/db'
import { reconcileLifecycleMessage } from '../../orchestration/lifecycle-reconciliation'
import type { OrcaRuntimeService } from '../../orca-runtime'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
describe('orchestration RPC methods', () => {
const h = createOrchestrationRpcHarness()
@@ -28,7 +29,7 @@ describe('orchestration RPC methods', () => {
describe('orchestration.check', () => {
function createDispatchedTask(assigneeHandle = 'term_worker', assigneePaneKey?: string) {
const task = db.createTask({ spec: 'manual check work' })
const dispatch = db.createDispatchContext(task.id, assigneeHandle, assigneePaneKey)
const dispatch = createRootDispatch(db, task.id, assigneeHandle, assigneePaneKey)
return { task, dispatch }
}
@@ -432,9 +433,9 @@ describe('orchestration RPC methods', () => {
it('does not complete worker_done for a stale inactive dispatch', async () => {
setup()
const task = db.createTask({ spec: 'retry-sensitive work' })
const staleDispatch = db.createDispatchContext(task.id, 'term_old')
const staleDispatch = createRootDispatch(db, task.id, 'term_old')
db.failDispatch(staleDispatch.id, 'retry elsewhere')
const activeDispatch = db.createDispatchContext(task.id, 'term_current')
const activeDispatch = createRootDispatch(db, task.id, 'term_current')
insertWorkerDone({
from: 'term_old',
taskId: task.id,
@@ -66,6 +66,65 @@ describe('orchestration RPC methods', () => {
})
}
it('rejects a declared caller that disagrees with complete attested evidence', async () => {
setup()
mockCurrentWorkerStart()
vi.mocked(runtime.getTerminalPaneKey).mockImplementation((handle) =>
handle === 'term_coord' || handle === 'term_other'
? coordinatorPaneKey
: handle === 'term_worker'
? 'tab_worker:leaf_worker'
: null
)
const attestedEvidence = {
terminalHandle: 'term_attested',
paneKey: 'tab_attested:leaf_attested',
launchToken: 'attested-launch-token'
} as const
vi.spyOn(runtime, 'verifyOrchestrationCompatibilityCaller').mockReturnValue({
terminalHandle: attestedEvidence.terminalHandle,
paneKey: attestedEvidence.paneKey,
processIncarnation: 'runtime_test:attested:1',
launchTokenHash: 'attested-launch-token-hash',
hostScope: { kind: 'local', hostId: 'local' }
})
ctx = { ...ctx, orchestrationCompatibilityEvidence: attestedEvidence }
const task = db.createTask({ spec: 'mismatched caller' })
await expect(
call('orchestration.workerStart', {
task: task.id,
from: 'term_other',
agent: 'codex'
})
).rejects.toMatchObject({ code: 'consumer_fenced' })
expect(db.getDispatchContext(task.id)).toBeUndefined()
})
it('deliberately permits present but unverifiable restored-terminal evidence', async () => {
setup()
mockCurrentWorkerStart()
// Restored/adopted terminals have no launch token, so verification returns null; this
// fail-open is deliberate compatibility behavior, not an oversight.
const task = db.createTask({ spec: 'restored caller limitation' })
ctx = {
...ctx,
orchestrationCompatibilityEvidence: {
terminalHandle: 'term_worker',
paneKey: 'tab_worker:leaf_worker'
}
}
const result = (await call('orchestration.workerStart', {
task: task.id,
from: 'term_coord',
agent: 'codex'
})) as { state: string }
expect(result.state).toBe('ready')
expect(db.getDispatchContext(task.id)).toBeDefined()
})
it('starts a fresh agent in the coordinator current worktree', async () => {
setup()
mockCurrentWorkerStart()
@@ -0,0 +1,27 @@
import type { DispatchCreator } from '../../orchestration/db/dispatch-depth'
import type { OrcaRuntimeService } from '../../orca-runtime'
/**
* Identify a CLI caller for nesting-depth purposes.
*
* Pane key and process incarnation come from the runtime's dispatch authority
* rather than the caller's params: remote attachment matching needs the exact
* incarnation, and a caller cannot be trusted to report its own.
*/
export function resolveDispatchCreator(
runtime: OrcaRuntimeService,
callerHandle: string | undefined
): DispatchCreator {
if (!callerHandle) {
// No declared caller means no resolvable parent. Depth 0 is the same answer
// the pre-existing Run-binding check already gives this case.
return { kind: 'system' }
}
const authority = runtime.getOrchestrationDispatchAuthority?.(callerHandle)
return {
kind: 'terminal',
handle: callerHandle,
paneKey: authority?.paneKey ?? runtime.getTerminalPaneKey(callerHandle) ?? undefined,
processIncarnation: authority?.processIncarnation ?? undefined
}
}
@@ -21,6 +21,7 @@ import {
type OrchestrationWorkerLaunchReceipt
} from './orchestration-worker-launch-preferences'
import { validateFederatedWorkerStartPlacement } from './orchestration-worker-start-validation'
import { resolveDispatchCreator } from './orchestration-dispatch-creator'
export async function startFederatedWorker(args: {
params: WorkerStartInput
@@ -97,6 +98,8 @@ export async function startFederatedWorker(args: {
const setupDecision = createsWorktree ? (params.setup ?? 'run') : 'not_applicable'
const started = db.createStartingWorkerDispatch({
creator: resolveDispatchCreator(runtime, params.from),
maxDepth: runtime.getNestedWorkerMaxDepth(),
taskId: task.id,
retryOf: params.retryOf,
startOptions: {
@@ -135,6 +138,9 @@ export async function startFederatedWorker(args: {
dispatchId: started.dispatch.id,
taskId: task.id,
taskSpec: task.spec,
// Carry the home dispatch depth across the federation boundary so a
// remote worker cannot be mistaken for a root when it dispatches again.
depth: started.dispatch.depth,
protocolVersion: federationProtocolVersion,
worktree,
name: params.name,
@@ -59,6 +59,7 @@ describe('federated worker agent launch', () => {
dispatchId: 'ctx_remote',
taskId: 'task_remote',
taskSpec: 'remote cursor worker',
depth: 2,
protocolVersion: 3,
worktree: 'folder:remote-workspace',
agent: 'cursor',
@@ -90,6 +91,7 @@ describe('federated worker agent launch', () => {
effective: { agent: 'cursor', model: 'gpt-5.3-codex', effort: 'high' }
}
})
expect(db.getRemoteDispatchAttachment('ctx_remote')?.depth).toBe(2)
expect(createTerminal).toHaveBeenCalledWith(
'id:folder:remote-workspace',
expect.objectContaining({
@@ -91,6 +91,8 @@ describe('orchestration federation control mail', () => {
runId = run.id
const task = homeDb.createTask({ spec: 'Wait for coordinator guidance', runId })
const started = homeDb.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
federation: {
@@ -118,6 +118,8 @@ describe('orchestration federated setup evidence', () => {
})
const task = db.createTask({ spec: 'remote setup', runId: run.id })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
runtimeEpoch: runtime.getRuntimeId(),
@@ -6,6 +6,8 @@ export const FederationAttachStartParams = z.object({
dispatchId: requiredString('Missing Dispatch ID'),
taskId: requiredString('Missing Task ID'),
taskSpec: requiredString('Missing Task spec'),
/** Depth stamped by the Run home; omitted by older clients and defaults to 1. */
depth: z.number().int().min(1).optional(),
protocolVersion: z.union([z.literal(1), z.literal(2), z.literal(3)]),
worktree: requiredString('Missing remote worktree selector'),
name: OptionalString,
@@ -57,6 +57,7 @@ export const ORCHESTRATION_FEDERATION_ATTACH_METHODS: RpcMethod[] = [
homePeerFingerprint: orchestrationMutation.callerFingerprint,
protocolVersion: params.protocolVersion,
runtimeEpoch: runtime.getRuntimeId(),
depth: params.depth,
mutationReceipt: orchestrationMutation
})
const effects: FederationEffect[] = []
@@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest'
import { OrcaRuntimeService } from '../../orca-runtime'
import { OrchestrationDb } from '../../orchestration/db'
import { ORCHESTRATION_METHODS } from './orchestration'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
describe('manual Dispatch observation', () => {
let db: OrchestrationDb | undefined
@@ -111,7 +112,8 @@ describe('manual Dispatch observation', () => {
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'injected lane', runId: run.id })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
'term_worker',
'tab_worker:leaf_worker',
@@ -207,6 +209,39 @@ describe('manual Dispatch observation', () => {
})
})
it('lists an unsupervised context-only dispatch even when process identity is absent', async () => {
db = new OrchestrationDb(':memory:')
const runtime = new OrcaRuntimeService()
runtime.setOrchestrationDb(db)
const run = db.createRun({
objective: 'context-only listing',
coordinatorHandle: 'term_coord',
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'operator lane', runId: run.id })
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
const workerListMethod = ORCHESTRATION_METHODS.find(
(candidate) => candidate.name === 'orchestration.workerList'
)
if (!workerListMethod) {
throw new Error('Missing method orchestration.workerList')
}
const result = (await workerListMethod.handler(
workerListMethod.params?.parse({ run: run.id }),
{ runtime }
)) as { workers: { dispatchId: string; workerState: string; terminalState: string | null }[] }
expect(result.workers).toEqual([
expect.objectContaining({
dispatchId: dispatch.id,
workerState: 'unsupervised',
terminalState: 'retained'
})
])
expect(db.getDispatchContextById(dispatch.id)?.process_incarnation).toBeNull()
})
it.each([
['orchestration.workerStop', 'stopped'],
['orchestration.workerAbandon', 'abandoned']
@@ -216,7 +251,8 @@ describe('manual Dispatch observation', () => {
runtime.setOrchestrationDb(db)
const closeTerminal = vi.spyOn(runtime, 'closeTerminal')
const task = db.createTask({ spec: 'operator-owned lane' })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
'term_worker',
'tab_worker:leaf_worker',
@@ -158,6 +158,8 @@ describe('manual Dispatch release', () => {
function createSupervisedWorker(): string {
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: createTask('supervised'),
startOptions: {}
})
@@ -8,6 +8,7 @@ import { OrchestrationDb } from '../../orchestration/db'
import { RpcDispatcher } from '../dispatcher'
import { ORCHESTRATION_METHODS } from './orchestration'
import { startFederatedWorker } from './orchestration-federated-worker-start'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
describe('orchestration migration behavior', () => {
const databases: OrchestrationDb[] = []
@@ -124,7 +125,7 @@ describe('orchestration migration behavior', () => {
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'legacy worker', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatcher = new RpcDispatcher({ runtime, methods: ORCHESTRATION_METHODS })
const response = await dispatcher.dispatch({
@@ -7,6 +7,7 @@ import type { RpcContext, RpcRequest } from '../core'
import { RpcDispatcher } from '../dispatcher'
import { ORCHESTRATION_METHODS } from './orchestration'
import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
type SendWarning = { code: string; recipient: string; message: string }
type SendResult = {
@@ -170,7 +171,7 @@ describe('orchestration recipient routing oracle', () => {
it('normalizes an active Dispatch owner even when no pane is live', async () => {
setup()
const task = db.createTask({ spec: 'detached worker' })
const dispatch = db.createDispatchContext(task.id, 'term_detached', 'tab_gone:leaf_gone')
const dispatch = createRootDispatch(db, task.id, 'term_detached', 'tab_gone:leaf_gone')
const result = (await call({
from: 'term_coord',
@@ -194,7 +195,7 @@ describe('orchestration recipient routing oracle', () => {
coordinatorPaneKey: 'tab_foreign:leaf_coord'
})
const task = db.createTask({ spec: 'detached foreign worker', runId: foreignRun.id })
db.createDispatchContext(task.id, 'term_detached_foreign', 'tab_gone:leaf_gone')
createRootDispatch(db, task.id, 'term_detached_foreign', 'tab_gone:leaf_gone')
await expect(
call({
@@ -211,7 +212,7 @@ describe('orchestration recipient routing oracle', () => {
setup()
const overlapPane = 'tab_overlap:leaf_overlap'
const task = db.createTask({ spec: 'overlapped worker' })
db.createDispatchContext(task.id, 'term_overlap', overlapPane)
createRootDispatch(db, task.id, 'term_overlap', overlapPane)
const recipientRun = db.createRun({
objective: 'Overlapping coordinator',
coordinatorHandle: 'term_overlap',
@@ -2,7 +2,10 @@ import type { OrchestrationCompatibilityEvidence } from '../../../../shared/orch
import { orchestrationSkillRecoveryData } from '../../../../shared/orchestration-rpc-contract'
import { OrchestrationError } from '../../orchestration/orchestration-error'
import type { RunRow } from '../../orchestration/types'
import type { OrcaRuntimeService } from '../../orca-runtime'
import type {
OrcaRuntimeService,
OrchestrationCompatibilityCallerAuthority
} from '../../orca-runtime'
export type RunScopeParams = {
runId?: string
@@ -33,6 +36,49 @@ export function assertCallerHandleMatchesEvidence(
}
}
export type OrchestrationCallerParams = {
callerTerminalHandle: string
callerEvidence?: OrchestrationCompatibilityEvidence
callerAuthority?: OrchestrationCompatibilityCallerAuthority
/** Preserve legacy callers that treated a missing pane as an ordinary fence. */
requireStablePane?: boolean
/**
* Skip attestation here because the caller performs it itself run-use must run
* its legacy-takeover check between pane resolution and attestation. Setting this
* without asserting elsewhere reopens the hole this helper exists to close.
*/
evidenceAssertedByCaller?: boolean
}
/** Resolve the caller's runtime pane and, by default, attest its declared handle. */
export function resolveOrchestrationCaller(
runtime: OrcaRuntimeService,
params: OrchestrationCallerParams & { requireStablePane: true }
): string
export function resolveOrchestrationCaller(
runtime: OrcaRuntimeService,
params: OrchestrationCallerParams
): string | null
export function resolveOrchestrationCaller(
runtime: OrcaRuntimeService,
params: OrchestrationCallerParams
): string | null {
if (!params.evidenceAssertedByCaller) {
assertCallerHandleMatchesEvidence(runtime, params.callerTerminalHandle, params.callerEvidence)
}
const paneKey =
params.callerAuthority?.terminalHandle === params.callerTerminalHandle
? params.callerAuthority.paneKey
: runtime.getTerminalPaneKey(params.callerTerminalHandle)
if (!paneKey && params.requireStablePane) {
throw new OrchestrationError(
'stable_pane_required',
'The coordinator terminal has no stable pane identity. Run this command inside a live Orca terminal.'
)
}
return paneKey ?? null
}
// Why: task and gate mutations must share one Run-binding rule.
export function resolveRunScope(runtime: OrcaRuntimeService, params: RunScopeParams): RunRow {
const db = runtime.getOrchestrationDb()
@@ -2,12 +2,11 @@ import { z } from 'zod'
import { defineMethod, type RpcMethod } from '../core'
import { OptionalBoolean, OptionalString, requiredString } from '../schemas'
import { ORCHESTRATION_RUN_PAGE_LIMIT } from '../../../../shared/orchestration-run-pagination'
import type {
OrcaRuntimeService,
OrchestrationCompatibilityCallerAuthority
} from '../../orca-runtime'
import { OrchestrationError } from '../../orchestration/orchestration-error'
import { assertCallerHandleMatchesEvidence } from './orchestration-run-scope'
import {
assertCallerHandleMatchesEvidence,
resolveOrchestrationCaller
} from './orchestration-run-scope'
const RunCreateParams = z.object({
objective: requiredString('Missing --objective'),
@@ -27,31 +26,16 @@ const RunListParams = z.object({
})
const RunShowParams = z.object({ id: requiredString('Missing --id'), from: OptionalString })
function requireCallerPane(
runtime: OrcaRuntimeService,
handle: string,
callerAuthority?: OrchestrationCompatibilityCallerAuthority
): string {
const paneKey =
callerAuthority?.terminalHandle === handle
? callerAuthority.paneKey
: runtime.getTerminalPaneKey(handle)
if (!paneKey) {
throw new OrchestrationError(
'stable_pane_required',
'The coordinator terminal has no stable pane identity. Run this command inside a live Orca terminal.'
)
}
return paneKey
}
export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [
defineMethod({
name: 'orchestration.runCreate',
params: RunCreateParams,
handler: (params, { orchestrationCompatibilityEvidence, runtime }) => {
assertCallerHandleMatchesEvidence(runtime, params.from, orchestrationCompatibilityEvidence)
const paneKey = requireCallerPane(runtime, params.from)
const paneKey = resolveOrchestrationCaller(runtime, {
callerTerminalHandle: params.from,
callerEvidence: orchestrationCompatibilityEvidence,
requireStablePane: true
})
const db = runtime.getOrchestrationDb()
const priorRun = db.getCurrentRunForPane(paneKey)
const run = db.createRun({
@@ -78,7 +62,13 @@ export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [
orchestrationCompatibilityCallerAuthority: callerAuthority
}
) => {
const paneKey = requireCallerPane(runtime, params.from, callerAuthority)
const paneKey = resolveOrchestrationCaller(runtime, {
callerTerminalHandle: params.from,
callerEvidence: orchestrationCompatibilityEvidence,
callerAuthority,
requireStablePane: true,
evidenceAssertedByCaller: true
})
if (
params.takeoverLegacy &&
(callerAuthority?.terminalHandle !== params.from || callerAuthority.paneKey !== paneKey)
@@ -117,8 +107,11 @@ export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [
name: 'orchestration.runCurrent',
params: RunCurrentParams,
handler: (params, { orchestrationCompatibilityEvidence, runtime }) => {
assertCallerHandleMatchesEvidence(runtime, params.from, orchestrationCompatibilityEvidence)
const paneKey = requireCallerPane(runtime, params.from)
const paneKey = resolveOrchestrationCaller(runtime, {
callerTerminalHandle: params.from,
callerEvidence: orchestrationCompatibilityEvidence,
requireStablePane: true
})
return { run: runtime.getOrchestrationDb().getCurrentRunForPane(paneKey) ?? null }
}
}),
@@ -5,6 +5,7 @@ import type { OrcaRuntimeService } from '../../orca-runtime'
import { openDecisionGateFromMessage } from '../../orchestration/coordinator-decision-gates'
import { applyEscalationToDispatch } from '../../orchestration/coordinator-escalation-triage'
import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
describe('orchestration.send Dispatch authority', () => {
const harness = createOrchestrationRpcHarness()
@@ -29,7 +30,8 @@ describe('orchestration.send Dispatch authority', () => {
async (legacyAuthority) => {
setup()
const attackerTask = db.createTask({ spec: 'attacker assignment' })
const attacker = db.createDispatchContext(
const attacker = createRootDispatch(
db,
attackerTask.id,
'term_attacker',
'tab_attacker:leaf_attacker',
@@ -37,7 +39,7 @@ describe('orchestration.send Dispatch authority', () => {
legacyAuthority ? undefined : 'runtime_test:term_attacker:1'
)
const victimTask = db.createTask({ spec: 'victim assignment' })
const victim = db.createDispatchContext(victimTask.id, 'term_victim')
const victim = createRootDispatch(db, victimTask.id, 'term_victim')
vi.mocked(runtime.getTerminalPaneKey).mockImplementation((handle) =>
handle === 'term_attacker' ? 'tab_attacker:leaf_attacker' : harness.coordinatorPaneKey
)
@@ -74,7 +76,7 @@ describe('orchestration.send Dispatch authority', () => {
it('rejects a caller-spoofed canonical Dispatch sender', async () => {
setup()
const task = db.createTask({ spec: 'legacy victim assignment' })
const dispatch = db.createDispatchContext(task.id, 'term_victim')
const dispatch = createRootDispatch(db, task.id, 'term_victim')
const result = (await send({
from: `dispatch:${dispatch.id}`,
@@ -97,7 +99,7 @@ describe('orchestration.send Dispatch authority', () => {
async (type) => {
setup()
const task = db.createTask({ spec: 'legacy owned assignment' })
db.createDispatchContext(task.id, 'term_legacy')
createRootDispatch(db, task.id, 'term_legacy')
vi.mocked(runtime.getTerminalPaneKey).mockImplementation((handle) =>
handle === 'term_legacy' ? 'tab_legacy:leaf_legacy' : harness.coordinatorPaneKey
)
@@ -123,7 +125,7 @@ describe('orchestration.send Dispatch authority', () => {
async (type) => {
setup()
const task = db.createTask({ spec: 'legacy re-dispatch target' })
const first = db.createDispatchContext(task.id, 'term_legacy')
const first = createRootDispatch(db, task.id, 'term_legacy')
const sent = (await send({
from: 'term_legacy',
@@ -137,7 +139,7 @@ describe('orchestration.send Dispatch authority', () => {
expect(JSON.parse(sent.message.payload)).toMatchObject({ dispatchId: first.id })
db.failDispatch(first.id, 'worker stopped before coordinator read its mail')
const second = db.createDispatchContext(task.id, 'term_legacy')
const second = createRootDispatch(db, task.id, 'term_legacy')
if (type === 'escalation') {
applyEscalationToDispatch(db, db.getMessageById(sent.message.id)!, () => {})
@@ -7,6 +7,7 @@ import type { OrchestrationDb } from '../../orchestration/db'
import type { OrcaRuntimeService } from '../../orca-runtime'
import type { RuntimeTerminalSummary } from '../../../../shared/runtime-types'
import { ORCHESTRATION_CONTRACT_VERSION } from '../../../../shared/protocol-version'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
function lifecycleGroupRecipientError(
type: 'worker_done' | 'heartbeat' | 'escalation' | 'decision_gate'
@@ -92,7 +93,7 @@ describe('orchestration RPC methods', () => {
it('routes exact Dispatch mail independently of terminal handles', async () => {
setup()
const task = db.createTask({ spec: 'controlled worker' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const result = (await call('orchestration.send', {
from: 'term_coord',
@@ -117,7 +118,8 @@ describe('orchestration RPC methods', () => {
it('routes Dispatch mail by stable pane identity after worker handle remint', async () => {
setup()
const task = db.createTask({ spec: 'controlled worker after restart' })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
'term_worker_before',
'tab_worker:leaf_worker'
@@ -185,7 +187,7 @@ describe('orchestration RPC methods', () => {
it('completes an identity-less injected send through its explicit worker handle', async () => {
setup()
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
vi.spyOn(runtime, 'getTerminalPaneKey').mockImplementation((handle) =>
handle === 'term_worker' ? 'tab_worker:leaf_worker' : null
)
@@ -211,7 +213,8 @@ describe('orchestration RPC methods', () => {
it('fences a replacement process for a capability-less manual Dispatch', async () => {
setup()
const task = db.createTask({ spec: 'process-bound manual work' })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
'term_worker',
'tab_worker:leaf_worker',
@@ -257,7 +260,8 @@ describe('orchestration RPC methods', () => {
async (type) => {
setup()
const task = db.createTask({ spec: `process-bound ${type}` })
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
'term_worker',
'tab_worker:leaf_worker',
@@ -313,7 +317,7 @@ describe('orchestration RPC methods', () => {
setup()
const task = db.createTask({ spec: 'work' })
const dependent = db.createTask({ spec: 'dependent', deps: [task.id] })
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
vi.spyOn(runtime, 'getTerminalPaneKey').mockImplementation((handle) =>
handle === 'term_coord' ? 'tab_coord:leaf_coord' : null
)
@@ -338,7 +342,7 @@ describe('orchestration RPC methods', () => {
it('ignores caller-supplied pane claims and uses the runtime-observed pane', async () => {
setup()
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue('tab_worker:leaf_worker')
vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {})
vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {})
@@ -374,7 +378,7 @@ describe('orchestration RPC methods', () => {
it('requires the minted capability, exact pane, and process incarnation', async () => {
setup()
const task = db.createTask({ spec: 'capability work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
const capability = db.mintDispatchCapability({
dispatchId: dispatch.id,
paneKey: 'tab_worker:leaf_worker',
@@ -456,7 +460,7 @@ describe('orchestration RPC methods', () => {
it('does not wake waiters for a heartbeat suppressed at send time', async () => {
setup()
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
db.updateTaskStatus(task.id, 'completed')
vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {})
const notify = vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {})
@@ -476,7 +480,7 @@ describe('orchestration RPC methods', () => {
it('still wakes waiters for a heartbeat on an active dispatch', async () => {
setup()
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {})
const notify = vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {})
@@ -685,7 +689,7 @@ describe('orchestration RPC methods', () => {
it('continues to send worker_done to a concrete terminal handle', async () => {
setup()
const task = db.createTask({ spec: 'work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
const result = (await call('orchestration.send', {
from: 'term_worker',
@@ -832,7 +836,7 @@ describe('orchestration RPC methods', () => {
it('releases dispatch lock before waking recipients when worker_done is sent via send', async () => {
setup()
const task = db.createTask({ spec: 'lock-release work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
// Why: waiter notification must observe the settled Dispatch, not stale lifecycle state.
vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {
@@ -857,13 +861,13 @@ describe('orchestration RPC methods', () => {
expect(db.getActiveDispatchForTerminal('term_worker')).toBeUndefined()
// Lock released — a new dispatch to the same terminal must succeed.
const t2 = db.createTask({ spec: 'follow-up work' })
expect(() => db.createDispatchContext(t2.id, 'term_worker')).not.toThrow()
expect(() => createRootDispatch(db, t2.id, 'term_worker')).not.toThrow()
})
it('records heartbeat when heartbeat is sent via send', async () => {
setup()
const task = db.createTask({ spec: 'heartbeat work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {})
await call('orchestration.send', {
@@ -883,7 +887,7 @@ describe('orchestration RPC methods', () => {
it('does not release dispatch lock for non-lifecycle sends', async () => {
setup()
const task = db.createTask({ spec: 'in-flight work' })
const dispatch = db.createDispatchContext(task.id, 'term_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker')
vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {})
await call('orchestration.send', {
@@ -4,6 +4,7 @@ import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness'
import type { OrchestrationDb } from '../../orchestration/db'
import type { OrcaRuntimeService } from '../../orca-runtime'
import { buildInjectRejectionMessage } from './orchestration-inject-rejection-message'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
describe('orchestration RPC methods', () => {
const h = createOrchestrationRpcHarness()
@@ -116,7 +117,7 @@ describe('orchestration RPC methods', () => {
setup()
const t1 = db.createTask({ spec: 'ready work' })
const t2 = db.createTask({ spec: 'active work' })
const ctx = db.createDispatchContext(t2.id, 'term_worker')
const ctx = createRootDispatch(db, t2.id, 'term_worker')
const result = (await call('orchestration.taskList', {})) as {
tasks: {
@@ -176,7 +177,7 @@ describe('orchestration RPC methods', () => {
it('completion frees the active dispatch context', async () => {
setup()
const task = db.createTask({ spec: 'work' })
db.createDispatchContext(task.id, 'term_a')
createRootDispatch(db, task.id, 'term_a')
await call('orchestration.taskUpdate', {
id: task.id,
@@ -397,7 +398,7 @@ describe('orchestration RPC methods', () => {
setup()
const t1 = db.createTask({ spec: 'first' })
const t2 = db.createTask({ spec: 'second' })
db.createDispatchContext(t1.id, 'term_a')
createRootDispatch(db, t1.id, 'term_a')
await expect(call('orchestration.dispatch', { task: t2.id, to: 'term_a' })).rejects.toThrow(
/already has an active dispatch/
@@ -453,7 +454,7 @@ describe('orchestration RPC methods', () => {
it('shows dispatch context for a task', async () => {
setup()
const task = db.createTask({ spec: 'work' })
db.createDispatchContext(task.id, 'term_a')
createRootDispatch(db, task.id, 'term_a')
const result = (await call('orchestration.dispatchShow', {
task: task.id
@@ -474,7 +475,7 @@ describe('orchestration RPC methods', () => {
it('--preamble returns the preamble text', async () => {
setup()
const task = db.createTask({ spec: 'refactor auth' })
db.createDispatchContext(task.id, 'term_a')
createRootDispatch(db, task.id, 'term_a')
const result = (await call('orchestration.dispatchShow', {
task: task.id,
@@ -6,6 +6,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest'
import { OrcaRuntimeService } from '../../orca-runtime'
import { OrchestrationDb } from '../../orchestration/db'
import { ORCHESTRATION_METHODS } from './orchestration'
import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture'
vi.mock('electron', () => ({
BrowserWindow: { fromId: vi.fn(() => null) },
@@ -100,12 +101,12 @@ describe('worker-show interactive wait (STA-3714, STA-4513)', () => {
if (!paneKey || !incarnation) {
throw new Error('Runtime did not expose the worker pane identity.')
}
const dispatch = db.createDispatchContext(
const dispatch = createRootDispatch(
db,
task.id,
terminal.handle,
paneKey,
'launch-hash',
// A dispatch recorded against a process that has since been replaced.
'launch-hash', // A dispatch recorded against a process that has since been replaced.
opts?.breakIdentity === true ? `${incarnation}:replaced` : incarnation
)
db.mintDispatchCapability({
@@ -44,6 +44,8 @@ describe('worker-stop against a terminal we lost contact with', () => {
})
const task = db.createTask({ spec: 'stop worker', runId: run.id })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
runtimeEpoch: runtime.getRuntimeId()
@@ -61,6 +61,8 @@ describe('orchestration worker recovery', () => {
})
const task = db.createTask({ spec: 'recover worker', runId: run.id })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
runtimeEpoch
@@ -206,6 +208,8 @@ describe('orchestration worker recovery', () => {
})
const task = db.createTask({ spec: 'interrupted', runId: run.id })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
runtimeEpoch: 'previous_runtime'
@@ -244,6 +248,8 @@ describe('orchestration worker recovery', () => {
})
const task = db.createTask({ spec: 'stop remote worker', runId: run.id })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {},
runtimeEpoch: runtime.getRuntimeId(),
@@ -20,14 +20,24 @@ import {
} from './orchestration-worker-setup-gate'
import { failWorkerStartWithReceipt } from './orchestration-worker-start-receipt'
import { prepareLocalWorkerStart } from './orchestration-worker-start-validation'
import { resolveDispatchCreator } from './orchestration-dispatch-creator'
import { resolveOrchestrationCaller } from './orchestration-run-scope'
export const ORCHESTRATION_WORKER_START_METHODS: RpcMethod[] = [
defineMethod({
name: 'orchestration.workerStart',
params: WorkerStartParams,
handler: async (params, { runtime, orchestrationMutation }) => {
handler: async (
params,
{ runtime, orchestrationMutation, orchestrationCompatibilityEvidence }
) => {
const db = runtime.getOrchestrationDb()
const coordinatorPane = runtime.getTerminalPaneKey(params.from)
// Why: worker-start was the only Run-scoped verb that skipped this, so a
// declared --from could name someone else's pane and inherit their depth.
const coordinatorPane = resolveOrchestrationCaller(runtime, {
callerTerminalHandle: params.from,
callerEvidence: orchestrationCompatibilityEvidence
})
const run = coordinatorPane ? db.getCurrentRunForPane(coordinatorPane) : undefined
if (!run || (params.run && params.run !== run.id)) {
throw new OrchestrationError(
@@ -110,6 +120,8 @@ export const ORCHESTRATION_WORKER_START_METHODS: RpcMethod[] = [
: 'existing_worktree'
}
const started = db.createStartingWorkerDispatch({
creator: resolveDispatchCreator(runtime, params.from),
maxDepth: runtime.getNestedWorkerMaxDepth(),
taskId: task.id,
retryOf: params.retryOf,
startOptions,
@@ -2,6 +2,7 @@
import { z } from 'zod'
import { setImmediate as yieldToEventLoop } from 'node:timers/promises'
import { defineMethod, type RpcMethod } from '../core'
import { resolveDispatchCreator } from './orchestration-dispatch-creator'
import { OptionalFiniteNumber, OptionalString, OptionalBoolean, requiredString } from '../schemas'
import {
LEGACY_CONTRACT_VERSION,
@@ -1663,13 +1664,15 @@ export const ORCHESTRATION_METHODS: RpcMethod[] = [
}
revalidateLegacyCoordinator?.()
const ctx = db.createDispatchContext(
params.task,
to,
const ctx = db.createDispatchContext({
taskId: params.task,
assigneeHandle: to,
assigneePaneKey,
dispatchAuthority?.launchTokenHash ?? undefined,
processIncarnation
)
launchTokenHash: dispatchAuthority?.launchTokenHash ?? undefined,
processIncarnation,
creator: resolveDispatchCreator(runtime, params.from),
maxDepth: runtime.getNestedWorkerMaxDepth()
})
const dispatchCapability = params.inject
? db.mintDispatchCapability({
dispatchId: ctx.id,
@@ -23,6 +23,7 @@ import {
request,
type LegacyCompatibilityDispatcherHarness
} from './orchestration-legacy-compatibility-dispatcher-test-fixture'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
// Why: an unrelated caller the runtime CAN resolve to a pane — otherwise the refusal would be
// stable_pane_required and would prove nothing about Run authorization.
@@ -642,7 +643,8 @@ function createAdoptedDb(options: { settleWork: boolean }): {
const before = new OrchestrationDb(dbPath)
const task = before.createTask({ spec: 'legacy assignment', createdByTerminalHandle: 'term_old' })
before.createDispatchContext(
createRootDispatch(
before,
task.id,
'term_old_worker',
'tab_old:33333333-3333-4333-8333-333333333333'
@@ -15,6 +15,7 @@ import {
WORKER_HANDLE,
WORKER_PANE
} from './orchestration-legacy-compatibility-dispatcher-test-fixture'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
afterEach(() => {
cleanupLegacyCompatibilityDispatcherHarnesses()
@@ -287,7 +288,8 @@ function createCurrentDispatch(harness: ReturnType<typeof createHarness>): {
coordinatorPaneKey: CURRENT_COORDINATOR_PANE
})
const task = harness.db.createTask({ spec: 'current assignment', runId: run.id })
const dispatch = harness.db.createDispatchContext(
const dispatch = createRootDispatch(
harness.db,
task.id,
CURRENT_WORKER_HANDLE,
CURRENT_WORKER_PANE
@@ -326,7 +328,7 @@ async function createReusedCurrentDispatch(
coordinatorPaneKey: CURRENT_COORDINATOR_PANE
})
const task = harness.db.createTask({ spec: 'reused terminal assignment', runId: run.id })
const dispatch = harness.db.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(harness.db, task.id, WORKER_HANDLE, WORKER_PANE)
const capability = harness.db.mintDispatchCapability({
dispatchId: dispatch.id,
paneKey: WORKER_PANE,
@@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db'
import type { RpcRequest, RpcResponse } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
export const WORKER_HANDLE = 'term_legacy_worker'
export const WORKER_PANE = 'tab_worker:33333333-3333-4333-8333-333333333333'
@@ -55,7 +56,7 @@ export function createHarness(): LegacyCompatibilityDispatcherHarness {
spec: 'legacy assignment',
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE)
before.close()
const raw = new Database(dbPath)
@@ -15,6 +15,7 @@ import {
WORKER_HANDLE,
WORKER_PANE
} from './orchestration-legacy-compatibility-dispatcher-test-fixture'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
afterEach(() => {
cleanupLegacyCompatibilityDispatcherHarnesses()
@@ -191,7 +192,8 @@ describe('legacy compatibility through RpcDispatcher', () => {
processIncarnation: 'process-1'
})
harness.db.updateTaskStatus(harness.taskId, 'ready')
const currentDispatch = harness.db.createDispatchContext(
const currentDispatch = createRootDispatch(
harness.db,
harness.taskId,
'term_current_worker',
'tab_current_worker:77777777-7777-4777-8777-777777777777',
@@ -395,7 +397,8 @@ describe('legacy compatibility through RpcDispatcher', () => {
coordinatorPaneKey: 'tab_current_coord:55555555-5555-4555-8555-555555555555'
})
const task = harness.db.createTask({ spec: 'current assignment', runId: run.id })
const dispatch = harness.db.createDispatchContext(
const dispatch = createRootDispatch(
harness.db,
task.id,
'term_current_worker',
'tab_current_worker:66666666-6666-4666-8666-666666666666',
@@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db'
import type { RpcRequest, RpcResponse } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
const COORDINATOR_HANDLE = 'term_legacy_coord'
const COORDINATOR_PANE = 'tab_coord:44444444-4444-4444-8444-444444444444'
@@ -45,7 +46,7 @@ function createHarness(): Harness {
spec: 'legacy assignment',
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE)
before.close()
const raw = new Database(dbPath)
@@ -11,6 +11,7 @@ import { OrcaRuntimeService } from '../orca-runtime'
import type { RpcRequest } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
const COORDINATOR_HANDLE = 'term_legacy_coord'
const CURRENT_COORDINATOR_HANDLE = 'term_current_coord'
@@ -42,7 +43,7 @@ function createHarness(options?: { seedCutoverQuestion?: boolean; seedCutoverAns
spec: 'legacy assignment',
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE)
const cutoverQuestion = options?.seedCutoverQuestion
? before.insertMessage({
from: WORKER_HANDLE,
@@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db'
import type { RpcRequest } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
const WORKER_HANDLE = 'term_legacy_worker'
const WORKER_PANE = 'tab_worker:33333333-3333-4333-8333-333333333333'
@@ -53,7 +54,7 @@ function createHarness(): Harness {
spec: 'legacy assignment',
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE)
before.close()
const raw = new Database(dbPath)
@@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db'
import type { RpcRequest } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
const WORKER_HANDLE = 'term_legacy_worker'
const WORKER_PANE = 'tab_worker:33333333-3333-4333-8333-333333333333'
@@ -48,7 +49,7 @@ function createHarness(): Harness {
spec: 'legacy assignment',
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE)
before.close()
const raw = new Database(dbPath)
@@ -10,6 +10,7 @@ import { OrchestrationDb } from '../orchestration/db'
import { defineMethod, type RpcRequest } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
const Params = z.object({ subject: z.string() })
@@ -304,6 +305,8 @@ describe('durable orchestration mutation ledger', () => {
.update(JSON.stringify({ method: 'orchestration.workerStart', params }))
.digest('hex')
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: params.task,
startOptions: {},
mutationReceipt: {
@@ -371,7 +374,7 @@ describe('durable orchestration mutation ledger', () => {
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'ask', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker')
const capability = db.mintDispatchCapability({
dispatchId: dispatch.id,
paneKey: 'tab_worker:leaf_worker',
@@ -10,6 +10,7 @@ import { OrchestrationDb } from '../orchestration/db'
import type { RpcRequest, RpcResponse } from './core'
import { RpcDispatcher } from './dispatcher'
import { ORCHESTRATION_METHODS } from './methods/orchestration'
import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture'
const WORKER_HANDLE = 'term_pre_update_worker'
const WORKER_PANE = 'tab_pre_update:33333333-3333-4333-8333-333333333333'
@@ -64,7 +65,7 @@ function createUpdateHarness(): Harness {
spec: 'finish work across an app update',
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = oldRuntimeDb.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(oldRuntimeDb, task.id, WORKER_HANDLE, WORKER_PANE)
const capability = oldRuntimeDb.mintDispatchCapability({
dispatchId: dispatch.id,
paneKey: WORKER_PANE,
@@ -44,7 +44,12 @@ describe('Task/Dispatch state invariant', () => {
const task = harness.db.createTask({ spec: 'retain assignment', runId: harness.runId })
const dispatch =
dispatchStatus === 'pending'
? harness.db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }).dispatch
? harness.db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
}).dispatch
: await dispatchTask(harness, task.id, WORKER_HANDLE)
const response = await updateTask(harness, task.id, 'ready', 'must not persist')
@@ -285,7 +290,12 @@ async function createCapableDispatch(
status: 'pending' | 'dispatched'
): Promise<{ dispatch: { id: string }; capability: string }> {
if (status === 'pending') {
const dispatch = harness.db.createStartingWorkerDispatch({ taskId, startOptions: {} }).dispatch
const dispatch = harness.db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId,
startOptions: {}
}).dispatch
const capability = harness.db.prepareStartingWorkerAuthority({
dispatchId: dispatch.id,
handle: WORKER_HANDLE,
@@ -312,7 +322,12 @@ function createSupervisedDispatch(
taskId: string,
status: 'pending' | 'dispatched'
): { dispatch: { id: string }; capability: string } {
const dispatch = harness.db.createStartingWorkerDispatch({ taskId, startOptions: {} }).dispatch
const dispatch = harness.db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId,
startOptions: {}
}).dispatch
const capability = harness.db.prepareStartingWorkerAuthority({
dispatchId: dispatch.id,
handle: WORKER_HANDLE,
@@ -14,6 +14,7 @@ import {
waitFor,
seedSupervisedAskWorkers
} from './runtime-rpc-test-harness'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
vi.mock('../git/worktree', () => {
const worktrees = [
@@ -129,7 +130,7 @@ describe('OrcaRuntimeRpcServer', () => {
coordinatorPaneKey: 'tab_coord:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
})
const task = db.createTask({ spec: 'Wait for an answer', runId: run.id })
db.createDispatchContext(task.id, 'term_asker', askerPaneKey)
createRootDispatch(db, task.id, 'term_asker', askerPaneKey)
const server = new OrcaRuntimeRpcServer({
runtime,
userDataPath,
+2 -1
View File
@@ -1,6 +1,7 @@
import { createConnection, type Socket } from 'node:net'
import type { OrchestrationDb } from './orchestration/db'
import { ORCHESTRATION_CONTRACT_VERSION } from '../../shared/protocol-version'
import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture'
export async function sendRequest(
endpoint: string,
@@ -112,6 +113,6 @@ export function seedSupervisedAskWorkers(db: OrchestrationDb, workerHandles: str
})
for (const workerHandle of workerHandles) {
const task = db.createTask({ spec: 'Wait for coordinator input', runId: run.id })
db.createDispatchContext(task.id, workerHandle)
createRootDispatch(db, task.id, workerHandle)
}
}
+9 -3
View File
@@ -15,6 +15,7 @@ import { OrchestrationDb } from '../runtime/orchestration/db'
import { OrcaRuntimeService } from '../runtime/orca-runtime'
import type { HostCliPassthroughOptions } from './ssh-remote-cli-host-passthrough'
import { runRemoteOrcaCli } from './ssh-remote-orca-cli'
import { createRootDispatch } from '../runtime/orchestration/db/root-dispatch-test-fixture'
// Why: pointing the passthrough at a missing CLI entry forces the legacy
// in-process fallback, which is what these dispatch tests exercise.
@@ -241,7 +242,7 @@ describe('runRemoteOrcaCli', () => {
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'remote work', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_ssh', 'tab_owner:leaf_owner')
const dispatch = createRootDispatch(db, task.id, 'term_ssh', 'tab_owner:leaf_owner')
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue('tab_foreign:leaf_foreign')
try {
@@ -299,7 +300,7 @@ describe('runRemoteOrcaCli', () => {
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'remote work', runId: run.id })
const dispatch = db.createDispatchContext(task.id, 'term_ssh', 'tab_owner:leaf_owner')
const dispatch = createRootDispatch(db, task.id, 'term_ssh', 'tab_owner:leaf_owner')
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue('tab_owner:leaf_owner')
try {
@@ -358,7 +359,12 @@ describe('runRemoteOrcaCli', () => {
coordinatorPaneKey: 'tab_coord:leaf_coord'
})
const task = db.createTask({ spec: 'remote work', runId: run.id })
const started = db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
const started = db.createStartingWorkerDispatch({
creator: { kind: 'system' },
maxDepth: Number.MAX_SAFE_INTEGER,
taskId: task.id,
startOptions: {}
})
const capability = db.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_ssh',
@@ -17,6 +17,7 @@ import type Database from '../sqlite/sync-database'
import type { HostCliPassthroughOptions } from './ssh-remote-cli-host-passthrough'
import { runRemoteOrcaCli } from './ssh-remote-orca-cli'
import { acknowledgeRemoteOrcaCliPostOutput } from './ssh-remote-orchestration-post-output'
import { createRootDispatch } from '../runtime/orchestration/db/root-dispatch-test-fixture'
const LEGACY_FALLBACK_OPTIONS: HostCliPassthroughOptions = {
execPath: '/host/electron',
@@ -57,7 +58,7 @@ function createLegacyRuntime() {
runId: run.id,
createdByTerminalHandle: COORDINATOR_HANDLE
})
const dispatch = db.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE)
const dispatch = createRootDispatch(db, task.id, WORKER_HANDLE, WORKER_PANE)
const sqlite = (db as unknown as { db: Database.Database }).db
sqlite
.prepare(
+1
View File
@@ -281,6 +281,7 @@ export function getDefaultSettings(homedir: string): GlobalSettings {
artifactsEnabled: true,
artifactSharingEnabled: false,
agentSkillSharingEnabled: false,
nestedWorkerMaxDepth: 1,
showArtifactsButton: false,
showSkillsButton: false,
showMobileButton: true,
+4
View File
@@ -230,6 +230,10 @@ export type GlobalSettings = {
artifactSharingEnabled?: boolean
/** Capability gate for agent/CLI skill publishing; manual reviewed publishing remains available. */
agentSkillSharingEnabled?: boolean
/** How deep dispatched workers may nest. 1 = workers cannot dispatch sub-workers.
* Renderer-writable only: omitted from the SettingsUpdate RPC schema so a worker
* cannot raise its own cap via `orca settings update`. */
nestedWorkerMaxDepth?: number
/** Only toggles the sidebar shortcut; Artifacts stay reachable from Settings. */
showArtifactsButton?: boolean
/** Only toggles the sidebar shortcut; Skills stay reachable from Settings. */
+44
View File
@@ -0,0 +1,44 @@
import { describe, expect, it } from 'vitest'
import {
NESTED_WORKER_MAX_DEPTH_DEFAULT,
nestedWorkerDepthExceededMessage,
resolveNestedWorkerMaxDepth
} from './nested-worker-depth'
describe('resolveNestedWorkerMaxDepth', () => {
it('defaults to 1 when unset', () => {
expect(resolveNestedWorkerMaxDepth(undefined)).toBe(1)
expect(resolveNestedWorkerMaxDepth(null)).toBe(1)
expect(resolveNestedWorkerMaxDepth({})).toBe(1)
})
it('accepts whole numbers at or above 1', () => {
expect(resolveNestedWorkerMaxDepth({ nestedWorkerMaxDepth: 1 })).toBe(1)
expect(resolveNestedWorkerMaxDepth({ nestedWorkerMaxDepth: 3 })).toBe(3)
})
// A malformed setting must not become a way to get unlimited nesting, so every
// rejected shape falls back to the default rather than disabling the cap.
it.each([
['a numeric string', '2'],
['a boolean', true],
['zero', 0],
['negative', -1],
['fractional', 1.5],
['NaN', Number.NaN],
['Infinity', Number.POSITIVE_INFINITY],
['null', null]
])('falls back to the default for %s', (_label, value) => {
expect(resolveNestedWorkerMaxDepth({ nestedWorkerMaxDepth: value as unknown as number })).toBe(
NESTED_WORKER_MAX_DEPTH_DEFAULT
)
})
})
describe('depth-exceeded message', () => {
it('names both depths and tells the worker to finish the task itself', () => {
const message = nestedWorkerDepthExceededMessage(2, 1)
expect(message).toContain('depth 2 (max 1)')
expect(message).toContain('Complete this task yourself')
})
})
+42
View File
@@ -0,0 +1,42 @@
import type { GlobalSettings } from './global-settings-types'
/**
* How deep dispatched workers may nest. 1 means a coordinator dispatches
* workers and those workers may not dispatch further the behaviour Orca
* documented but never actually enforced.
*/
export const NESTED_WORKER_MAX_DEPTH_DEFAULT = 1
/** Root coordinators are depth 0; the first generation of workers is depth 1. */
export const ROOT_DISPATCH_DEPTH = 0
export const NESTED_WORKER_DEPTH_EXCEEDED_CODE = 'nested_worker_depth_exceeded'
export function nestedWorkerDepthExceededMessage(childDepth: number, maxDepth: number): string {
// Why "complete this task yourself": a refusal alone leaves the worker looping
// on a capability it will never get.
return (
`Sub-worker dispatch is not permitted at depth ${childDepth} (max ${maxDepth}). ` +
'Complete this task yourself.'
)
}
export const NESTED_WORKER_DEPTH_EXCEEDED_NEXT_STEPS: readonly string[] = [
'Do the work in this terminal instead of dispatching a sub-worker.',
'To allow deeper nesting, open Settings → Agents in the Orca desktop app and raise "Nested worker depth".'
]
/**
* Clamp to a usable integer. Anything that is not a whole number >= 1 falls back
* to the default rather than disabling the fence: a malformed setting must not
* be a way to get unlimited nesting.
*/
export function resolveNestedWorkerMaxDepth(
settings: Pick<GlobalSettings, 'nestedWorkerMaxDepth'> | null | undefined
): number {
const raw = settings?.nestedWorkerMaxDepth
if (typeof raw !== 'number' || !Number.isInteger(raw) || raw < 1) {
return NESTED_WORKER_MAX_DEPTH_DEFAULT
}
return raw
}