Commit Graph
30 Commits
Author SHA1 Message Date
Jinwoo Hong 1a60595845 fix(orchestration): record orchestration as the origin of nested workers on connected servers; lead the guide with new-child (#40091)
worker-start --on <server> --worktree new-child now tells the server that
orchestration picked the parent, so the server records the child's lineage as
orchestration, as a local child is, instead of cli. Older coordinators omit the
field and keep cli. The guide now shows new-child first and new-top-level as
the no-parent choice.
2026-11-09 02:48:36 -05:00
Jinwoo Hong 555bd481bc Let multi-repo folder workspaces parent and nest workspaces, including on other machines (#38904)
Multi-repo folder workspaces can parent workspaces on this machine and on paired servers, and their children nest under them in the sidebar and on the phone. Task parents are inferred at creation instead of on every lineage refresh.
2026-11-06 19:03:51 -05:00
Jinwoo Hong a9ba59c944 Nest workspaces under parents on other machines (#38740)
* feat(lineage): store cross-store parent links beside same-store lineage (inert)

A workspace's parent can live in another Orca store. Save that link in the
child's store under a new top-level key, crossStoreLineageByChildKey, so older
builds' same-store cleanup never reads it. Nothing writes records yet.

- Types, load normalization (malformed entries dropped one by one), section label.
- Store get/set/remove by child; a child keeps at most one parent record.
- Every maintenance path acts by the child only: deletion and missing-worktree
  pruning, repo removal and deregistered-repo residue, identity migration,
  profile project transfer, WSL move projection and UNC drive conversion, and
  the orcad migration dormant payload (export, validation, import, subtraction).

* fix(lineage): gate cross-store links on host capability; one parent per child

- Migration stage/commit carrying crossStoreLineage requires the new
  orcad.migration-cross-store-lineage.v1 capability; older hosts are refused
  before staging instead of silently dropping the links. Empty arrays are omitted.
- Payload validation and destination prepare reject a child that has both a
  cross-store and a same-store parent, through one shared conflict check.
- Saved entries stay on disk as written; the map is Record<string, unknown> and
  readers go through validCrossStoreLineage. Re-keying moves unreadable entries unchanged.
- Tests: literal expectations; drop touched mock-call and work-count assertions.

* test(lineage): drop clone and save work-count assertions from project group removal test

* feat(lineage): resolve cross-store parents into the sidebar parent map (inert)

- One resolver: a saved record nests its child under the single visible
  row whose store reports the parent's installationId and whose instance
  matches; copied installs, unknown stores and stale instances stay standalone.
- getSidebarLineageParentsByIdentity merges resolved cross-store parents with
  same-store edges before cycle breaking; a valid same-store edge wins.
- Workspace delete collects descendants only from the parent's own store.
- Store identity: paired stores from their verified status hostDescriptor;
  this store from the published descriptor, else host-installation.json.
- Nothing passes a view or writes records yet, so the sidebar is unchanged.

* fix(lineage): import WorkspaceKey from workspace-scope after main move

* fix(lineage): resolve sidebar parents once over complete rows; keep unowned rows' delete rule

- resolveSidebarLineageParentIdentities resolves same-store edges and cross-store
  records once over every visible store's unfiltered rows, then drops cycles.
  Ancestor injection and nesting reuse that map, so a filtered-out twin still
  makes a parent ambiguous.
- Removed, retired or refusing paired servers leave the view: their records stay
  saved and their children standalone. Offline servers keep their last-known id.
- A row with no owner is unknown, not ours: it never resolves, and delete only
  excludes descendants whose store is known and differs from the parent's.

* fix(lineage): refuse UNC rekey collisions and transfers giving a child two parents

The UNC drive conversion rekeys cross-store child keys through the shared
collision merge, so a UNC key and its drive key with different records throw
instead of one silently overwriting the other. Profile project transfer runs the
shared child-key conflict check before writing the target.

* feat(lineage): write, carry and nest cross-machine parents

- Capability worktree.lineage.crossMachine on the child's store; the CLI
  refuses before any request when the server lacks it.
- worktree.set / worktree.create accept crossStoreParent
  { installationId, workspaceKey, instanceId } and write only
  crossStoreLineageByChildKey (refused when it names this store, or the
  child has no identity). Remove Parent clears both.
- worktree.crossStoreParentRef: the parent's own store names it with its
  installationId; reparent refuses a parent whose chain already reaches
  another store. Same-store Set Parent refuses the same.
- worktree.ps rows and lineage lists carry the records; same-store lineage
  fields stay null so older readers show the child standalone.
- The desktop sidebar resolves parents over its own and paired stores'
  records (PR 2's resolver) at the visible-worktree and row-plan call sites.
- CLI: active/current or an inferred terminal parent against a server is
  sent as a cross-store parent.

* fix(lineage): read worktree rows through the record selector; mark the test runtime cast

* fix(lineage): hydrate every store, fence refreshes, honest CLI fallback

- Each host's catalog hydration fetches its lineage snapshot, so this
  desktop's identity and records load whichever store is focused.
- Cross-store records join the per-child request-start fence: a stale
  lineage list can't restore a removed record or override a newer one.
- A failed inferred cross-store parent sends noParent instead of
  forwarding the selector a server could match to a same-store twin.
- Inferred cross-store creates record env-workspace/inferred capture.
- The writer states its trust contract; the chain rule runs at the mint.

* feat(lineage): cross-machine parents on the phone, pickers, chat resume and labels

- Phone: the list nests a child under a parent on another machine, from the
  records rows already carry and the store identities its host verified
  (status.get descriptor; relayed servers' installationId on the host list).
- Set Parent popover, context menu and the composer's parent picker offer
  parents in other visible stores. The parent's own store names it
  (worktree.crossStoreParentRef); the window checks it against that store's
  verified installationId and the child store's capability before any write.
  Remove Parent and Open parent cover cross-machine links.
- Native chat resume ancestors follow resolved cross-machine parents.
- Host sections keep a nested child under its parent and keep its own
  machine label.
- Main sidebar feed resolves cross-store parents (rows carry records; desktop
  facts and paired-host facts carry identities); paired web windows read
  their host's installationId; reveal without an occurrence plan resolves
  them; `worktree create --agent` names a parent on another machine.

* test(lineage): type the mint test's parent row through unknown

* test(lineage): give the cross-machine tests' partial-row casts their SAFETY reasons

* fix(cli): worktree set names a parent on another machine

* feat(lineage): name the machine of parent candidates on another machine

* fix(lineage): Set Parent and Remove Parent act on the clicked row's own host

* fix(lineage): phone resolves parents over every row before the repo filter

* fix(lineage): chat resume nests by host-qualified identity across machines

* fix(lineage): local create refuses a same-install parent before checkout and shares the RPC recorder

* fix(lineage): name parent machines without a focus read; keep create's probes synchronous

* fix(lineage): chat resume keys offers by their own machine; only cross-machine parents leave a section

* fix(lineage): name an older parent server before asking it for a cross-machine ref

* fix(lineage): an app-picked parent on another machine records as manual

* fix(lineage): chat resume keeps base grouping; only resolved cross-store parents move a chat

* revert(lineage): drop chat resume from cross-machine parents; it groups by machine as before

* refactor(lineage): CLI builds cross-machine parents; explicit only

- Drop worktree.crossStoreParentRef and its reparent chain walk: the CLI
  builds the ref from this desktop's installationId (worktree.lineageList
  now reports it) and the parent row it reads. Cross-machine chains are
  allowed; same-store cycle validation and same-install refusal stay.
- No inferred cross-machine parent: remove crossStoreParentInferred and
  the fallback states. Against a server, this machine's terminal, cwd and
  env context is never sent; the CLI prints how to nest explicitly.

* refactor(lineage): build cross-machine parent refs from verified ids; pickers hide only loops; one desktop install id source

* refactor(lineage): refuse moving a linked workspace instead of carrying the link

Orcad migration and profile project transfer now refuse, before any staging or
write, when a workspace in the move has a cross-store parent record (raw entries,
readable or not). Remove the carry machinery: the dormant crossStoreLineage
field, its runtime capability and destination gate, import/subtract/claim code,
transfer carry and the two-parent conflict checker. Also drop the unused
removeCrossStoreWorkspaceLineage.

* docs(lineage): describe cross-store parent refs without the removed mint RPC

* test(lineage): drop the RPC call-list assertion from the cross-store ref test

* fix(lineage): refuse linked moves before any write and for shared-repo children

- Migration refusal passes the child's own host stamp to the scope matcher and
  refuses an unstamped key whose shared repo id the move includes, so such a
  child can't move while its link stays behind.
- Profile transfer checks the read-only source snapshot before pending-move
  recovery or storage migration; the payload check stays as the final guard.

* fix(lineage): refuse a linked profile move before the profile lookup can restore the index

* fix(lineage): startup hydration of other stores loads only cross-machine links

* fix(lineage): list the child's own machine first in the parent picker

* fix(lineage): keep profile argument errors ahead of the linked-move preflight

* fix(worktrees): let the first listing replace a restored server workspace stub

A restored active server workspace starts as a session stub with no head.
If git status fills in its branch before the first listing lands, the
listing merge took that as a concurrent branch switch and kept the stub,
dropping the catalog row (instance id, lineage, provenance). The workspace
then showed standalone until the next listing.

* fix(worktrees): accept a headless row's listing only when it confirms git status

A stale listing could otherwise roll back a branch switch git status saw on
an empty-head row (e.g. an SSH fallback row) during the first connected listing.

* feat(lineage): CLI nests a create on another machine under the caller workspace

- No parent flag and no --no-parent against a server: the caller's workspace
  on this machine (env, else cwd) is sent as an inferred cross-store parent,
  the way --parent-worktree active builds it. The host records it as
  inferred from environment or cwd.
- A server without worktree.lineage.crossMachine, or a caller workspace that
  can't be linked, still creates unnested and prints one note.
- An inferred parent falls away for folder targets, as same-machine folder
  creates record none.

* fix(lineage): CLI cross-machine inference reads the caller terminal and suppresses fallbacks

- The caller terminal's workspace joins env and cwd as an inferred parent,
  with same-machine precedence: env and terminal must agree, cwd only alone.
- A fallback (old server, unlinkable or unresolved caller, ambiguous cwd)
  prints one note and sends noParent, so the server's own create-time
  inference can't nest what the CLI reported as not nested.

* feat(sidebar): drag a workspace onto one on another machine to nest it

Dropping a workspace on a workspace from another machine now sets the
parent through the same cross-machine path as Set Parent: same eligibility
(loops, older servers), same refusal messages. Drop targets carry host
identity so the child's own copy is the one written.

* refactor(lineage): this machine's Orca resolves the caller parent for a create on another machine

- New read-only worktree.resolveCallerParent runs the same-machine create
  resolver over the caller's env, terminal and cwd context, so stale
  contexts are skipped and only validated parents can conflict, and returns
  the cross-store ref or a reason.
- The CLI's own precedence and terminal lookup are removed; it keeps the
  capability gate, one note with noParent on any fallback, and silence with
  no Orca context.

* feat(orchestration): remote worker-start --worktree new-child nests under the coordinator's workspace

worker-start --on <server> --worktree new-child creates the server's worktree
under the coordinator's workspace on this machine, using the same verified
cross-machine parent ref as the CLI's caller auto-nest. A server without
cross-machine parents still starts the worker, top-level, with a warning in
the receipt. Guide, CLI help and docs describe it.

* fix(lineage): resolveCallerParent reads this store's identity without creating it

- Uses the published host descriptor's installationId, else the existing
  host-installation.json; never creates or repairs it. Missing identity
  returns a reason, so the CLI creates unnested with its note.
- Drops the read-only wording: parent resolution does the same catalog
  upkeep a same-machine create does.

* fix(sidebar): validate every cross-machine parent before a multi-select drop writes

A drop of several workspaces builds every cross-machine parent ref first, so an
older server's refusal leaves the whole selection unchanged. Drops the touched
preview-timing test cases.

* refactor(orchestration): build the remote new-child parent ref with the shared caller resolver

Orchestration still picks the caller workspace; resolveCallerParentForAnotherStore
verifies it and builds the ref, and both read the installationId the same way.
Drops the copy and mock-call tests.

* fix(lineage): narrow the inferred source with a guard instead of casts

* fix(lineage): a composer parent picked on another machine nests normally once the destination is that machine

The retained host-qualified pick is classified against the current destination:
same host gives the ordinary parent fields, another store the cross-machine ref.
Also corrects two stale comments.

* perf(lineage): read only the requested child's cross-store entry

* perf(lineage): request each store's lineage list once at startup

* perf(lineage): cap optional caller-parent inference at 3 s and skip a timed-out local Orca

* perf(lineage): return cached sidebar parents when selector inputs are unchanged

* fix(lineage): snapshot selector inputs without a type assertion

* Route startup lineage and the sidebar lineage memo through their owners
2026-11-05 19:42:19 -05:00
Jinwoo Hong 7854cb61b1 Delete the legacy orchestration takeover guide (#38013)
* fix(automations): say the run's worktree was deleted; document plain run-use recovery

* revert(automations): drop the deleted-worktree wording change (not reproducible)

* docs(orchestration): list Tasks on recovery and note run-use drops other Runs

* docs(orchestration): delete the legacy takeover guide; keep orchestration docs to the normal flow
2026-11-04 01:37:09 -05:00
Brennan BensonandOrca QA c18c358fce Keep child-worktree creation in the caller’s project (#35787)
* fix(cli): keep inferred worktree creation in the caller project

* Fix caller checkout ranking and creation guidance

* Share caller-host containment preparation across CLI selections

* Correct Windows CLI fixture syntax and snapshot command

* Match worktree fixture output to typed execution hosts

* Regenerate bundled workspace placement guidance

* Refuse ambiguous workspace ownership and preserve launch replay

* Scope runtime filesystem mocks to their aggregate tests

* Preserve launch proof semantics under strict test diagnostics

* Keep launch proof imports within their compiler project boundary

* Consolidate journal cleanup test imports

* Adapt launch fixtures to the current host settings contract

* Keep the CLI error formatter under the file-size limit

Move the selector-ambiguity workspace-choice check into selector-ambiguity-recovery.ts, which already owns that error's recovery text. Behavior is unchanged.

* Expect main's concise orchestration output in the workspace-choice CLI tests

Orchestration commands now print the one-line failure on stderr and the workspace choices and next steps on stdout (or compact JSON plus the stderr line with --json), and print concise success fields.

---------

Co-authored-by: Orca QA <orca-qa@users.noreply.github.com>
2026-11-01 13:35:47 -08:00
Jinwoo Hong 0668ba1a33 Orchestration: fix agent instructions that misdirect workers and coordinators (#36803) 2026-10-31 22:40:10 -04:00
Jinwoo Hong 6c2fde67f5 Let send talk to a worker in any state, reopening closed workers for follow-ups (#36533)
orca orchestration send --to dispatch:<id> now works whatever the worker's state: a working worker reads it now; a finished worker still open gets it as a follow-up Task in place; a closed worker is reopened in its saved conversation, or started fresh with the earlier task and report when no conversation is saved. Unread mail at worker_done becomes a follow-up automatically. An older paired server refuses send to a finished worker with an update message.
2026-10-31 02:27:21 -04:00
Jinwoo Hong 99cddd9796 Name worker-keep-open in the orchestration recovery reference (#36599)
The recovery reference now names worker-keep-open instead of the renamed worker-retain. Guide wording only.
2026-10-31 01:38:55 -04:00
Jinwoo Hong 8138232c4f Print concise orchestration CLI output for agents, with --full for the full record (#36498)
Orchestration commands print small flat fields in text and --json by default; --full prints the previous output. Failures exit non-zero with error: <code>: <message> on stderr.
2026-10-30 23:11:56 -04:00
Brennan Benson d879474a03 docs(orchestration): explain recovery after repeated worker startup failures (#36335)
* docs(orchestration): explain user-directed task recovery

* docs(orchestration): correct recovery placement and terminal cleanup
2026-10-30 15:51:30 -07:00
Jinwoo Hong a1f7f97671 Orchestration v3: coordinators end their turn and Orca wakes them (#36137)
* fix(orchestration): keep a terminal's mail from being consumed by another terminal, and let settled structured workers take further tasks

A plain orchestration check consumed the named terminal's mail whatever terminal ran it, so a
coordinator checking a worker's handle took the worker's message. A consuming check from a
terminal whose own live pane is a different terminal's is now refused; --peek and --all still read
it without consuming.

worker-start --terminal and dispatch --inject routed a structured worker (structworker_ handle) as
a terminal and failed with terminal_unsupported_for_agent_session. Any agent-session party now
takes its task as a send into its session, while a structured worker keeps the pane and process
authority its first Dispatch had.

* fix(orchestration): fence check by live terminal incl. acks; defer injecting structured workers

Review fixes for the mail fence and structured worker reuse:
- The check fence compares the caller's and the mailbox's live terminal process instead of
  recorded panes, so exited terminals keep the old behaviour and two processes sharing a
  recorded pane are told apart. It also covers --ack in every read mode. The caller lookup is
  shared with the lifecycle caller fence.
- dispatch --inject to a structured worker stays refused as before: an injected Dispatch takes
  no custody, so releasing the first Dispatch could close a session working on the second.
  worker-start --terminal reuse transfers custody and stays supported.
- worker-start decides pane-less chat authority from the resolved party and passes the session
  down, so delivery and next commands no longer re-derive it from the address spelling.
- Receipts name an agent session, not a chat.

* fix(orchestration): an exited caller's env pane names no caller

The caller lookup fell back to the env pane key whenever the env handle was not live, so an
exited terminal's pane resolved to whatever terminal now occupies it, and a tmux worker's own
check was refused. The pane fallback now applies only when the env handle is not proven exited.

* fix(terminal): keep surviving terminals streaming to main after a restart

Main now attaches every live local daemon session it adopts at startup,
not only ones a window pane or remote view is showing, and re-attaches
after a daemon endpoint drop. The attach snapshot seeds main's screen,
tail and title, and an attached-but-silent agent gets an output clock so
readiness can see it has gone quiet. Waits on a survivor main could not
attach fail at once instead of timing out.

* fix(terminal): restore only pane-less streams after endpoint loss; one attach state

- Endpoint loss re-attaches in main only sessions no renderer view reports; a pane restores its own.
- The attach hide skips views mid-attach and leaves a pane's own hidden claim alone on failure.
- One per-session stream state (attaching, attached, lost) replaces three sets and is forgotten on exit.
- Waits retry the attach before failing, including after a failed re-attach.
- The output clock starts wherever a reattach result is seeded, so pane reattaches get it too.

* refactor(terminal): attach survivors at inventory adoption only; waits on unattached survivors fail at once; one ptyId output clock

* test(e2e): scope the restart attach spec's terminal listing to the unopened workspace

* fix(terminal): narrow survivor attach: waits behave as on main, one quiet clock per generation, keep main's hidden mark on failed attach

* Teach the orchestration guide when to hand off and to end the turn after dispatch

* Rewrap a messaging reference line

* Keep the original placement default and skill trigger; keep the accuracy fixes

* Point DAG retries at --retry-of and drop the last blocking-wait instruction

* Check the served orchestration guide by its new worker heading

* Apply council review: best-effort wake, full completion rule, restored safety rules, fixed scope and settled wording

* Say when the orchestration skill applies, and prefer own subagents otherwise

* Name the polling, steering, checkout, and worker fan-out habits to avoid

* fix(orchestration): wake a coordinator whose turn ended while its program still reports working

Claude Code keeps its OSC 7501 report on working while an agent-teams
teammate is alive, even after the lead's own turn ends. The mailbox gate,
idle edges and tui-idle all read that report, so worker mail was stored and
never typed. A hook turn end newer than the program's working report now
counts as rest; hook rows re-classify the PTY so the existing idle edge
delivers. Agent Teams panes now join their claude hook rows.

* fix(orchestration): compare the lead's own turn end, not the row's receipt

A child's hook restamps the row's receivedAt while the lead's turn is still
the old one, which could mark rest under a newer program working report.
Use the lead's turn-end stamp on this runtime's clock, after the latest input.

* feat(orchestration): a consuming check marks the batch it returns as handled

* Say why watch scripts and open waits block the wake

* feat(orchestration): release settled workers automatically after a grace period

* fix(orchestration): use checked row reads for delivery cursors

* fix(orchestration): trust a live own-process turn end instead of comparing clocks

Drop programActivitySince and the turn-end clock mapping: replays and SSH
host clocks made the comparison unreliable. A Claude lead counts as at rest
under a program working report when its hook turn end is a live transition
from this pane's host and launch, past the existing input and respawn floors.

* test(orchestration): drop cast in ack no-op check

* test(orchestration): seed legacy outstanding batch for held chat nudges

* test(orchestration): built CLI check hands its batch over

* test(orchestration): annotate unchanged casts and stub the keep item in tab menu tests

* fix(orchestration): let real input, explicit release and host releases converge with automatic release

- Record worker takeover at the owning runtime when it accepts UI input, before the bytes reach the PTY.
- Run remote attachment releases through the same per-resource single-flight as local ones.
- Apply the confirmed-release home projection when host observation reports a host-side release, and carry the host deadline in fleet snapshots.
- Prove restart recovery with a file-backed database reopened by a fresh runtime.

* test(orchestration): classify the keep RPC's worker fields instead of counting methods

* fix(orchestration): checked delivery reads and observer and recovery help

* refactor(orchestration): end a check when it hands over a batch; wake rides the wait result

* fix(orchestration): refuse an automatic close after real input, join remote releases, converge the home

- An automatic close checks the terminal's last driving-input time right before closing and keeps a worker someone typed into after worker_done; typing itself does no orchestration database work.
- Remote releases share the per-resource single-flight with local release, so a home worker-release joins the host timer's close.
- Host fleet snapshots and federationShow report released and the pending deadline; the home applies the existing release projection inside its observation fence.
- Real restart tests: a file-backed database reopened by a fresh runtime releases a missed deadline once and finishes interrupted requests.

* fix(orchestration): receipt a shared wake batch on arrival and report the applied Run ack

* fix(orchestration): deliver the wake to a working coordinator; agents queue mid-turn input

The wake was typed only into a live idle pane, so a Claude lead whose program
status stayed working (teammate alive) never got it. Agents queue or steer text
typed mid-turn, so a pane takes mail when live idle or working; dialogs,
permission prompts, exited agents, unsettled panes and cold-parked Enter still
hold it. Drops the held-batch Enter deferral and the per-turn-end hook logic.

* fix(orchestration): record each remote release request before joining, publish archives only through the conditional commit

- Each remote release caller records its own request before joining a close under way, and the close decides on the current row, so an explicit release that overlaps the host timer clears the automatic deadline.
- Remote capture publishes its archive through commitWorkerTerminalArchiveForRelease, so a retain during capture leaves no stale archive for worker-read.
- Remote worker-show projects the fenced host deadline and its next action.
- The input guard lives in its own module instead of the scheduler.

* feat(orchestration): start a worker in one step; worker-start creates the coordinator's Run when none is bound

worker-start --spec now creates and binds a Run for a root coordinator with
no bound Run, through the same path run-create uses, and names the Run it
created or joined in every receipt. Workers on a Dispatch keep starting
helpers in their own Run and never get a new one. A new CLI against an older
runtime runs run-create and retries; old CLIs need no change.

* fix(orchestration): drop the older-runtime fallback; check the spec before a Run is created

The CLI fallback matched a refusal text the client decorates, could not
prove the caller was a root coordinator on an older runtime, and recovered
an inner run-create under the worker-start command. Older runtimes keep
their refusal and the guide says to run-create first there. The prompt
budget check now runs before worker-start can create a Run.

* Route report-back requests to coordinators, complete the steer command, and label the wake loop branches

* Say workers are done only when worker_done confirms, wait for helpers to settle, and let bare-shell workers exit

* Route report-back requests away from handoff and steer supervised workers through their mailbox

* docs(orchestration): teach auto-handled check, retry-request and inbox recovery in the guides

* Describe the mid-turn wake in the messaging reference

* docs(orchestration): say terminal workers are released 2 minutes after worker_done

- The guide's Completion section, the reuse recipe and the recovery reference now describe the automatic release and its autoReleaseAt deadline: reuse or worker-retain before it; explicit worker-release still works and is required for chat workers and workers without a deadline.
- The worker preamble says the terminal stays available until Orca releases it, about 2 minutes after worker_done unless retained.
- Regenerated the bundled guides.

* feat(orchestration): receipts say when to end the turn

* feat(orchestration): tell the coordinator when a worker stops without worker_done

* test+docs: worker stop reports

* fix(orchestration): judge the empty-check stop with execution-host verdicts

* test: drop casts from worker stop tests

* Say that report-back requests still mean end the turn

* Say that report-back requests still mean end the turn

* fix(orchestration): stop reports wait for children, keep their Dispatch, and relay intact

* fix(orchestration): a stop belongs to the Dispatch active when it was seen; restored stamps announce nothing

* fix(orchestration): never say to stop while a remote worker runs

* Shrink the orchestration guide to the end-state workflow

* fix(orchestration): stop reports quote the agent's last words, or the screen for errors and prompts

* fix(orchestration): withhold the stop while chat failure mail is off

* test: a clean finish quotes the agent's last words

* fix(orchestration): stop asking workers for heartbeats

Workers are no longer told to send periodic heartbeats. A heartbeat that
still arrives from an older worker or peer only records last_heartbeat_at:
it is stored already handled, so it never lands in a coordinator inbox and
never wakes a coordinator or a check --wait. Removes the stale-heartbeat
warning and the heartbeat interval plumbing that only fed the preamble.

* fix(settings): remove the worker heartbeat controls

The Worker heartbeats switch and Heartbeat interval field only changed the
worker preamble text, which no longer mentions heartbeats. Saved values are
dropped on load as retired settings.

* fix(orchestration): wake a coordinator that reports program status but never sets a title

Program status only re-classified a pane that had a title, so a titleless
agent (agy) never got a live status and the wake gate refused it.

* fix(orchestration): a new Run no longer cuts a coordinator off from its earlier Run

* test(orchestration): drop heartbeat-batch cases the handled heartbeat row removed

* fix(orchestration): redact capabilities from a stop report's quoted output

* fix(orchestration): never wake a mailbox for a heartbeat

One rule at the runtime notification entry replaces per-call-site read
checks, so mutation replay and federated control-mail import stay silent
for heartbeats too.

* Shrink the orchestration references and worker preamble to the end-state workflow

* Name the older-Orca bind refusal as worker-start prints it

* Owner-fence test: a refused foreign ack changes nothing, whatever the worker's own check already handled

* Integration: drop the duplicate consumingCheck after merging #34043 and #35293

* Integration: await the worker-start next receipt before naming its Run

* Apply review round 1: drain to zero, release without auto-release, retry after stop, Run before a DAG

* Apply review round 2: start ready DAG Tasks, block when no failure wake, replay sends by request ID

* Apply review round 3: one receipt-based wait rule for starts and wakes

* fix(orchestration): a worker stopped without reporting needs action, and stop keeps its output

worker-list projected a stop-reported attempt as live, in_progress, no action. It now
projects finished_unverified with a worker-read recovery until mail reaches the worker.
worker-stop closed the terminal before anything archived it, so release reported the
archive unavailable; stop now archives the output first.

* Revert "Owner-fence test: a refused foreign ack changes nothing, whatever the worker's own check already handled"

This reverts commit 336ff91737.

* Revert "Merge #34043 (terminal mail guard, chat worker reuse) into integration"

This reverts commit 9c01b1a06e, reversing
changes made to d4b0c6c5f7.

* Revert "Merge #33720 (restart: attach surviving terminals) into integration"

This reverts commit d4b0c6c5f7, reversing
changes made to 53f9735bfe.

* Integration: restore #35293's check-methods after removing #34043

* fix(orchestration): bound the pre-stop archive and keep it through retain

* fix(orchestration): a failed pre-stop archive never blocks the stop

* fix(orchestration): answering a question says to end the turn while the worker runs

* fix(orchestration): a settled worker whose terminal moved owes nothing

* Cut the orchestration guide to what Orca does not already enforce or explain

* Drop the terminal-send recovery the CLI already prints

* Restore a short never-do block: no polling, no terminal steering, no hand-made checkouts

* Trigger on named-agent review requests, define failure in specs, show the reuse commands

* Restore six guide rules from the cut review and drop the preamble heartbeat block

* Integration: drop the removed heartbeat interval from preamble tests

* Drop the outside-Orca wait and run-use lines; Orca's errors teach both

* Release finished chat workers like terminal workers unless the user touched them

A finished chat worker now closes ~2 min after its accepted worker_done, like a
terminal worker. Clients report a touch when a chat is on screen at rest (opened
idle, or its turn ends on screen); the owning host keeps a finished worker's
chat on that report, until its automatic release commits the archive and closes
in the same tick. Chat workers arm only while every connected client advertises
the touch capability; old clients keep today's manual release.

* Decide an automatic chat close in the session lane, recheck clients there, re-touch after reconnect

The automatic close now asks the database in the chat's session lane right before
Stop: a touch until then keeps the chat, and a client that cannot report touches
(joined during the grace or after a restart) sends it back to manual release.
Touch reports wait for a read chat, so a reconnect re-reads and reports again.

* Keep explicit chat releases eager and re-touch a desktop chat after lost contact

* Resolve a chat touch by durable worker identity, so a closing chat's hidden tab stays touchable

* Keep the mobile touch report free of route and app-state hooks: shown counts as seen

* List workerChatTouch's identity fields and the close's optional admission in their census tests

* Release a settled remote worker from the home when its host predates automatic release

* Replace Keep Worker Open with an auto-close setting, and tell agents when an older server needs a Run

The coordinator's runtime decides whether finished workers close: it gates local arming, the
settlement ack's autoRelease opt-in to a remote host, and the home deadline for older hosts.
worker-start and run-create name an older server's one-Run-per-coordinator behaviour instead
of failing or rebinding silently.

* Escalate a parent's prompt or error even while its helper is open

* Report a chat touch when a worker chat is opened while working, and on each turn edge

* Record worker takeover when the host admits a person's chat send, so mobile sends keep the worker

* End a reported worker stop when the worker resumes, not only when Dispatch mail arrives

* Recheck a chat mail notice at admission so mail a check consumed starts no turn

* Type the resumed-turn stop test's pane move directly

* Keep the stop decision in an empty check, report muted chat failures as stops, and bind a new Run only once a start passes preflight

A start or reply receipt now sends the caller to drain its mail with check instead of ending its turn, so a mid-batch answer cannot skip a later report and a replayed receipt carries no stale stop. With chat error mail off, a failed chat request still reports a plain stop through the existing stop report path. worker-start creates and binds its Run just before the Task and Dispatch, so a start refused in preflight leaves no Run behind. worker-release help no longer says chats never auto-release.

* Reuse a Run a parallel first start bound, refuse dependencies before creating a Run, and relay a paired-server worker exit home as a stop report

* Read a reported worker stop against live agent status instead of recording resume observations

A stop report is mail the coordinator reads once. Whether a worker shows as
finished now comes from its live status: a fresh working agent status (local
pane, chat session, or the execution host's fleet snapshot) outranks the
report, so nothing needs clearing on resume.

* Drop agent steps Orca already covers: start receipts say Orca wakes you, a check asks for another only when a capped batch leaves mail, workers get no mail-check or ack lines, and chat errors always report with the exact error

* Keep the home's remote auto-release deadline until the host answers, and let a retain win at the send

Claiming a due deadline postpones it instead of moving it to a releasing state, so a restart or
a failed attempt runs it again; the release send rechecks the deadline with no await before it.
A host that cannot be asked at settlement still gets the deadline, and a host upgraded since
settlement is released by the home, since it never armed its own.

* Close a finished worker automatically only if the user never opened it

Seeing a worker's terminal or chat now writes the user's Keep record (owned,
retained) on its host, at any point before its close is admitted, so a worker
seen while it worked is never armed and a pending automatic close is cancelled;
the coordinator can still release, stop or re-engage it. Desktop reports from
the on-screen attention scan for every on-screen terminal pane and chat once a
present user is on that workspace; mobile reports the tab a session opens on
and each tab switch. The weaker chat-only touch hooks are removed. A chat send
admitted after the automatic close committed its archive now keeps the chat.

* Show only live facts for a worker that reported a stop

A stop report is escalation mail that wakes the coordinator once. worker-list
and worker-show no longer project it as finished_unverified with a recovery
action, so the live-working override that cleared that label goes too.

* Report a paired-server worker's process exit home even while its ask or check --wait is still registered

* Report a seen worker until its host records the keep, including chats shown before custody and mobile auto-selections

The touch reply now says pending while a structured worker's custody row is
not yet recorded; desktop and mobile keep a subject unreported after a
failed or pending reply and retry on the next scan, snapshot or reconnect.
Mobile reports whichever terminal or chat is displayed, tapped or selected
by a snapshot, instead of only the first per workspace.

* Record a person's chat send or command takeover inside the session lane

Takeover moves from the RPC wrappers into the host's in-lane acceptance
of userSend sends, /compact and /clear, so a close queued behind an
accepted send cannot admit before the keep, including during /compact's
handover wait. Host orchestration sends never set userSend.

* Say which stop reports a deliberate wait suppresses

* Align merged tests with the shorter preamble and the fixed start hint

* Retry an unrecorded mobile worker touch when the phone reconnects

* Drop the removed worker check line from the CLI boundary test

* fix(orchestration): wake a coordinator that survived an app restart

After a restart, main typed the mail wake only into an agent whose status it
had seen live since starting, so a coordinator whose turn had already ended
never woke. At reattach, take the agent's own last hook report, which the hook
server restores from disk and from its offline spool, as that observation while
an agent still holds the surviving process.

* fix(orchestration): restore the lead's own turn, hold a restored draft, fence respawns

Read the lead agent's turn from the restored report so background work does
not hide an ended turn, keep the wake out of a composer that still holds text
typed before the restart, and drop a restore that a respawn has overtaken.

* fix(orchestration): restore a reattached status only from a current screen without a draft

The whole-screen accessor hides a model the provider still owns after attach, so a
draft there read as none. Read the screen the way wait gates do, provider image
included, and decline the restoration when it is unavailable.

* fix(orchestration): restore a reattached status without reading the screen

After a restart main has no record of typing, so the existing draft hold cannot
apply; a screen read here could miss a draft and, when slow, strand the
coordinator for good. A draft left before the restart now meets the same rule
as an abandoned one.

* fix(orchestration): let a slow foreground probe still restore a reattached status

A probe answering after the interactive two-second bound was dropped for good,
stranding the coordinator. Restoration has no caller waiting, so it takes the
real answer; the existing fences still guard application.

* test(orchestration): mark the slow foreground probe as fake-clock driven

* Report a terminal worker whose agent died while its shell lives

A killed agent sends no stop hook and its shell keeps the PTY open, so its
Dispatch stayed open and the coordinator was never woken. The shell's
command-finished already retires the PTY's launched-agent authority; the
panes it retires now go to the worker stop reporter as an 'exited' stop
(same mail and wake as a PTY exit, once per Dispatch, none once settled).

* refactor(orchestration): key mail, waits and settle checks by terminal

Reapplies #34056 on the integrated orchestration work: mail delivery, the
settle recheck and terminal waits are keyed by ptyId instead of the window
pane, deleting the pane-keyed leaf lane and the headless-only PTY lane.

Keeps #34706's delivery rule (a terminal taking input, idle or mid-turn,
gets the wake; dialogs, permission prompts and unsent drafts hold it; the
cold-park fence still defers a parked Enter) and #35274's multi-Run
coordinator mailboxes.

* fix(orchestration): keep best-effort delivery's catch; stage the remount e2e

deliverForHandle keeps the integration branch's catch: ownership resolution
still reads SQLite, and a notification queued before teardown must not throw.

The remount e2e now asserts each stage on its own (mid-turn mail before the
workspace opens, the wait across the remount, mail after it), since #34706
types mail mid-turn and the old single mail check could pass before the
remount happened.

* refactor(orchestration): settle a new terminal wait from its first reading

A wait now keeps the PTY record it resolved, and nothing runs between the
first evaluation and the waiter's registration, so the second evaluation
re-read the same evidence. Starts the poll from the first verdict instead.

Deletes the leaf wait-result builders the leaf lane left without callers,
and adds a unit test that a tui-idle wait survives its pane remounting
(the e2e cannot order the wait's registration before the remount).

* Report a dead worker agent only once its shell is proven alone

A command-finished marker alone does not prove the launched agent exited: a
nested interactive shell in a live worker emits the same marker. The exit
report now waits for the execution host's shell-alone proof and is fenced to
the Dispatch and launch seen at the marker, so a relaunch in the pane is not
reported by an older completion. It no longer hangs on launch-authority
retirement, so a nested completion that retires metadata does not stop a later
real death from being reported. The agent-exited fact feeds the same path for
shells without OSC 133.

The shell-alone report and a paired-server worker's PTY exit share one exit
entry point and key, so the Run home gets one report per process.

* fix(orchestration): resolve a terminal's mailbox across every pane that shows it

Two panes can show one PTY; ownership read only the first listed pane, so a
Run bound to the other was never woken. The mailbox terminal now carries
every pane alias and ownership, discovery and the delayed-Enter recheck
consult them all.

* Report a killed worker agent from its launch's exit receipt

The launch command now ends with a per-shell marker the shell prints once
the agent returns, however it ended. The execution host scans it before
thinning (main and daemon), checks it against the PTY's current launch and
incarnation, and reports through reportWorkerExit. A nested shell has no
receipt and a relaunch names a new one, so the shell-alone inference is gone.
A later local PTY exit reuses the same exit message id, so it settles the
Dispatch without a second report.

* Revert "Revert "Merge #33720 (restart: attach surviving terminals) into integration""

Reapplies survivor attach (72b3b79fd3, fe906ea2a2, f46777ba18) so the
#33720 merge that follows carries it; integration reverted it in 2f945fa715.
The output clock now feeds v3's terminal-keyed wait path; the leaf lane it
also fed is gone in v3.

* test(orchestration): give exit-path DB stubs the remote-attachment lookup

The paired-server exit relay now asks the Orchestration DB for an active remote
attachment on every local pty exit; partial stubs without it threw.

* Report a worker killed after a restart from its Dispatch's launch commitment

The exit receipt ID is now derived from the launch-token hash a Dispatch
persists, so a PTY reattached after restart (no launch token in memory)
accepts its receipt when the pane's active Dispatch committed that launch
in the same shell process.

* test(orchestration): stub main's session-dispatch presentation read in the stale-leaf DB

Main's projection scope now reads listSessionDispatchesForPresentation on every
projection; v3's hand-rolled stub predates it.

* test(orchestration): drop the stale-leaf stub key integration's merge now supplies

* test(orchestration): align main's new tests with integration's mid-turn mail, admitted close and nameless worktrees

Main's titleless-turn test assumed mail waits for Stop; v3 types it into a working agent.
Main dropped the --name requirement, so that preflight case now fails on the missing repo.
The evict close now forwards the session lane's admitClose.

* perf(orchestration): read the next federated release deadline through its partial index

The armed-state filter alone cannot use the non-null partial index, so every rearm walked retained federation history.

* perf(terminal): keep unviewed restart survivors to the daemon's live window

Main now attaches every surviving terminal at startup; each mirror took the full
5,000-row depth with no viewer. A survivor nobody views keeps the 1,000 rows the
daemon itself holds, and grows to full depth from those rows once a view subscribes.

* chore(orchestration): clear check:changed findings on the main merge

Format a listing statement, prune the flaky-pattern baseline, and justify or remove the
test casts the changed-code gate flags.

* Revert the unviewed-survivor mirror cap: it can lose desktop history after a restart

A capped survivor mirror is still served as complete history to desktop recovery, and a desktop pane
added after inventory never lifts the cap. Restart memory for unviewed survivors moves to a follow-up
that defers the screen model instead of truncating it.

This reverts commit 4ea934ca7f.

* Key main's new null-agent idle-wait test by terminal, as integration's waits are

* fix(orchestration): type mid-turn mail only into a recognized agent

An unrecognized terminal's working verdict may be a running program reading stdin,
so null-agent panes keep waiting for quiet before the pointer is typed.

* Remove killed-worker exit receipts from orchestration v3

The launch-bound receipt added a visible printf to every agent launch line. Owner decision: ship v3
without killed-worker detection (behaves as main does today) and revisit it later, e.g. by watching the
agent's hook-reported process.

This reverts merge b9a30b6e74.

* Stop re-arming a delivery recheck once the quiet window has passed

A refusal that outlives the quiet window (an unrecognized terminal still reading as working) is not
time-bound, so a 50 ms recheck spun forever. The next output, title or mail edge rechecks it.

* Keep retrying deliverable mail past the quiet window; gate mid-turn delivery on terminal agent identity

A refusal outliving the quiet window can wait on a hook final-frame deadline or a late foreground
recognition, neither of which emits a delivery edge, so poll slowly while that terminal still has
deliverable mail and stop once none is pending. Mid-turn delivery now asks whether the terminal agent
is recognized at all, not whether it has readiness rules, so a recognized reusable agent keeps it.

* Keep structured-agent-session.ts within the 300-line cap

* fix(orchestration): arm delivery rechecks only for actionable delivery work

The recheck predicate now counts persisted pointer reservations awaiting
recovery as well as selectable new mail, and gates every recheck (quiet
window, post-quiet, and holds) on it, using the canonical output clock.

* Check the served orchestration guide by a heading the v3 guide keeps

* test(e2e): launch the mail-pane fake under its agent CLI's name

v3 types mid-turn mail only into a recognized terminal agent, and a title alone no longer
recognizes one, so the fake runs as `node …/codex` (or cursor-agent) like an npm shim.
The hold-while-working case now models the unrecognized program that contract still holds for.

* test(e2e): align idle mail delivery spec with v3 auto-ack, mid-turn Enter and multi-Run coordinators

A consuming check now marks its batch handled (8fbb395a32), so the live
waiter's mail is pulled once returned and the held-batch case is seeded
as an older Orca leaves it. Agents queue mid-turn input (e0cb9a68d2), so
the held reminder's Enter is no longer withheld. A coordinator keeps its
earlier Run (#35274), so Run A direct mail is pointed with --run A.

* Wake a Run's coordinator through its pane when the Run's stored terminal handle belongs to the pane's previous PTY

* Make check --ack acknowledge only, so it never takes new mail

A check carrying --ack acknowledged that Delivery and then took (and,
since checks now acknowledge what they show, consumed) the next batch.
A caller that discarded the output lost that mail. --ack now returns
after acknowledging; newer mail stays unread, so the wake path still
fires and the next plain check shows it. The receipt says --ack is not
needed. Removes the ack-then-take receipts this made dead.

* Print every message body and payload in a plain orchestration check, since current hosts acknowledge the batch on read

* Keep acknowledge-then-take for check --ack --wait, and reject --ack on direct mail

A waiting caller reads what the wait takes, so --ack --wait acknowledges
and then waits for and takes the next batch, as before; --ack alone
stays acknowledge-only. Direct mail has no Delivery, so --ack there now
fails as a foreign delivery instead of consuming the mail.

* Keep a user-kept worker terminal kept when a new Dispatch reuses it, since its on-screen pane reports only once

* Tell a coordinator to check a remote worker itself when its host cannot report turn stops

* Find a Run's pane terminal for mail without re-issuing its handle mid-reload

* Keep the original ack-then-take path for check --ack --wait

Composing an ack with a second check lost the acknowledged consumer
generation and persisted subcall receipts. --ack --wait now runs the
original single-generation path unchanged; only --ack without --wait
returns after acknowledging, and direct mail still rejects --ack.

* Settle the e2e agent idle before newer mail so the mid-turn pointer cannot race the idle title

* Record a user's view of a worker terminal as user_seen so only it, not a coordinator's retain, survives reuse by a new Dispatch

* Let a worker ask, end its turn, and be woken with the answer

* Drop casts from the ask tests and the pending-question query

* Tell agents to run a plain check after acknowledging a held Delivery

--ack without --wait no longer takes newer mail, so the held-delivery
pointer now names the follow-up check for the new messages.

* Deliver Run and Dispatch mail to the pane's current PTY, issuing its handle only when none exists

* Wait for a PTY's exported handle before mail, and wake it when that handle is adopted

* Keep a cancelled ask from taking its answer, scope the asked-question stop mute to the asking turn, and end a waiting ask when its Dispatch settles

* Order the asked-question stop mute by message sequence, not same-second timestamps

* Wake a PTY's deferred mail whenever it gets a handle, by any path

* Make the headless Dispatch adoption test hold in both layout modes

* Mock the finished-worker strip states with neutral pane and chat notice fixtures

* Mock the temporary-worker strip with the owner's copy and the Allow auto-close tab menu item

* Mock chat workers with a goal-style composer strip instead of the composer notice

* Keep a worker open only by Keep open; hold its countdown while its agent works again

* Remove viewing and typing as implicit keeps on desktop and mobile

* Update the routing specs to acknowledge a held Delivery, then take with a plain check

* Show the temporary-worker strip with Keep open and its countdown, and Allow Auto-Close in tab menus

* Share the worker auto-close poll and key its read to the subject

* Re-read kept workers often so Allow Auto-Close shows its countdown promptly

* Hold auto-close through every turn, refuse stopping kept workers before anything changes, and read worker state only for known workers

* Keep a claimed close final, check the agent before claiming, refuse remote stops without a Keep open answer, and keep re-reading known workers

* Classify workerChatTouch caller fields and drop a test-only cast
2026-10-30 14:20:02 -04:00
Jinwoo Hong 93ff4832e4 feat(orchestration): worker-start new worktrees default to a generated name (#35363) 2026-10-28 17:51:08 -04:00
BingZandNeil 7e4993e255 docs(orchestration): document worker-stop terminal-close contract (#13198)
Explain the worker-stop receipt's processAction values in the recovery
guide and command notes: closed_agent_terminal means Orca already issued
the close (no follow-up terminal close), and stop_unknown exits 1 while
possibly still reporting closed_agent_terminal.

Fixes #13197

Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com>
2026-10-28 01:22:08 -07:00
Naveen ChatlapalliandNeil dad5c7ddb4 fix(orchestration): teach workers to acknowledge and drain mail batches (#26768)
Dispatched workers were told to run `orchestration check` but never to ack
the returned deliveryId, so the same batch replayed on every check and newer
coordinator mail stayed hidden. The dispatch preamble, orchestration skill
guide, and worker contract now tell workers to ack each processed batch with
`--ack <delivery_id>` (which also reads the next one) and repeat until count
is 0.

Co-authored-by: Neil <neil@stably.ai>
2026-10-26 08:00:46 -07:00
Brennan Benson f71120551f Stop workers from taking over the Run they work for (#29008)
* fix(orchestration): refuse run-use by a worker on the Run it works for

A worker holding an active Dispatch in a Run could bind that Run as its coordinator.
That silently unbound the real coordinator, remembered the worker's address as a
coordinator address (so the worker's own mail in that Run routed to the Run mailbox
from then on), and when the worker later bound its own sub-Run, left the parent Run
with no coordinator at all.

bindRun now refuses a caller whose handle holds an active Dispatch in the Run, with
worker_cannot_coordinate_own_run, before any effect. Creating its own Run for
sub-workers is unchanged.

* fix(orchestration): tell a worker naming its own Run where its mail is

Workers that made a sub-Run ran `check --run <parent>`, were told "This
coordinator terminal is bound to X, not Y", and ran run-use on the parent
to "switch back", unbinding their coordinator. That refusal now says the
Run belongs to their coordinator, that their own mail arrives through
plain `check`, and that they report with `send`; it never mentions run-use.

The run-use refusal now matches the caller the way check and send do
(address, then pane, so a reminted terminal handle is still caught),
also covers Runs above the caller's own through recorded creator_dispatch_id
lineage, skips context-only self-dispatches so a coordinator can still
rebind its own Run, and passes its code through the RPC wire.

* feat(orchestration): refuse run-use over a live coordinator without --takeover

run-use silently took a Run from whoever coordinated it, even a chat or
agent that was still running. It now refuses with run_coordinator_live
when there is positive evidence the current coordinator is running (its
chat is still reachable, or the agent hooks prove its terminal agent
live) unless the caller passes --takeover. Anything unproven (closed chat,
exited or unhooked terminal, another host) rebinds as before, so restart
handoffs need no flag. The probe is async in the RPC handler; bindRun
enforces it only if the Run is still bound to the probed party.

A caller working a Dispatch can never take over, with or without the
flag (worker_cannot_take_over_run). The help text now says run-use is
for resuming a Run and that workers never use it.

* fix(orchestration): stop pointing workers at run-use

Workers learned run-use from error text: "No Run is bound. Use run-create
or run-use first", "Rebind with orchestration run-use", and "Rebind this
terminal with orchestration run-use". Each now says what the caller can
do instead. An unbound worker is told its mail arrives through plain
check and it reports with send; a displaced coordinator is told to stop
and take the Run back only if the user asks. The orchestration skill now
says run-use resumes your own Run, --takeover needs the user's word, and
workers never run it.

* fix(orchestration): stop routing mail to addresses that grabbed a Run's seat

Every address that ever bound a Run is remembered as one of its
coordinator addresses, so mail to it routes to the Run mailbox. When a
worker bound the Run it was working for, its own address joined that
list for good, and its direct mail in that Run kept landing in the
coordinator's inbox. The live database has 45 such rows since
2026-08-20.

All four readers of the remembered addresses now skip one that was first
bound while it worked an unsettled, delegated Dispatch in the same Run.
It is derived from existing rows, so no migration or delete is needed and
rows written by older builds stop mattering. The active-Dispatch check
also matches a worker's orca_session_id spelling, not only its handle.

* fix(cli): keep the orchestration spec file within its line limit

* fix(orchestration): read remembered coordinator Run ids without a type assertion

* fix(orchestration): compare seat-grab times at whole seconds

first_bound_at has no milliseconds, so a worker bound in the same second its
Dispatch settled was counted as grabbing the seat. Compare completed_at at
whole seconds with a strict >.

* fix(orchestration): a worker coordinates only Runs it created

The worker refusals depended on proving the target's coordinator alive, so
a worker could still take any Run whose coordinator was closed, unhooked,
remote or headless. Replace them with one rule derived from existing rows:
a caller working a Dispatch may bind only a Run it legitimately
coordinated before (a remembered, non-seat-grab address of that Run, or a
Run it dispatched work into), never the Run it works for, and never with
--takeover. A same-binding no-op is never refused, and workers are
classified before the legacy gate so they never see its
--takeover-legacy hint. The refusal names the Dispatch and how to end it
with worker-abandon if it is stale. The ancestor walk is gone.

run-use now records which coordinator it probed. If the binding moved
before the transaction, it probes again (three tries) and then refuses
with a retryable run_binding_changed instead of binding over a party
nobody probed. run_coordinator_live names the chat by its title and
worktree, or the terminal by its title and worktree, says it is still
open, and that closing it clears the refusal.

* fix(orchestration): say what happens to mail on takeover and stop naming run-use to readers

The help now says --takeover replaces the coordinator and replays its
unacknowledged mail to you. A replaced mailbox reader is told only that
another session coordinates the Run and to stop reading it.

* refactor(orchestration): throw one non-null refusal from the live gate

* refactor(runtime): host the coordinator presence probe beside the hook presence seam

Main moved the orphan-adoption runtime file; keeping the one-line probe next
to checkHookAgentPresenceFn leaves this PR off that file.

* fix(orchestration): use findLast for the worktree name

* refactor(runtime): move the coordinator presence probe beside the pty presence recheck

Main is moving the runtime construction files; the probe now sits with
recheckHookAgentPresenceForPty, which reads the same seam, so this PR stays
off the files being moved.

* fix(orchestration): keep workers out of their own Run seat

* fix(orchestration): clarify worker Run handoffs in skill

* test: align explicit Run gate refusal wording

* Move coordinator presence beside terminal presence checks

* Walk complete worker creator lineage when binding Runs

* Restore main documentation after merge hook formatting

* perf(orchestration): check run bindings without terminal snapshots

* test(orchestration): narrow SQLite instrumentation before access

* test(orchestration): import the explicit RPC methods module

* test: cover orchestration restart and continued Run handoff

* test: place orchestration CLI integration in CLI project

* test(cli): use explicit module extensions in restart handoff test

* test(orchestration): isolate source CLI integration in subprocesses

* refactor(orchestration): leave the live-coordinator takeover rule to #29292

#29008 is the worker fix: a worker cannot bind the Run it works for or an
ancestor, cannot abandon itself to get around that, and addresses it held while
grabbing a seat stop redirecting its mail. The separate rule that refuses any
second coordinator while the first is still open (the presence probe, the
--takeover flag and the run_coordinator_live / run_binding_changed refusals)
moves to #29292, which models Run ownership and hand-over and already builds on
it. run-use binds synchronously again, as on main.

* test(orchestration): cover delegated worker legacy takeover refusal

* test(orchestration): explain source CLI integration timeout
2026-10-26 00:46:27 -07:00
Jinwoo Hong 2f25699094 fix(orchestration): let a new check --wait start while an old one is still waiting (#33013)
Fixes #30895
2026-10-25 15:00:51 -04:00
NeilandClaude 3d6306789a fix(cli): reject ask waits above the documented 30-minute maximum (#32044)
* fix(cli): reject orchestration ask waits above the documented maximum

* fix(cli): keep ask timeout help ASCII and trim redundant cases

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-10-22 21:22:50 -07:00
Brennan Benson 1cf24e5e6f fix(orchestration): let a worker start helpers on the Run it works for (#29205)
* fix(orchestration): let a worker start helpers on the Run it works for

A dispatched worker could only start helper workers by taking a coordinator
seat on a new Run, and taking a seat moved its inbox. worker-start now also
accepts a worker on an active Dispatch: its helpers run in the same Run, report
to it (worker_done, heartbeat, escalation, status, ask), and it answers their
questions. The worker's own mailbox never moves. When it settles, whatever its
helpers sent that it never handled goes to the Run's coordinator.

Authority comes from the worker's own Dispatch, re-checked inside the
reservation transaction. Helpers are fresh agents on this host; a worker may
retry only its own helper's latest attempt. Remote workers keep the old
own-Run instructions. No saved field or wire change.

* fix(orchestration): read helper rows without type assertions

* fix(orchestration): hand a settled helper creator's mail to the Run as live delivery

When a worker that started helpers settled, its helpers' unhandled mail was
re-addressed to the Run by settlement SQL alone. Nothing woke the Run's
coordinator, delivered rows kept their old delivery stamp so pointers and the
restart scan skipped them, payload-free status reports were left behind
because selection required payload.dispatchId, and a question asked while the
creator was being stopped stayed with the stopped creator.

- Select handed-back mail by its recipient mailbox and Run (unread reports
  plus pending questions, even ones already read), reset delivery and pointer
  state on the moved rows, and fence the old mailbox's Deliveries.
- After the settling transaction ends, the runtime re-reads exactly the moved
  rows and wakes the Run with their real types; a rollback wakes no one and a
  failure falls back to the existing repoint retry. Restart recovery is the
  existing unread-undelivered scan.
- Hand mail back again at the final stop settlement, which also closes the
  stopped Dispatch's own questions asked during the stop.
- A worker_done receipt replay re-offers the handed-back targets.
- Scope markAsDelivered to the mailbox it pointed, so a late pointer stamp
  from the old owner cannot hide mail now owned by the Run.

* test(orchestration): drop an unused inbox type

* fix(orchestration): read only the Run when handing a settled worker's mail back

Every settlement runs the hand-back, so a full dispatch row read there added a statement per settlement. Also expect the mailbox-scoped delivered stamp in two delivery tests from main.

* fix(orchestration): re-offer handed-back Run mail on every worker_done replay

A helper creator's worker_done hands its helpers' unread reports and pending
questions to the Run. The send stored the Run's extra wake targets in the
mutation receipt, but the executor rewrites that receipt on completion and on
each replay, so an identical retry only woke the original destination.

Derive the targets at replay time instead: for the settled Dispatch (when it
has same-Run helpers), notify run:<id> for each type of unread report or
pending question there. Notification only; nothing is re-settled or inserted.
The receipt augmentation is removed.

* style(orchestration): format with the formatter version main now pins
2026-10-22 14:43:06 -07:00
Brennan Benson b70f13ff89 fix(orchestration): stop retried failed worker starts from piling up live agents (#27795)
* fix(orchestration): stop retried failed worker starts from piling up live agents

A failed worker-start keeps the terminal it created (a pane the user typed
into is theirs), so a coordinator retrying a start that can never succeed
left one more live agent behind on every retry, with nothing to stop it.

- A --retry-of start now releases the failed attempt's still-owned terminal
  through the normal worker-release path (output archived, user takeover
  kept, recovery finishes it after a restart).
- A failed start now counts toward the Task's existing three-failure circuit
  breaker, carried across retries; a retry of a circuit-broken Dispatch is
  refused with a plain error.

Linear: STA-7113

* test(orchestration): type the failed-start retry fixtures without casts

* fix(orchestration): carry failed-start count only along --retry-of; plain breaker refusal

- Only a --retry-of start inherits its predecessor's failure count, so worker
  exits across separate starts keep the old behaviour.
- The circuit-broken refusal carries failureCount and next steps that say to
  stop retrying, read the last attempt, and that its pane stays open.
- A repeated failed report on a circuit-broken Dispatch is a duplicate.

Linear: STA-7113

* fix(orchestration): breaker refusal only points at a terminal that is still open

Linear: STA-7113

* fix(orchestration): log a retry's release that could not prove the old terminal closed

Linear: STA-7113

* Preserve worker diagnostics before terminal release and mask wrapped tokens
2026-10-13 15:20:19 -07:00
Brennan Benson 564693f176 fix(cli): honor Chat UI setting for worktree agent creation (#27796)
* fix(cli): honor Chat UI setting for worktree agent creation

* fix(cli): preserve older host launch and worktree lineage warnings
2026-10-12 02:14:43 -07:00
+2 8e5080c132 Validate OpenCode worker model preferences on the execution host (#24624)
* feat(orchestration): support OpenCode worker model selection

Allow supervised OpenCode workers to use per-launch model overrides through the existing launch-preference and receipt path.
Preserve existing OpenCode agent arguments while replacing only model flags, and reject unsupported effort values explicitly.
Keep Native Chat option exposure unchanged and update the worker CLI and orchestration guidance.

* fix: wait for OpenCode worker composer before first dispatch

Reuse captured composer readiness on local and paired execution hosts and revoke launching-shell paste anchors.

Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>

* feat(opencode): probe execution-host CLI capabilities

* fix(opencode): select plugin default for execution host loader

* fix(opencode): limit prompt prefill capability to verified release

* feat(orchestration): gate OpenCode worker model preferences by host capability

Co-authored-by: user141514 <user141514@users.noreply.github.com>

* feat(opencode): probe launch capabilities on the execution host

* feat(opencode): probe execution-host CLI capabilities

* feat(opencode): probe launch capabilities on the execution host

* feat(orchestration): resolve explicitly configured command aliases

* fix(opencode): select plugin loader for the launched host binary

* fix(opencode): match WSL probe cwd and declared guest environment

* fix(orchestration): verify available OpenCode model on execution host

* fix(opencode): preserve launch environment deletion boundaries

* wip(opencode): authorize native startup prompt intent at execution owner

* fix(opencode): atomically replace status plugin entrypoints

* fix(opencode): retain plugin permissions across restrictive umasks

* test(opencode): resolve permission fixture from primary cwd

* feat(opencode): install startup prompt plugin independently of status hooks

* fix(opencode): wait for admitted startup intent and preserve failed-launch briefs

* fix(opencode): confine overlay manifest cleanup to owned directories

Co-authored-by: Adnan Khan <adnank11427@gmail.com>

* fix: wait for OpenCode worker composer before first dispatch

Reuse captured composer readiness on local and paired execution hosts and revoke launching-shell paste anchors.

Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>

* feat(opencode): probe execution-host CLI capabilities

* fix(opencode): select plugin default for execution host loader

* fix(opencode): limit prompt prefill capability to verified release

* feat(opencode): probe launch capabilities on the execution host

* fix(opencode): select plugin loader for the launched host binary

* fix(opencode): match WSL probe cwd and declared guest environment

* fix(opencode): preserve launch environment deletion boundaries

* wip(opencode): authorize native startup prompt intent at execution owner

* fix(opencode): atomically replace status plugin entrypoints

* fix(opencode): retain plugin permissions across restrictive umasks

* test(opencode): resolve permission fixture from primary cwd

* feat(opencode): install startup prompt plugin independently of status hooks

* fix(opencode): wait for admitted startup intent and preserve failed-launch briefs

* fix(opencode): unsubscribe hook settings during async host shutdown

* fix(opencode): confine overlay manifest cleanup to owned directories

Co-authored-by: Adnan Khan <adnank11427@gmail.com>

* test(readiness): census recorded OpenCode composer boots

* fix(opencode): reject redirected overlay parents before cleanup

* fix(orcad): retain runtime cleanup when subscribing to hook settings

* refactor(launch): extract OpenCode config and attachment authority

* fix(opencode): retain host version selection across relay restarts

* fix(opencode): pass run prompts as positional messages

Preserve run flags and use the existing shell quoting and run-command detector
to append the initial message after --, reusing an existing separator.
TUI launches retain their version-selected prompt transport and draft behavior.

Original run-order work: @coelho-doti (#13065, tracked in #17551).

* fix(opencode): keep wrapped run tasks positional

Recognize supported environment prefixes and PowerShell call operators without
mistaking prompt arguments for executables. Keep environment and run separators
separate, preserve the task text and exclude run commands from native submission.

Source-parent: 23fc08b4e9
Related-to: stablya/orca#17551
Credits: @coelho-doti (stablya/orca#13065)

* Prepare complete private OpenCode launch validation source

Integrate the complete reviewed readiness, capability, native prompt, overlay and positional-run source onto frozen main. Preserve canonical atomic ACL retry, status generator/disposal, restrictive-umask fixtures and unowned source. Keep supported wrapped run commands positional.

Private-validation-source: a44345ce49
Original-full-source: 23fc08b4e9
Original-core-base: 8186ded0bd
Frozen-main: 08ee7ba9ef
Owned-source-paths: 111
Publication-policy: private validation only; preserve the six separate PR boundaries and held model/provider drafts

Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
Co-authored-by: Adnan Khan <adnank11427@gmail.com>
Credits: juli-gonzalez readiness contribution; Ahmed Nagy atomic plugin writer; coelho-doti positional run contribution

* Prepare private complete 111-path launch validation on current main

Private validation only. Preserve main credential additions and original launch ownership. Held model and provider topics remain excluded.

* Recognize env options before positional OpenCode run messages

* STRICT launch CI contract correction

* CAPS launch CI contract correction

* INTENT launch CI contract correction

* test(opencode): wait for malformed claim retries before expiring intent

Observe real endpoint I/O completion under fake timers before forcing expiry.

* test: initialize Claude prompt state in output retention fixture

* Verify OpenCode catalog model launches and preserve current launch behavior

* Verify OpenCode catalog model launches and preserve current launch behavior

* Wait for OpenCode location hydration in intent startup

* Refuse unverified new-worktree OpenCode model launches and record startup attribution

* fix(opencode): bind startup readiness to the composer location

* Bind OpenCode startup readiness to the current location in intent startup

* Restore the owning Orca CLI path after shell profiles

* Use a literal marker for the Bash lookup regression

* Preserve plain panes and initialize zsh after prompt hook replacement

* Preserve user line-editor dispatchers during deferred startup

* fix: retain CLI startup when global Zsh replaces prompt hooks

* test: replay global Zsh hook replacement after host startup

* test: isolate controlled Zsh widgets from distro keyboard setup

* fix(shell): preserve user hooks during deferred zsh initialization

* Retry interrupted OpenCode startup prompt claims

* Keep completed Zsh startup hooks retired when the wrapper is sourced again

* Reject truncated OpenCode catalogs and explain worktree model limits

* Use a template literal in truncated-catalog coverage

* Refuse unfinished OpenCode model catalogs

---------

Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
Co-authored-by: user141514 <user141514@users.noreply.github.com>
Co-authored-by: Ahmed Nagy <ahmednagy25t@gmail.com>
Co-authored-by: Adnan Khan <adnank11427@gmail.com>
Co-authored-by: Orca startup hydration review <agents@stably.ai>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Orca maintenance <orca-maintenance@users.noreply.github.com>
Co-authored-by: Orca <dev@stably.ai>
Co-authored-by: Orca campaign <orca-campaign@local.invalid>
Co-authored-by: Orca OpenCode issue campaign <opencode-campaign@stably.ai>
Co-authored-by: OpenCode issue campaign <codex@localhost>
2026-10-04 16:44:38 -07:00
Jinwoo Hong 3047353017 fix(orchestration): worker-abandon settles a stuck worker and records who did it (#23983)
* fix(orchestration): worker-abandon settles a stuck worker and records who abandoned it

worker-abandon refused or no-oped in the states it exists to escape: a stop
stranded by a dead runtime, an active attempt that was no longer the Task's
latest, and settled workers whose terminal release was stuck at requested or
unknown. It now settles every non-terminal worker except this runtime's own
in-flight stop, records who abandoned it, and retains (never closes) an owned
terminal whose release is not already in flight.

Part of STA-8833.

* refactor(orchestration): abandon retains through worker-retain's rule; record cancellations

- One retain helper serves worker-retain and worker-abandon. A committed release (releasing, unknown) keeps its state and archive, since the tab may already be closed; a retained terminal drops its stale archive.
- worker-abandon keeps the published stale field (this attempt was not the Task's current one) on the worker path.
- task-list shows a failed Task's reason, whitespace-collapsed; task-update help and the recovery guide document cancel = failed + --result cancelled.

* test(orchestration): drop duplicate abandon asserts; leave task-list output unchanged

Cancellation stays documented in task-update help and the recovery guide.

* fix(orchestration): abandoning an already-settled worker changes nothing

Only the settling path retains an owned terminal.

* fix(orchestration): attribute abandon only to a verified caller and keep the prior diagnostic

* fix(orchestration): report an already-settled abandon as stale, as main did
2026-09-30 13:41:34 -04:00
Jinwoo Hong 9afd1101ff fix(orchestration): stop minting and printing the dispatch capability (#23994)
* fix(orchestration): authorize worker reports without the dispatch capability

Worker lifecycle reports and questions no longer depend on the per-dispatch
capability token that lives only in the agent's conversation. The host now:

- ignores capability_hash/capability_revoked_at for authorization on every
  row and checks the exact worker process instead (ask gains that check);
- refuses a report whose calling terminal is provably another orchestration
  party (a Run coordinator or another Dispatch's worker), treating env that
  names no live pane here as absent;
- applies one worker-state rule locally and remotely: a stop in flight
  refuses, while stop_unknown and start_unknown accept and settle.

Minting and printing the flag are unchanged, so an older host and older
preambles keep working.

* fix(orchestration): stop minting the dispatch capability

Dispatches no longer mint a per-Dispatch token, and preambles, the bundled
skill guide and the ask resume hint stop printing --dispatch-capability. The
consumer-generation bump and delivery fence that minting carried stay, now
as setDispatchConsumer. Readers that inferred meaning from capability_hash
read what they meant instead: worker-show's injected stage comes from the
attached consumer, and a failed start copies custody identity only when no
authority was ever attached. The CLI keeps accepting and forwarding the
flag for older hosts.

Cancelling a Task is recorded as failed with a reason; task-list now shows
that reason and the guide and task-update notes document the recipe.

* fix(orchestration): name the fenced party without implying which Dispatch it owns

* refactor(orchestration): one worker report rule, fence only a different party

- One module owns the unproven/settleable worker states and the refusal rule; local send records it, ask and remote throw it. A stale process is worker_identity_changed on every path.
- The caller fence passes the worker's own terminal when its --from handle went stale.
- Document the shared-tmux-server limit; drop the dead dispatch_capability_invalid rejection member; tests assert dispatch state, not the capability column.

* refactor(orchestration): drop setDispatchConsumer and the dead capability retention

- dispatch --inject no longer re-points the row createDispatchContext just wrote; worker-show reports every worker-less Dispatch as context_only, since Orca keeps no record of the paste. Tests re-point through a fixture.
- failWorkerStart always records when the lifecycle closed; nothing authorizes on it.
- Restore the ask resume hint's echo of a passed --dispatch-capability: an old host checks it before --resume.
- Move the cancellation convention to #23983.

* test(orchestration): drop capability-era assertions other tests already cover

* refactor(orchestration): drop the host-side capability field and no-op test fixtures

- RpcRequest and the SSH bridge stop carrying orchestrationCapability; the CLI's wire field stays for older hosts.
- Fixtures pass identity to createRootDispatch instead of re-pointing to the same values; drop absence checks for a flag that can no longer be produced.

* test(orchestration): cover a current process whose terminal moved to another pane

* chore(orchestration): finish the capability cleanup in test stubs and skill wording

* test(orchestration): drop needless response casts; mark the db stub cast safe
2026-09-30 13:26:06 -04:00
Jinwoo Hong e03870403e fix(orchestration): accept worker reports without the dispatch capability (#23982)
* fix(orchestration): authorize worker reports without the dispatch capability

Worker lifecycle reports and questions no longer depend on the per-dispatch
capability token that lives only in the agent's conversation. The host now:

- ignores capability_hash/capability_revoked_at for authorization on every
  row and checks the exact worker process instead (ask gains that check);
- refuses a report whose calling terminal is provably another orchestration
  party (a Run coordinator or another Dispatch's worker), treating env that
  names no live pane here as absent;
- applies one worker-state rule locally and remotely: a stop in flight
  refuses, while stop_unknown and start_unknown accept and settle.

Minting and printing the flag are unchanged, so an older host and older
preambles keep working.

* fix(orchestration): name the fenced party without implying which Dispatch it owns

* refactor(orchestration): one worker report rule, fence only a different party

- One module owns the unproven/settleable worker states and the refusal rule; local send records it, ask and remote throw it. A stale process is worker_identity_changed on every path.
- The caller fence passes the worker's own terminal when its --from handle went stale.
- Document the shared-tmux-server limit; drop the dead dispatch_capability_invalid rejection member; tests assert dispatch state, not the capability column.

* test(orchestration): drop capability-era assertions other tests already cover

* test(orchestration): cover a current process whose terminal moved to another pane
2026-09-30 13:01:53 -04:00
Neil 52a1e2875b feat(orchestration): accept Muse model and effort for supervised workers (#22383)
* feat(orchestration): accept Muse model and effort for supervised workers

`worker-start --agent muse` already launched, but `--model` was refused because
Muse had no session-option catalog. Add one that maps worker preferences to
`muse --model <id>` and `--reasoning-effort <level>`; it seeds no models, so
native-chat surfaces show no picker.

opencode stays without `--model`: the opencode 2 TUI (now shipped as
`opencode`) rejects the flag, so the refusal now tells callers to rely on the
agent's own config. Help, skill guide, and docs list valid `--agent` ids and
the agents that accept `--model`.

Refs #19823

* test(mobile): repin session route closure for the Muse option catalog
2026-09-22 22:20:35 -07:00
eb92222e7f feat: support Antigravity as supervised worker (#21705)
* feat: add supervised Antigravity worker support

* fix: address Antigravity worker review findings

* fix: stabilize Antigravity readiness detection

* fix: allow Antigravity resume footer after readiness

* fix(antigravity): make agy reach worker_done as a supervised worker

Three defects each blocked `orchestration worker-start --agent antigravity
--worktree new-child` at the agent_readiness stage.

1. Readiness never fired. The composer check required the trimmed line to be
   exactly one character, but agy 1.2.7 launches in accept-edits mode and paints
   it into the caret row (`> Accept-edits mode: ...`). Widened narrowly to a bare
   `>` or `> <name> mode:`; matching any `> <text>` would make every menu dialog
   read as ready, since they all prefix their highlighted row the same way.

2. No trust artifact for agy. Added markAntigravityWorkspaceTrusted, writing
   ~/.gemini/antigravity-cli/settings.json under `trustedWorkspaces` — verified
   empirically against agy 1.2.7, and distinct from the Gemini CLI's
   trustedFolders.json, which agy does not consult. Trust is exact-path and not
   inherited by subdirectories, so each child worktree needs its own entry.

3. The orchestration path skipped the preset. Orca has two trust dispatch
   chains: the renderer's preflightAgentTrust and the main-process
   markLocalWorktreeTrusted. worker-start only takes the second, which matched
   cursor/copilot/codex and fell through for antigravity, so the trust write
   never happened while renderer-side tests passed.

Verified live end to end: the dispatch settles `succeeded` with worker_done
carrying the right task and dispatch ids, and the worktree is appended to agy's
settings with sibling keys untouched.

Known gap: remote-agent-trust-presets.ts has no antigravity branch. The SSH
artifact path is unverified, so agy over SSH still stalls at agent_readiness.
Recorded in a comment there rather than guessed at.

* fix(antigravity): wire trust preset through preload safely

* fix: preserve Antigravity readiness across transcript tails

---------

Co-authored-by: Neil <neil@stably.ai>
Co-authored-by: LielinaH <lielinah@gmail.com>
2026-09-21 20:22:08 -07:00
Jinwoo Hong 3336933cc8 fix(orchestration): list worker Dispatches newest first and warn when the page truncates (#21523)
* fix(orchestration): list worker Dispatches newest first and warn when the page truncates

`worker-list` paged `ORDER BY d.rowid ASC` with a 100-row cap, so a Run with
more than 100 Dispatches answered with its OLDEST 100. The workers a coordinator
had just started, and the rows carrying `projection.attention.requiresAction`,
were on a page nobody fetched, while `counts` and `page.total` covered the whole
Run so the receipt read as complete.

One ordering, flipped: the detail query and the terminal-state scan it pages by
both order `d.rowid DESC`, and the cursor fence walks down (`d.rowid < anchor`).
The snapshot fence is unchanged — `d.rowid <= snapshot` still means "nothing
created after the first call".

When the page truncates the receipt now carries a `warnings` string, the same
shape `worker-output` already uses, alongside `page.hasMore`. Text output keeps
its `More: --cursor` line and prints the warning through the block it already
had for partial-host errors.

Refs STA-7861

* fix(orchestration): make the worker-list truncation warning true on every page

The warning said "Showing the N newest of T Dispatches" unconditionally, but
`hasMore` is true on every page except the last, so page 2 of a 300-Dispatch Run
claimed to be the newest 100 while showing rows 200..101. This PR exists because
a receipt read as complete when it was not; that warning shipped a receipt that
read as the newest page when it was not.

The page count and the ordering are separate facts, so state them separately:
"Showing N of T Dispatches, newest first; more are on later pages." True on page
one and page N alike, no extra state. The 105-row case only ever reached the
last page, where `hasMore` is false, which is why it missed this; a new case
walks 6 Dispatches at `--limit 2` so a page that is truncated AND not page one
is covered.

Also: the `worker-list` --help note and the recovery-and-cleanup reference still
described the oldest-first contract; both now say newest first. The snapshot
test is renamed to the property it actually proves — under DESC a later insert
is unreachable by arithmetic, so what the `d.rowid <= snapshot` fence still
earns is pinned `page.total` and `counts`, not row exclusion. The continuation
comment says "below the anchor" next to `d.rowid < ?`, and the two SAFETY
rationales now say what they are: an unchanged cast the gate flagged because the
diff moved inside its span.

Refs STA-7861
2026-09-18 17:38:22 -04:00
Jinwoo Hong 4b1b7178ad fix(orchestration): scope @ group addresses to the sender's Run (#19783)
* fix(orchestration): scope @ group addresses to the sender's Run

`@all`, `@idle`, and the agent-name groups (`@claude`, `@codex`, ...) resolved
against every terminal on the host. A coordinator meaning "my three reviewers"
reached 126 agents across every open project, twice in one day, and every
unrelated agent burned a turn discarding mail that was never for it.

Every group except `@worktree:<id>` now means the live Dispatches of the
sender's own Run, each addressed as `dispatch:<id>` so delivery is durable
even when the worker terminal is not attached yet. A sender bound to no Run is
refused with `invalid_argument` naming `run:<id>` / `dispatch:<id>`; there is
no host-wide fallback and the host's terminals are never enumerated for it.
`@idle` and the agent-name groups filter within that set by the same terminal
status and host-resolved identity as before. `ask --to @group` returns the
same code and points at the owning Run mailbox.

Federated Dispatches read relayed control mail rather than a local mailbox,
so a Run-scoped fan-out skips them with a `recipient_unreachable` warning
naming the direct `dispatch:<id>` address.

Group addresses are resolved host-side, so no RPC or stream shape changes; an
older CLI sending `@all` to a new host gets the Run-scoped meaning.

Claude-Session: run-scoped-group-addresses

* fix(orchestration): revalidate legacy takeover before the recipient verdict

A legacy coordinator taken over while `listTerminals` was in flight reported
`runtime_error` instead of `legacy_read_only`: Run scoping made "no live
Dispatch in this Run" the first thing the group send could fail on, and that
threw before the takeover check ran. Takeover is a precondition, not a
commit-time detail — the sender must be told it is read-only whatever else is
wrong with its recipient set.

Revalidation moves to immediately after the only `await` in the path.
Everything below it is synchronous, so the commit-time window it used to guard
is unchanged; only the error paths now see it.

The legacy partition test gave `term_current_worker` no Dispatch, so under Run
scoping it is correctly not a recipient. It now holds a real current-contract
Dispatch in the same adopted Run, which is what the test is named for: one
`legacy_direct` and one `current_delivery` recipient in one fan-out.

Claude-Session: run-scoped-group-addresses

* fix(orchestration): address the Run a nested coordinator created, not its parent

A nested coordinator is both a worker of its parent Run and the coordinator of
the Run it created. `resolveMessageRun` answers with the parent, correctly,
because that is where its own `worker_done` belongs — but audience is a
different question. Scoping `@all` to that Run sent a nested coordinator's
"shared context" to the siblings it was started beside instead of the workers
it started, and reported success, so it never learned its sub-workers heard
nothing. Before Run scoping the host-wide fan-out reached the sub-workers by
accident; this turned an over-broad delivery into a wrong-audience one, the
exact failure class the change exists to remove.

Group audience now resolves off the Run the sender coordinates, falling back
to its Dispatch's Run. A leaf worker coordinates nothing and is unaffected.
This is a separate question from `routing.run`, not a second answer to the
same one, so `resolveMessageRun` keeps its meaning for point-to-point mail.

Also: when every live Dispatch in a Run is federated, the fan-out skipped them
all and threw a bare `Error` that discarded the warnings naming those remote
workers and how to address each one. The sender was told "no recipients" while
three remote workers existed. That throw now carries a code and the skip
explanations.

Claude-Session: run-scoped-group-addresses

* docs(orchestration): say that no group address reaches a coordinator

A coordinator is not a Dispatch, so Run-scoped groups never include one. That
follows from the rule, but nothing said it, and the old host-wide meaning did
include the coordinator — a worker sending `@all` to raise a blocker would be
heard by its siblings and by nobody who can act. The guide, the CLI note, and
the docs page now say to use `run:<id>` for that, and that a worker which
created its own Run addresses that Run's workers.

Also restores the `@cursor` case dropped when the group tests moved: a Claude
pane titled "Fix the text cursor blink" must not receive Cursor's mail. That
hazard was recorded from real titles and `@droid` alone did not cover it.

Claude-Session: run-scoped-group-addresses

* fix(orchestration): preserve group audience and mailbox identity

* fix(orchestration): validate group scope before dispatch routing

* fix(orchestration): preserve pane identity and exclude coordinator dispatches
2026-09-10 15:06:03 -04:00
Jinwoo Hong 2283f8ba4e docs(orchestration): never pick a worker model the user did not name (#19109)
The sonnet examples were added for a test cohort. Orchestration must not
choose a model on the user's behalf: pass --model only when the user named
one, otherwise inherit the configured agent default.
2026-09-06 14:51:37 -04:00
Jinwoo Hong 06a607a1d7 feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​12401 |

<!-- /orca-pr-loc -->

## ELI5

Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.

## What changed

- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.

## Why

User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.

## Linked issues

Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.

## Review record

This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.

**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:

- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.

Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).

A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.

A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.

Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).

## Testing

- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).

## Notes

- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
2026-09-06 14:34:03 -04:00