Commit Graph
23 Commits
Author SHA1 Message Date
Jinwoo Hong 3336933cc8 fix(orchestration): list worker Dispatches newest first and warn when the page truncates (#21523)
* fix(orchestration): list worker Dispatches newest first and warn when the page truncates

`worker-list` paged `ORDER BY d.rowid ASC` with a 100-row cap, so a Run with
more than 100 Dispatches answered with its OLDEST 100. The workers a coordinator
had just started, and the rows carrying `projection.attention.requiresAction`,
were on a page nobody fetched, while `counts` and `page.total` covered the whole
Run so the receipt read as complete.

One ordering, flipped: the detail query and the terminal-state scan it pages by
both order `d.rowid DESC`, and the cursor fence walks down (`d.rowid < anchor`).
The snapshot fence is unchanged — `d.rowid <= snapshot` still means "nothing
created after the first call".

When the page truncates the receipt now carries a `warnings` string, the same
shape `worker-output` already uses, alongside `page.hasMore`. Text output keeps
its `More: --cursor` line and prints the warning through the block it already
had for partial-host errors.

Refs STA-7861

* fix(orchestration): make the worker-list truncation warning true on every page

The warning said "Showing the N newest of T Dispatches" unconditionally, but
`hasMore` is true on every page except the last, so page 2 of a 300-Dispatch Run
claimed to be the newest 100 while showing rows 200..101. This PR exists because
a receipt read as complete when it was not; that warning shipped a receipt that
read as the newest page when it was not.

The page count and the ordering are separate facts, so state them separately:
"Showing N of T Dispatches, newest first; more are on later pages." True on page
one and page N alike, no extra state. The 105-row case only ever reached the
last page, where `hasMore` is false, which is why it missed this; a new case
walks 6 Dispatches at `--limit 2` so a page that is truncated AND not page one
is covered.

Also: the `worker-list` --help note and the recovery-and-cleanup reference still
described the oldest-first contract; both now say newest first. The snapshot
test is renamed to the property it actually proves — under DESC a later insert
is unreachable by arithmetic, so what the `d.rowid <= snapshot` fence still
earns is pinned `page.total` and `counts`, not row exclusion. The continuation
comment says "below the anchor" next to `d.rowid < ?`, and the two SAFETY
rationales now say what they are: an unchanged cast the gate flagged because the
diff moved inside its span.

Refs STA-7861
2026-09-18 17:38:22 -04:00
Jinwoo Hong 12d744f253 fix(skills): keep computer-use off filesystem and shell tasks (#21069)
* fix(skills): keep computer-use off filesystem and shell tasks

STA-7615: "On my desktop create a folder" was matching computer-use because
discovery copy said OS/window-level and neighboring skills advertised desktop UI.
Scope the trigger to visible GUI with no CLI path, and exclude files/folders/git/shell.

* fix(skills): prefer programmatic paths over computer-use

State the last-resort rule in discovery copy instead of enumerating
files/folders/git/shell. computer-use prefers shell, filesystem, git, HTTP,
CLIs, and Playwright/CDP; neighboring skills route to Computer Use only when
a visible window needs GUI control those cannot do.

* fix(skills): stop advertising computer-use from orchestration

Orchestration coordinates workers; it does not drive a GUI. Drop Computer Use
and Playwright/embedded-browser routing from its discovery description so
those tools are not pulled in from a coordination skill.

* fix(skills): drop Playwright from orca-cli discovery

orca-cli should not prescribe Playwright or CDP. Those tools may not be
installed, and page automation is not this skill's job.

* fix(skills): drop the page-only ban from computer-use discovery

Page automation is a preference, not a prohibition. If Playwright or CDP is
not available, a visible browser window is valid Computer Use. Keep the
hard split for Orca's embedded browser (`orca-cli`) only.
2026-09-16 15:43:11 -04:00
Jinwoo Hong 98fdbc4ade fix(orchestration): file federated worker mail under the coordinator Run (#19542) 2026-09-08 04:03:07 -04:00
Jinwoo Hong 2283f8ba4e docs(orchestration): never pick a worker model the user did not name (#19109)
The sonnet examples were added for a test cohort. Orchestration must not
choose a model on the user's behalf: pass --model only when the user named
one, otherwise inherit the configured agent default.
2026-09-06 14:51:37 -04:00
Jinwoo Hong 06a607a1d7 feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​12401 |

<!-- /orca-pr-loc -->

## ELI5

Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.

## What changed

- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.

## Why

User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.

## Linked issues

Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.

## Review record

This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.

**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:

- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.

Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).

A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.

A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.

Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).

## Testing

- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).

## Notes

- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
2026-09-06 14:34:03 -04:00
Jinwoo Hong 54a8afc91d fix(orchestration): typed error codes for dispatch and worker-start refusals (#18902)
* fix(orchestration): typed error codes for dispatch and worker-start refusals

orchestration dispatch (and worker-start, which composes it) surfaced task
not found, task not ready, and inject rejected as the same bare
runtime_error, so an agent reading the receipt could not choose between
creating the task, waiting on dependencies, or picking another terminal.

Add task_not_found (data.taskId), task_not_ready (data.status,
data.unmetDependencies), and inject_rejected (data.terminal, data.reason),
each carrying data.nextSteps so every shipped CLI already prints the
recovery. worker-start's not-ready refusal moves from task_not_startable
to task_not_ready with the same detail. runtime_error stays for genuinely
unexpected failures.

Proven red-first from RpcDispatcher through the CLI's own failure
formatting, plus an SSH bridge test that the host CLI's typed refusal
relays unchanged.

* test(orchestration): load CLI formatter at runtime in the dispatch-code test

The composite node typecheck (config/tsconfig.node.json without
--composite false, as CI runs it) rejects a static import of src/cli from
a main test with TS6307. Load the formatter and error class dynamically
behind narrow structural types, as the CLI/runtime boundary test does.

* fix(orchestration): keep task_not_startable and split the CLI-format proof

Review on #18902:

- Drop task_not_ready. worker-start already published task_not_startable
  for a not-ready Task, so renaming it would change an existing receipt
  value under old clients. dispatch now emits task_not_startable too (it was
  a bare runtime_error before, so this is purely additive), with the new
  data.status / data.unmetDependencies / data.nextSteps.
- Move the refusal receipts (code, message, data) into
  src/shared/orchestration-dispatch-refusal-contract.ts so the runtime
  emits them and the CLI test formats the identical envelope. The RPC test
  under src/main asserts toEqual against the contract; the new
  src/cli/orchestration-dispatch-refusal-format.test.ts feeds those same
  receipts to formatCliError / reportCliError. Neither tsconfig widens and
  the composite typecheck CI runs is clean.

* fix(orchestration): keep published refusal messages and type the DB claim guards

Codex review of #18902:

- Every call site keeps the exact message it published on main
  ("Task not found: <id>", "only a ready Task can start.", "cannot retry
  from Dispatch"); the shared contract now takes the message per site and
  only owns the code and data. Baseline strings are pinned as literals.
- createDispatchContext's own missing/non-ready guards, including the
  atomic-claim loser, now emit the same typed receipt instead of a bare
  Error, so a dispatch that races a status change no longer flattens to
  runtime_error. Covered by a dispatcher-level race test.
- Invalid --retry-of keeps task_not_startable but now carries status,
  unmetDependencies, retryOf, and a retry-specific next step.
- Dependency recovery text distinguishes waiting on running deps from
  retrying/unblocking failed ones.
- CLI test adds an unknown-code case so the old-client claim rests on an
  assertion, not a comment; SSH test asserts exact stdout.
- Guide table narrowed to the covered preflight cases; occupancy stays
  runtime_error and is named as such.
2026-09-05 20:27:29 -04:00
Jinwoo Hong 3e4fd4a7af Shorten orchestration skill description under the Agent Skills 1024-char limit (#18683)
* Shorten orchestration skill description under the Agent Skills 1024-char limit

The folded description was 1038 chars, so spec-conforming installers such
as SkillStar rejected the bundled orchestration skill. Drop the two clauses
already covered elsewhere in the same description: "decomposing work across
agents" (implied by "structured multi-agent coordination") and "automation
of the browser embedded inside Orca" (restated by the locked `orca-cli`
embedded-pages sentence). Every routing trigger asserted by
orchestration-skill-guidance.test.mjs, the orca-cli handoff boundary, and
the Computer Use boundary are unchanged. Result: 958 chars.

Add config/scripts/skill-description-length.test.mjs, which parses every
skills/*/SKILL.md frontmatter with `yaml` and fails on an empty or >1024
char description, so the regression cannot return. orca-cli sits at 1015
and is left as is.

Fixes #17935

* Keep the embedded browser in the orchestration description's orca-cli routing

Restores the word "browser" in the orca-cli sentence ("and the Orca embedded
browser") so agents scanning for it still route embedded-browser control to
orca-cli. Description is 985 chars, 39 under the spec limit.
2026-09-04 14:27:07 -04:00
Jinwoo Hong b44ef1e59d fix(skills): narrow computer-use discovery boundary (#17736)
* fix(skills): narrow computer-use discovery boundary

* chore: remove merge-formatting noise

* fix(skills): name browser page automation surfaces
2026-08-31 18:57:52 -04:00
Brennan Benson cc6b600e21 Fix orchestration CLI recovery, settled-Dispatch mail, and guide defects (#16919)
* Fix orchestration CLI recovery, settled-Dispatch mail, and guide defects

Five reported orchestration CLI defects, verified individually before fixing.
Two were real code defects, one was a docs error, one was correct as-is, and
one was correct on both ends except for its recovery wording.

- Mail addressed to a settled `dispatch:<id>` was accepted and silently dropped.
  Local sends bypassed the settlement check the federated branch already had, so
  the caller was told success for a delivery no worker would ever read. Reject
  with `dispatch_inactive` and name the Run mailbox to use instead.

- A lost mutation response offered no read-only way to ask whether it took
  effect. `--retry-request` does dedupe correctly, but the recovery guidance
  emitted a query command only when the payload carried a dispatch id, which is
  exactly what a lost response lacks. Add read-only
  `orca orchestration request-show --request <id>` over the durable receipt
  ledger, and always emit a read-only step before the keyed retry.

- The bundled `orca-cli` guide documented `check --unread --inject`, a flag the
  parser rejects. Correct it to `--format` and add a ratchet that runs every
  orchestration invocation in the bundled guides through the real CLI parser.

- `check --json` is one stdout document and its keepalives are stderr-only; the
  reported `Extra data: line 2` came from merging the streams. Document the
  contract rather than changing the wire.

- A rejected lifecycle message is loud on both ends already, but the rejection
  never named the flag that supplies the missing capability. Name it.

* Harden orchestration mutation recovery guidance
2026-08-30 18:12:58 -07:00
Brennan Benson 9135b6f004 feat(orchestration): surface nested worker depth and propagate it across hosts (#16669)
* feat(orchestration): surface nested worker depth and propagate it across hosts

Builds on the depth enforcement in the previous commit, which shipped with the
setting reachable only by editing settings.json and with workers never told they
could nest.

Adds the Settings -> Agents control (a 1/2/3 select rather than a free-form
number, which bounds the value without inventing a numeric input primitive). The
key stays absent from the SettingsUpdate RPC schema, matching agentSkillSharingEnabled:
settings.update is reachable from the CLI, so an RPC-writable depth would let a
worker raise its own cap.

Adds a SUB-DISPATCH block to the dispatch preamble, emitted only when the worker
actually has budget left. A worker told it "usually cannot" delegate still tries and
then reports the refusal as a blocker, so the section is omitted entirely rather
than softened.

Propagates depth to federated worker hosts. Previously the home side computed and
stored a depth the remote host never received, so a remote attachment always read
as depth 1. That is correct at the default cap and wrong as soon as the cap is
raised — precisely when someone starts relying on nesting. The field is optional,
so an older Run home simply omits it and the attachment's NOT NULL DEFAULT 1 keeps
the fail-closed behaviour. Enforcement still runs on the executing host against
that host's own cap, consistent with the SSH execution boundary.

* fix(orchestration): close nested depth readiness gaps

* fix(settings): defer nested depth translations

* fix(orchestration): drop federated depth keys that main already landed

The enforcement PR's review pass added the same federated depth propagation
before it merged, so replaying this branch onto main produced duplicate object
keys. Keep main's versions -- its schema entry validates an integer >= 1 rather
than any finite number.

* fix(settings): label nested worker depth select

* fix(settings): move nested depth to orchestration

* fix(settings): refine nested depth placement
2026-08-26 16:16:05 -07:00
Brennan Benson 8a07bbd8cf fix(orchestration): enforce nested worker depth instead of an accidental fence (#16668)
* fix(orchestration): enforce nested worker depth instead of an accidental fence

Orca documented that "dispatched workers cannot spawn their own sub-workers
(worker-start is coordinator-fenced)". No such check existed. What existed was a
single Run-binding check in the workerStart RPC: a worker's terminal is not bound
to a Run, so worker-start happened to fail. The rule was emergent, asserted by no
test, and written in no doc — and it leaked. A worker could run-create its own
Run, task-create, and worker-start: now bound, the check passed.

Replace it with a real, configurable depth cap.

Depth is derived from the caller's own active Dispatch rather than from Run
binding, which is what dissolves the run-create bypass: creating a Run does not
stop you being a worker. Enforcement lives in a single dispatch-row writer that
owns all three INSERTs that mint a live worker — the generic claim, the supervised
worker-start path (including every retry), and the remote attachment. Two of those
were missed by earlier drafts of this change, so `creator` and `maxDepth` are
required parameters: a new spawn path cannot compile without deciding, and a
boundary test refuses the SQL anywhere else.

Schema v30 adds depth to dispatch_contexts and remote_dispatch_attachments,
NOT NULL DEFAULT 1 and backfilled to 1 so an unstamped or pre-upgrade row fails
closed rather than reading as a root coordinator. The attachment pane indexes
widen to the five states in which a remote worker may still be running:
loss of contact is not evidence of process death, so an unverifiable worker still
counts as a nesting parent.

Also adds the caller-evidence assertion that workerStart was the only Run-scoped
verb to skip, so a declared --from cannot name another terminal's pane and inherit
its depth.

Default is 1, so behaviour is unchanged unless the new setting is raised. Two
limitations are deliberate and documented rather than papered over: this is a
guardrail and not a security boundary, since a caller whose launch evidence is
unverifiable (any ordinary restored terminal) can declare another handle; and it
is enforced at supervised dispatch creation, so a settled worker whose process is
still alive counts as a root again.

* fix(orchestration): share caller resolution and pin worker gaps

* refactor(orchestration): make the caller resolver's pane contract explicit

Overloads so requireStablePane callers get a non-null string instead of casting,
and rename the attestation opt-out to say what it means: the caller asserts it
itself. A flag called assertEvidence:false reads as "attestation optional",
which is the hole this helper exists to close.

* fix(orchestration): propagate dispatch depth to federated workers

* chore(cli): refresh bundled orchestration guide
2026-08-26 13:22:09 -07:00
Jinwoo Hong 0bedeea642 fix(orchestration): expose unsupervised dispatch lanes (#15105) 2026-08-17 13:53:26 -07:00
Brennan Benson cbca291aa7 fix(orchestration): preserve direct user authority after worker_done (#14192)
* fix(orchestration): preserve direct user authority

* test(orchestration): assert settled dispatch boundaries
2026-08-13 12:00:04 -07:00
Jinwoo Hong d349f9a972 Use provider-neutral Opus alias in orchestration guide (#14119) 2026-08-12 16:03:29 -07:00
Jinwoo Hong b0ba51831c Add per-worker model and effort overrides (#12851) 2026-08-05 21:17:45 -07:00
Brennan Benson 39c3c58d55 perf(runtime): gate terminal.list visual layouts (#12450)
* perf(runtime): gate terminal.list visual layouts and stop the false writable claim

visualLayouts is ~31% of a large terminal.list payload (44,208 B of 137,412 B on a live 134-terminal remote runtime) and has exactly one consumer: the human-readable CLI formatter. Gate it behind an includeVisualLayouts request param that defaults to included, so pre-flag clients are unaffected, and have every --json/internal caller opt out.

Also drop the record-backed builder's writable, which was a verbatim copy of connected. terminal.show now states writability explicitly as exactly what terminal.send's PTY gate enforces.

* test(runtime): type the payload-size fixture arrays for tsc

* fix(runtime): preserve terminal list compatibility

* test(runtime): guard terminal list optimization

* fix(cli): preserve agent access to terminal layouts
2026-08-04 17:50:52 -07:00
Brennan BensonandOrcaWin f4b2b782b5 feat(orchestration): coordinator-driven release of settled worker terminals (STA-905) (#12355)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-08-03 17:17:26 -07:00
OrcaWin 9a2676023c fix(orchestration): prefer current authority over legacy fallback (#11737) 2026-07-31 01:56:13 -07:00
OrcaWinandOrcaWin 363e478909 fix(orchestration): preserve active workers across updates (#11271)
* fix(orchestration): preserve active workers across updates

* test(ssh): model absent legacy adoption

* test(orchestration): align compatibility contracts

* fix(windows): escape updater PowerShell booleans

* fix(windows): restore stock uninstall process check

* fix(orchestration): keep recovery off renderer startup barrier

* fix(orchestration): harden legacy recovery migration

* fix(orchestration): close recovery review gaps

* fix(orchestration): complete legacy worker cutover recovery

* fix(orchestration): preserve legacy workers across updates

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-29 11:31:35 -07:00
OrcaWinandOrcaWin 77d4c64f7a Improve orchestration migration safety for live legacy workers (#11107)
* fix(orchestration): clarify legacy migration safety

* fix(cli): sanitize legacy formatted messages

* test(runtime): allow near-cap fuzz under shard load

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-28 00:36:25 -07:00
OrcaWin cd05f2ff93 Implement robust orchestration primitives and connected-server workers (#9925) 2026-07-27 12:31:37 -07:00
Jinjing a6df23c762 Tighten orchestration worktree isolation policy (#9482)
* docs(orchestration): require true isolation for worker worktrees

* docs(orchestration): deduplicate parallel worker guidance

* docs(orchestration): clarify isolation exceptions

* docs(orchestration): clarify checkout preference

* Tighten orchestration worktree isolation policy guidance

- Clarify that same-worktree workers remain orchestration children
  despite appearing as peers in the sidebar
- Define new-worktree creation as required only for explicit requests
  or concrete filesystem/checkout conflicts, not convenience
- Distinguish stacked worktrees from independent ones via --no-parent
- Rename agent-first guidance to reflect conditional requirement

* Release orchestration skill v25 with tightened worktree policy
2026-07-19 17:17:40 -07:00
Brennan Benson 31f643ca42 Add version-matched skill guides to the CLI (#8624)
* Add version-matched bundled skill guides

* Clarify skill freshness rollout PRs

* Add canonical skills show alias

* fix(skills): address guide review feedback

* fix(skills): make guide commands cross-platform

* fix(skills): apply the ORCA convention to the emulator guides

Review follow-up: the emulator guides still instructed literal
`orca emulator ...` in sh fences with no Linux disambiguation, so on
unmanaged Linux they could launch the GNOME screen reader — the exact
failure the executable-selection preamble prevents. Both emulator
guides now carry the preamble and ORCA placeholder across fences,
tables, and prose, and the cross-platform safety test covers all four
converted guides. Also replaces computer-use's "unless a block names a
shell" carve-out, which contradicted its own POSIX example, with the
unconditional placeholder rule.
2026-07-14 02:17:55 -07:00