Commit Graph
19 Commits
Author SHA1 Message Date
Jinwoo Hong 06a607a1d7 feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​12401 |

<!-- /orca-pr-loc -->

## ELI5

Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.

## What changed

- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.

## Why

User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.

## Linked issues

Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.

## Review record

This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.

**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:

- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.

Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).

A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.

A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.

Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).

## Testing

- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).

## Notes

- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
2026-09-06 14:34:03 -04:00
Brennan BensonandMerge Sim aabcc57366 fix(runtime): publish remote control outages to host surfaces (#17531)
* fix(runtime): publish remote control diagnostics to renderer

* test(runtime): account for diagnostics bridge listener

* fix(i18n): add runtime connection state labels

* test(runtime): clean up shared control connection

* fix(runtime): fence diagnostics by shared-control capability

* fix(runtime): preserve authoritative transport state

* fix(runtime): preserve diagnostic overlay lifecycle

* fix(runtime): avoid publishing unchanged diagnostics state

---------

Co-authored-by: Merge Sim <sim@local>
2026-08-31 12:25:17 -07:00
Brennan Benson 913509edeb fix(orchestration): prevent slow worker-start stalls (#16300)
* Extend orchestration agent submission timing budgets

* fix(orchestration): preserve mutation recovery identity

* fix(orchestration): preserve recovery executable identity

* fix(orchestration): keep worker starts and recovery commands safe

* test(orchestration): cover federated worker preflight

* fix(orchestration): harden mutation recovery

* fix(orchestration): redact dispatch recovery credentials

* chore: preserve upstream skill dialog formatting

* test(orchestration): stabilize agent prompt submit e2e

* fix(orchestration): validate federated start receipts

* perf(runtime): cache unchanged prompt verification tail

* fix(orchestration): reject worker-start timer overflow

* fix(orchestration): normalize worker-start timeout defaults

* fix(orchestration): normalize worker-start readiness budgets

* fix(orchestration): normalize federated readiness timeout

* test(runtime): tolerate current-main degradation exports

* chore: preserve current-main orcad formatting

* chore: drop unrelated formatting carryover
2026-08-27 15:25:30 -07:00
Neil 09048c63d4 feat(orcad): add headless browser providers (#16193)
* feat(orcad): add headless browser providers

* fix(orcad): merge the duplicate runtime-browser type import
2026-08-24 21:11:45 -07:00
Neil a61b39a9a6 fix(runtime): stamp a runtime's own project setups as local, and report remote status about the remote (STA-4792) (#15376)
* fix(runtime): stamp a runtime's own project setups as local, and report remote status about the remote (STA-4792)

Two independent frame-of-reference bugs, both from code describing one machine
while labelled as another.

#15366 — projectHostSetup.* persisted the caller's host id verbatim. Those
`runtime:<environment-id>` ids are minted by the calling client's own pairing
store, so they name a machine only relative to that client. A client sending
one is addressing this runtime, and runtimes do not proxy these calls onward,
so the host it names is us. Storing the client's spelling made one machine look
like a different host to every other client, hid its rows from them, and
defeated the (projectId, hostId) duplicate check — two laptops paired to one
server each created their own setup for the same checkout. Re-spell it as
`local` at the RPC boundary. Rows written earlier keep their old stamp; readers
already project `local` back to `runtime:<their-id>`, so the client-visible
model is unchanged and no ids are rewritten.

STA-4792 defect 4 — `status --environment <name>` hardcoded app.running:false
to mean "no desktop on THIS machine" while every other field in the same object
described the target, including a desktopWindowStatus echoed straight from it.
The result contradicted itself and read as "that run was headless" when the
remote GUI was up. `app` now describes the target, keyed off the one window
status that requires a live renderer, and the result names its own subject so
the frame can't be misread again. The remote pid is not knowable, so it stays
null.

STA-4792 defect 2 gets a regression test rather than a fix: routing already
made the client remote, which is what stops a Windows destination being joined
to the local cwd. The test pins the exact reported invocation.

* fix(status): share the remote app projection with the SSH host passthrough, and name the version gap on project host setup

Two review follow-ups.

The SSH host passthrough answered `app.running: true` unconditionally for the
Orca host a caller reached over SSH, claiming a desktop app even for a headless
`serve`. That is the same defect as the paired-server path, one transport over,
so the projection moved to shared and both now answer the question the same way.

`--host runtime:<id>` routes project commands to a paired server, which means a
client can reach a server that predates project host setup without meaning to.
That answered a raw `method_not_found`, which reads as an Orca bug rather than a
version gap; the CLI now names it the way the desktop already does.

Reverted a third change: making the persistence duplicate check treat `local`
and `runtime:*` as one machine. That assumption holds at the RPC boundary, where
a `runtime:` host means the runtime being addressed, but not in the store, which
also records independent provisioning metadata for machines that are not itself.
An existing test covers exactly that, and it was right. The duplicate
convergence therefore stays bounded to rows written after the normalization.
2026-08-19 17:12:17 -07:00
Neil 4882eeb8ac rm git shim: neutralize stale wrappers without a host gate (#14255)
* Revert "fix terminal attribution shim removal edge cases (#14187)"

This reverts 585dd6d3a9. Re-landed in the next commit without the host capability gate. Nothing shipped with it, so no migration constraint.

* rm git shim: neutralize stale wrappers without a host gate

Re-lands the cleanup half of #14187: pass-through tombstones for retained wrapper paths, env/PATH scrubbing at every spawn owner, and the retired setting drop.

Only writes tombstones when the legacy directory already exists, so a clean install no longer has it created. Leaves out the terminal.attribution-removed.v1 capability gate: the tombstone neutralizes each host locally, so refusing terminal create/split against older hosts denied service without adding cleanup.

* rm git shim: surface neutralization failures and fix rollback marker

Readiness review follow-ups: warn on each failed attempt and on give-up (was silent and undiagnosable); write a VERSION marker distinct from the retired shim's '7' so a rolled-back build rewrites its own wrappers; clear a captured ORCA_REAL_* path that no longer exists so the cmd wrapper's where.exe fallback can run; stop a locked temp file masking the real error. Adds retry-exhaustion coverage.

* rm git shim: pin the cmd fallback order and correct the give-up count

Round-2 review follow-ups: string-pin that a stale ORCA_REAL_* is cleared before the where.exe fallback, and count the initial attempt in the give-up warning so it agrees with the per-attempt line.

* rm git shim: keep the split-failure toast

The revert took a toast that #14187 added alongside the gate but which stands on its own: without it a failed remote split only reaches the console and the pane silently never appears. Also pins attempt ordinals in the retry-exhaustion test.
2026-08-13 03:01:45 -07:00
Neil 585dd6d3a9 fix terminal attribution shim removal edge cases (#14187)
* fix(terminal): fully retire attribution shim

* fix(terminal): harden shim tombstone path lookup
2026-08-12 23:22:48 -07:00
Brennan Benson 8c65dd5094 perf(runtime): keep PowerShell ACL work and a second auth off the remote command path (#12451)
* perf(runtime): keep PowerShell ACL work and a second auth off the remote command path

Two costs sat on the remote authentication path on Windows:

- The E2EE handshake persisted `lastSeenAt` inline, and every secure-file write
  spawns PowerShell synchronously twice to reapply the registry ACL, so the
  client's `e2ee_authenticated` waited on both spawns.
- Every remote CLI command except `status.get` opened a second full WebSocket
  connection just to re-read status for the protocol-compat check, doubling the
  authentications per command.

The first sighting of a device still persists inline (rotation drops entries
disk says were never scanned); later refreshes update memory now and coalesce
onto one deferred write. The compat verdict is saved against the runtime's
per-launch `runtimeId`, so a restarted or upgraded runtime retires it.

* fix(runtime): preserve compatibility on one remote auth

* fix(runtime): flush registry after transport shutdown
2026-08-04 17:04:51 -07:00
OrcaWinandOrcaWin 363e478909 fix(orchestration): preserve active workers across updates (#11271)
* fix(orchestration): preserve active workers across updates

* test(ssh): model absent legacy adoption

* test(orchestration): align compatibility contracts

* fix(windows): escape updater PowerShell booleans

* fix(windows): restore stock uninstall process check

* fix(orchestration): keep recovery off renderer startup barrier

* fix(orchestration): harden legacy recovery migration

* fix(orchestration): close recovery review gaps

* fix(orchestration): complete legacy worker cutover recovery

* fix(orchestration): preserve legacy workers across updates

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-07-29 11:31:35 -07:00
NeilandOrca 6677b5f171 perf(cli): construct the runtime client only when a command needs it (#10919)
src/cli/index.ts was the only eager value-import of RuntimeClient, and five
other eager modules imported just RuntimeClientError / RuntimeRpcFailureError
from the runtime-client barrel -- dragging in client -> pairing -> zod -> ws
-> e2ee on every invocation. Those error classes live in runtime/types.ts,
which has zero children, so the five imports now point there and the client
loads through the existing (already lazy by design) ctx.client getter.

Eager modules 199 -> 46, with node_modules dropping 94 -> 0.
`orca --help` 2.04x (59.6 -> 29.2 ms); the same for help, no-args, and both
error paths, which return before constructing a client. Commands that DO
construct one still gain 1.10-1.12x from not eagerly parsing the transport
the local path never uses.

Correction to an earlier note: websocket-transport alone is ~24 modules /
~8 ms, not the 107 / 28 ms once recorded -- that figure wrongly charged it
for zod, which enters through shared/pairing on a different edge. Marginal
cost, never isolated cost.

Co-authored-by: Orca <help@stably.ai>
2026-07-27 17:16:01 -07:00
OrcaWin cd05f2ff93 Implement robust orchestration primitives and connected-server workers (#9925) 2026-07-27 12:31:37 -07:00
OrcaWin 0326594d52 Update paired Orca servers from the active client (#9839) 2026-07-22 18:52:37 -07:00
Jinjingandbbingz 1d2aaf1bf5 Fix recipe serve desktop promotion (#8646)
* fix(runtime): preserve terminals during headless desktop activation

* rm design doc

* Fix desktop activation launch ordering and blocked-window status resolut

- Check desktopWindowStatus before spawning the Orca app so a blocked
  runtime no longer launches a doomed second instance.
- Reuse resolveDesktopWindowStatus for remote runtime status so it
  honors the same authoritativeWindowId fallback as local status.
- Re-check the authoritative window at spawn time instead of trusting
  a possibly-stale snapshot, since it can be destroyed mid-await.
- Harden the e2e activation spec against silent spawn failures.

---------

Co-authored-by: bbingz <zzb@gxsmjx.com>
2026-07-13 19:41:24 -07:00
Jinjing ffbc4c3cfb fix: tighten CLI contract validation (#3874) 2026-05-30 12:55:47 -07:00
Jinwoo HongandOrca 30a09f3bd9 Add mobile terminal shortcut bar customization (#3012)
Co-authored-by: Orca <help@stably.ai>
2026-05-29 16:39:52 -07:00
Jinjing 78fc047646 Fix terminal wait RPC idle timeouts 2026-05-15 19:11:19 -07:00
Jinwoo HongandOrca a22717bb35 Refactor runtime app architecture (#1878)
Co-authored-by: Orca <help@stably.ai>
2026-05-14 23:13:37 -07:00
Jinwoo HongandOrca ea8ea08116 feat(orchestration): bundled improvements — check-wait, stale-base, preamble+ask, QoL (#1403)
* feat(orchestration): transport keepalive + delivered_at split for check --wait

Implements the four §3 fixes from the check-wait design doc:

- §3.1 Transport keepalive: long-poll RPCs (orchestration.check --wait) emit
  `{"_keepalive":true}` frames every 10s so neither server nor client tears
  the socket down on idle. A `longPoll` admission counter capped at 16 fails
  fast with `runtime_busy` when saturated; an AbortController wired through
  the RPC dispatcher cancels the inner waiter the moment the socket closes.
- §3.2 delivered_at split: push-on-idle now stamps `delivered_at` instead of
  flipping `read`, so the check caller remains the sole consumer of its
  queue. Adds a synchronous idempotent schema migration that hard-fails on
  error.
- §3.3 inbox/check parity: `orchestration inbox --terminal <handle>` and
  `orchestration check --all` agree on the same rows (sequence DESC, no
  mark-read). `check --unread=false` kept for one release as a compat shim.
- §3.4 CLI heartbeat: `orca orchestration check --wait` emits JSON heartbeat
  lines to stderr every 15s so Claude Code's Bash tool sees continuous
  output and doesn't auto-background the subprocess.

Tests: extends runtime-rpc, orca-runtime, envelope-schema, orchestration
method, and formatter suites; adds a subprocess test that spawns the built
CLI and verifies stderr line-flushing, heartbeat cadence, and stdout
cleanliness end-to-end.

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): preamble rules + heartbeat schema

- Preamble (#7, #15, #9): worker_done body ("3-sentence summary" + reportPath),
  BEHAVIOR RULE #1 forbidding AskUserQuestion, heartbeat every 5 minutes with
  taskId+dispatchId payload, AFTER YOU SEND grace window.
- Schema v2 migration: adds 'heartbeat' to messages.type CHECK, adds
  dispatch_contexts.last_heartbeat_at, gated by user_version PRAGMA with
  transactional rebuild + explicit CREATE INDEX to avoid silent perf regress.
- DB helpers: recordHeartbeat (dispatched-only), getStaleDispatches,
  getThreadMessagesFor (thread+handle scoped for ask).

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): coordinator heartbeat + stale detector

Handle incoming 'heartbeat' messages by calling recordHeartbeat keyed on
payload.dispatchId (strict — log-and-skip if missing, no taskId fallback so
a straggler heartbeat from a previously-failed dispatch cannot mask a hung
retry per §5.3.4). On every tick after the 10-minute threshold, emit one
log per stale dispatched row — no auto-fail.

Also threads dispatchId through buildDispatchPreamble so workers can
attribute their heartbeats back to the correct dispatch context.

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): orca orchestration ask verb

Adds a CLI verb that sends a decision_gate message and blocks on the
coordinator's reply, scoped to the outbound message's thread. Group
addresses (@all, @idle, …) are rejected — fan-out questions must use
send --type decision_gate explicitly.

--json emits bare single-line JSON (bypassing printResult) so workers can
pipe `orca orchestration ask … --json | jq -r .answer` without unwrapping
an RPC envelope; human mode prints just the answer. On timeout the verb
exits 1 and returns {answer: null, timedOut: true}.

This is the CLI surface BEHAVIOR RULE #1 in the dispatch preamble points
workers at instead of AskUserQuestion.

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): QoL bundle — preamble visibility, status enum, dispatch cross-ref, inbox --full

Addresses four items from ORCHESTRATOR_FEEDBACK:

- #5 preamble visibility: `dispatch-show --preamble` regenerates the preamble
  text for a task; `dispatch --inject --dry-run` previews without mutating
  state; `dispatch --return-preamble` echoes the injected preamble in the JSON
  response so coordinators can audit what a worker received.
- #6 status enum validation: CLI rejects unknown `task-update --status` values
  with `invalid status '<x>', expected one of: pending, ready, dispatched,
  completed, failed, blocked` before the RPC's generic Zod message. Valid
  statuses are listed under Notes in `task-update --help`.
- #13 task-list dispatch cross-ref: `task-list --json` now includes
  `assignee_handle` and `dispatch_id` for tasks in status=dispatched via a
  read-only LEFT JOIN on dispatch_contexts. Non-dispatched rows keep their
  legacy shape so existing consumers are unaffected.
- #14 inbox body visibility: `inbox --full` prints body + payload verbatim;
  default output is unchanged (id/from/to/subject only).

No DB migrations; join-only change on dispatch_contexts so the sibling
preamble PR's `last_heartbeat_at` column addition will not conflict.

Co-authored-by: Orca <help@stably.ai>

* fix(worktree): prevent stale-base worktree creation and dispatch

Addresses feedback #16 per DESIGN_DOC_STALE_BASE_FIX.md §0. Four v1
components coordinated by a single shared fetch cache on the runtime:

1. Concurrent-fetch-with-gate in UI create path: `createLocalWorktree`
   fires `git fetch` BEFORE the suffix loop / PR probe / path
   resolution, then awaits right before `addWorktree` so the new branch
   always spawns from a fresh remote tip. Renderer sees a two-phase
   spinner via the new `createWorktree:progress` IPC event. The cache
   is a `Map<repoPath::remote, Promise<void>>` + 30s success-only
   timestamp on `OrcaRuntimeService` (§7.1 — shared with dispatch).
2. Dispatch pre-flight drift guard in `Coordinator.dispatchTask`:
   probes `rev-list --left-right --count` against the target worktree
   and silently returns (preserves `ready`, no circuit-breaker burn)
   when `behind > 20` unless the task spec carries
   `allow-stale-base: true`. Parsing strips the flag so it never leaks
   into the worker's `--- TASK ---` block.
3. Preamble drift section: populated only when dispatch detected drift.
   Workers see `--- BASE DRIFT ---` with the N-most-recent subjects
   they don't have, so they can pull them in before running.
4. §3.3 Lifecycle: `.finally()` evicts Map entries on BOTH success and
   rejection; timestamp is written ONLY on success. Prevents a single
   DNS hiccup from wedging every future create on the repo until
   restart, and keeps the freshness window honest.

Defers the DB `allow_stale_base` column (§0.2) and the create-time
warn toast; both can layer in later without migration.

Tests: 35 new/updated unit tests covering drift preamble, dispatch
refusal, spec-text flag parsing, fetch Map eviction after rejection,
freshness-window short-circuit, and concurrent-caller serialization.

Co-authored-by: Orca <help@stably.ai>

* test(orchestration): seed v2 DB in migration hard-fail test

After consolidating the schema bump, fresh DBs are initialized directly at
v3 via createTables(), so the v2→v3 ALTER TABLE is skipped on new installs
and the prior test's stub never fired. Seed a v2-shape file on disk so the
guarded ALTER actually runs and the "simulated migration failure" stub
propagates as intended.

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-05-04 12:10:53 -07:00
Neil fef3f7d2f8 refactor(cli): split runtime-client.ts and add envelope schema validation (#1090)
* refactor(cli): split runtime-client.ts into runtime/ subsystem

Break the 413-line src/cli/runtime-client.ts into focused modules under
src/cli/runtime/:

- types.ts       — RuntimeRpcSuccess/Failure, RuntimeClientError,
                   RuntimeRpcFailureError
- metadata.ts    — readMetadata / tryReadMetadata /
                   getDefaultUserDataPath
- transport.ts   — sendRequest: Unix-socket newline-framed JSON with
                   id and runtimeId verification and timeout handling
- status.ts      — getCliStatus + buildCliStatusResponse +
                   isProcessRunning
- launch.ts      — launchOrcaApp + macOS .app-bundle resolution +
                   ELECTRON_RUN_AS_NODE env handling
- client.ts      — RuntimeClient class, now a thin composer
- index.ts       — subsystem barrel

runtime-client.ts becomes a backward-compat re-export barrel so
src/cli/index.ts and the existing tests import the same symbols from
the same path. No behavior changes.

Motivation: the file had an eslint-disable max-lines override and
mixed five concerns (envelope types, wire transport, metadata I/O,
status aggregation, cross-platform app launch). Splitting them makes
each concern independently testable and unblocks adding schema
validation at the RPC boundary.

* feat(cli): validate runtime RPC envelope with Zod at decode boundary

Add RuntimeRpcEnvelopeSchema and apply it inside sendRequest so every
response frame is validated against the id/ok/result/error/_meta shape
before the CLI hands it to the caller. The payload (`result`) is left
as unknown — the TResult generic remains the caller's responsibility —
so only the envelope itself is the contract this schema enforces.

Motivation: the CLI and the Orca main runtime are separate processes
and can drift in version (older CLI vs newer app, or vice versa during
dev HMR). A malformed or partial frame used to risk mis-typed field
access downstream; it now surfaces as a single structured
`invalid_runtime_response` error.

Behavior:
- Well-formed success and failure frames continue to decode unchanged.
- Failure frames without `_meta` are accepted (the runtime may fail
  before resolving its own runtimeId).
- Valid JSON that does not match the envelope shape now rejects with
  `invalid_runtime_response`, matching the existing error code for
  non-JSON frames.

Tests: adds a pure schema test file
(src/cli/runtime/envelope-schema.test.ts) covering accept/reject cases.
The existing integration tests in runtime-client.test.ts continue to
pass unchanged.
2026-04-25 13:51:50 -07:00