mirror of
https://github.com/stablyai/orca.git
synced 2026-10-30 16:47:34 +00:00
d9bc75752c8a2309049b58c3aa635b22ef8d4a7a
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
06a607a1d7 |
feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$12401 |
<!-- /orca-pr-loc -->
## ELI5
Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.
## What changed
- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.
## Why
User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.
## Linked issues
Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.
## Review record
This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.
**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:
- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.
Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).
A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.
A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.
Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).
## Testing
- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on
|
||
  
|
e741ff1318 |
fix(wsl): scan agent sessions only in running distros (#17072)
* fix(wsl): scan sessions only in running distros * test(ai-vault): pin WSL discovery platform * fix(wsl): suspend transcript watchers for stopped distros * test(wsl): pin transcript scan gate platform * fix(wsl): settle stopped transcript loading * fix(wsl): add last-known-good fallback and backoff to running-distro discovery listRunningWslDistrosAsync failed closed on any probe error (timeout, ENOENT, wsl.exe hiccup), indistinguishable from "no distros running". A 2s poll (wsl-transcript-running-observer.ts) calls it indefinitely while any WSL transcript tab is open, so a persistently broken wsl.exe silently made every WSL session vanish app-wide with no way to tell "discovery broken" from "distro stopped", and re-spawned wsl.exe every 2s forever. Extract a dedicated cache/backoff module (wsl-running-distro-cache.ts, mirroring the sibling machinery already in wsl.ts for the full distro list) so a probe failure falls back to the last-known-good running-distro list and backs off further probes, while a genuine empty result (no distros running) stays authoritative. Add a consumer-level test simulating a sustained wsl.exe outage across a live transcript-watcher polling session, asserting the observer keeps reporting "running" and that real wsl.exe spawns stay bounded. * fix(build): list the new WSL cache module in the web typecheck project config/tsconfig.tc.web.json enumerates its files explicitly, so a new module imported by wsl.ts fails the full typecheck with TS6307 until it is listed. pnpm tc:node passes without it, which is how this got missed. src/main/wsl.ts(13,8): error TS6307: File 'src/main/wsl-running-distro-cache.ts' is not listed within the file list of project 'config/tsconfig.tc.web.json'. --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> Co-authored-by: Neil <4138956+nwparker@users.noreply.github.com> |
||
 Brennan Benson
|
07f2e14c08 |
refactor(codex): make WSL account surfaces direct-home aware (#16499)
* refactor(codex): make WSL account surfaces direct-home aware * fix(wsl): keep Codex relay hooks on managed runtime home * test(wsl): assert relay hooks use managed Codex home |
||
 Jinjing
|
7b326e212f |
Gate WSL transcript filesystem ops with timeout and capacity limits
Why: WSL filesystem access can stall when a distro hangs, and stalled operations must degrade gracefully rather than misreport as "not found". Distinguish gate refusals (timeout, capacity, unavailable) from actual missing files so callers can retry or fall through to alternatives. Fail fast when stuck I/O holds a permit to prevent caller pileup. Route-level sequencing and waiter deadlines keep one stalled distro from blocking others. |
||
|
OrcaWinandOrcaWin
|
fb3a3c5643 |
perf(native-chat): probe WSL transcript paths asynchronously (#13265)
* perf(native-chat): probe WSL transcript paths asynchronously The WSL transcript resolvers probed `\wsl.localhost` UNC candidates with existsSync. Those paths are served over 9P, so a stopped or unreachable distro blocked the Electron main thread instead of falling through to the next candidate. - host-readable-transcript-path: the `pathExists` dep is now async, defaulting to fs/promises.access. The per-distro loop stays sequential — the ranked order exists so the owning distro wins, and probing every distro at once would fan 9P calls out to ones the user deliberately left stopped. - session-file-resolver: dropped the existsSync guard outright rather than converting it. walkSessionFiles already yields [] for a missing or unreadable root, so the guard was redundant as well as blocking. * perf(native-chat): stop the resolve poll mkdir-ing the Codex runtime home codexSessionsDirs() only needs the managed home's path to scan it, but called getOrcaManagedCodexHomePath(), which mkdirSyncs. That put a synchronous fs call back on the 500ms-5s resolve poll the surrounding async probe work just cleared, and materialized the runtime home as a side effect of a read-only lookup. Use the resolve-only variant the module already exposes for this case. A missing root walks to no matches, so behavior is unchanged. * perf(native-chat): gate WSL transcript fs access behind a shared queue Serializes and dedupes WSL 9P filesystem probes (access/readdir) so a stopped or slow distro can't exhaust the shared libuv threadpool or block unrelated local filesystem work. Routes Codex session-tree scans and path-existence checks through the new wsl-transcript-fs-gate, while sharing in-flight scans across concurrent callers. * perf(native-chat): prioritize exact WSL transcript probes over scans - Add cancellation (AbortSignal) throughout session resolution and directory walking so an unsubscribed transcript watch stops in-flight WSL filesystem work instead of leaking it. - Split the WSL fs task gate into exact vs scan priority lanes with per-route concurrency, so a live transcript access probe is never queued behind a directory scan on another distro/provider. - Extract Codex WSL session path scanning into its own module that shares one root snapshot across concurrent session-id lookups and refreshes on a shared miss to see post-start file creation. - Skip the async WSL probe entirely for local paths via existsSync. * fix(native-chat): harden WSL transcript cancellation * fix(native-chat): finish transcript cancellation --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
915050f30a |
fix(native-chat): resolve WSL Codex transcripts so Chat UI renders responses (#12473)
* fix(native-chat): resolve WSL Codex transcripts so Chat UI renders responses Codex reports a guest Linux transcript path. On a Windows host `existsSync` resolved it against the current drive (`C:\home\...`) and discarded it, and the id-based fallback only searched host roots. `resolveSessionFilePath` returned null forever while the watcher reported `watching: true`, leaving Chat UI permanently empty. Translate the guest path to its host-readable UNC twin, classifying **before** any `existsSync` probe on win32 so the `C:\home` false positive cannot fire. Adapted from #10639 with one required correction: it uses the **async cached** WSL seams (`listWslDistrosAsync` / `getWslHomeAsync`) rather than the `execFileSync` ones, which would stall the Electron main thread for up to 5s per tick of the resolve-poll loop on a cold distro. Fixes #10326 Co-authored-by: Orca <help@stably.ai> * fix(native-chat): stop the WSL transcript probe firing every poll tick Three follow-ups from review of the WSL Codex transcript fix: - The UNC translation was retried on every fast resolve-poll tick (measured 10 sync UNC stats per 100ms) because only a successful result was memoized. Gate the retry to the slow fallback cadence. - A non-empty WSL home list was cached for the process lifetime, so a distro that was still booting during the first probe stayed excluded forever. Expire both branches; getWslHomeAsync caches successes, so a refresh only re-spawns wsl.exe for the distros that actually failed. - codexSessionsDirs enumerated every distro's home eagerly, waking distros the user left stopped even for native-Windows panes. Make the WSL roots a lazy tier consulted only after the host's own Codex roots miss. The resolve-poll suite became platform-dependent and only passed off Windows; pin the platform and add explicit win32 coverage. --------- Co-authored-by: Orca <help@stably.ai> Co-authored-by: OrcaWin <alpha-eng@stably.ai> |