Ports the relay's #18722 pattern to the gateway: DDL runs on a one-connection
pool with statement_timeout 0 that is closed before the serving pool opens, and
SQLSTATE 57014 joins the bounded transaction retry path.
- a push with no epoch can no longer claim a seq-derived dedup key, in the
foreground or from the tray; a forged seq:N could otherwise swallow the
real bell at that seq
- a provider-delivered push with no host catalog, or no fingerprint at all,
stays unrouted instead of falling back to the hostId its raw data carries
- fetch never follows a redirect: a 307 would replay the host proof and the
phone's token to whatever origin the redirect named
- registerPush params are strict and the paired identity is spread last
- a per-device bucket (10/min) bounds a phone looping registerPush, which
costs a gateway write and a synchronous registry write each time
- app.onError logs only the error name and answers a bare 500; hono's default
handler printed the whole error, and a pg error carries the row in detail
- a second per-IP bucket (240/min) runs ahead of the bearer lookup on every
authenticated route, so forged bearers cannot spend the two-connection pool
- one live session per host: minting deletes the host's earlier row
- device-less hosts are pruned after 1 h, not 30 d; any keypair mints one free
- notificationId is printable ASCII, since it becomes the APNs collapse header
- the impersonated FCM probe token is masked in the workflow log
- prevent_destroy on the Apple secrets and the orca_push database
Cloud Run appends the connecting peer to x-forwarded-for; the limiter read
the left-most value, which the caller controls, so a forged first hop earned
a fresh bucket per request.
- Roll traffic back on a failed post-shift check; delete a candidate that
never took traffic; retry the origin probe and the FCM probe.
- Assert Terraform-owned scaling instead of mutating it from the workflow.
- Build before taking the Cloud SQL rollout lease.
- Declare the database pool in Terraform (2 per instance, max 2 instances)
and add the gateway to the connection budget; the previous default put the
shared instance 65 connections over its ceiling.
- State plainly that the shared deploy identity's relay authority is inherited.
- Consent generation guards a register that finishes after the switch went
off; the host is re-queued for unregister instead of recorded live.
- Foreground pushes seed the watermark before adopting the epoch, so a push
on a never-connected session cannot wipe a valid watermark.
- aps-environment follows the build via app.config.js; the iOS release
workflow sets it to production. A bare plugin entry wrote development.
- Pushes the OS showed while closed are marked seen before catch-up replay.
- Token null result is not cached; failed capability probes are retried and
never block an unregister; coalesced summaries are shown but not marked.
- Unresolvable fingerprint routes nowhere and is suppressed in foreground.
- Android channel ensured at boot; capability hook diffs clients by identity.
- A gateway registration the registry cannot persist is enqueued for delete
instead of leaking a live token.
- Unregister outbox re-reads pending per pass, honours enqueues during a
drain, and retries with backoff instead of waiting for the next launch.
- Dispatcher batches registrations by 20 rather than starving the rest.
- 401 compare-and-clear; a 401 after re-auth is unreachable; refused
handshakes and 429s are cached briefly instead of re-handshaking per event.
- Service is stopped on quit.
- Quota reservation takes a per-host advisory lock; READ COMMITTED admitted
a whole burst past the cap (80/80 without, 60/80 with, against Postgres 16).
- Challenge issuance no longer writes push_hosts; the row lands on proof
verification. Stale hosts prune after 30 days. Per-IP token bucket on the
two unauthenticated routes.
- Streaming body limit via hono bodyLimit; a chunked body bypassed the
Content-Length check.
- registrationIds deduped in the schema; per-host device cap of 64; list
bounded to its schema.
- Gateway-side challenge TTL is the specified 10 s, not 40 s.
- APNs stream settles on close as well as end/error.
The production domain-restricted-sharing policy rejects an allUsers
run.invoker member, which the runbook anticipated. Opt the service out of
invoker IAM the way the relay director already does; the host proof is the
authentication either way.
Cloud Verify installs only the cloud workspace, so the gateway suite cannot
import the desktop answerer. Replace the cross-workspace import with a fixed
challenge vector generated from the contract package; the gateway fixture and
the desktop answerer each replay it and must produce the same HMAC. A
transcript drift on either side now fails in that side's own suite.
Declares the Cloud Run service, runtime account, secrets, and orca_push
database behind push_gateway_enabled, true only in production. The deploy
workflow is gated like the relay's, deploys with no traffic, probes /ready
and a validate-only FCM send, then shifts traffic. It runs as the shared
production deploy account because the Cloud SQL rollout lease grant is
foundation-owned; its extra authority is three bindings on the push service.
docs/push-gateway.md carries the import commands for the resources created
by hand and the APNs key rotation procedure.
Fetches the native APNs or FCM token, registers it with every paired host
that advertises the capability, and re-registers on token change. Foreground
pushes are suppressed inside handleNotification against the same seen set
the socket path uses, so nothing shows twice. Taps route by host fingerprint.
One Background notifications switch, off by default, with the disclaimer and
needs-input / finished sub-switches; hidden until a paired desktop is new
enough. Adds google-services.json and the expo-notifications plugin.
Adds the notifications.remote-push.v1 capability, the registerPush and
unregisterPush RPCs on the mobile allowlist, a gateway client with a cached
session and 401 re-auth, a durable unregister outbox, and a dispatcher that
offers every mobile notification to the gateway after the socket fan-out.
The dispatcher is fire-and-forget with one retry and drops registrations the
gateway reports dead.
Puts agentState on the mobile frame and fixes the #4375 wording so a working
agent is never announced as finished. The relay host-proof code moves onto a
shared envelope module with no behaviour change.
A small open-source service that holds the APNs key and FCM credentials and
sends background push to paired phones on the desktop's behalf. Hosts
authenticate with a box challenge and HMAC proof on their pairing key, the
same shape the relay uses, so signed-in and accountless desktops share one
path. Tokens are stored; alert text is held only for the coalescing window.
The contract doc in docs/reference is the source of truth for every wire
shape. The interop test runs the real desktop answerer against a real
gateway-issued challenge so transcript drift fails in CI.
* feat(mobile): structured native Claude chat
Mobile already spoke the structured agent-session protocol for Codex, and the
host already had a Claude capability gate — mobile just never advertised it, so
`projectAgentSessionTabsOut` stripped every Claude tab before it left the
desktop. The structured lane in mobile/ turned out to be agent-agnostic already
(shared reducer, message projection, option catalog, prompt tokens), so this
opens the gate rather than building a second lane:
- advertise `agent-session.structured.claude.v1`
- resolve any structured provider in `resolveMobileNativeChat` via the shared
`isAgentSessionHandleProvider`, instead of a `'codex'` literal
- widen the `agent-session` route type off `'codex'`
- route bare Claude launches through `agentSession.createSupport` like Codex,
which still degrades to a terminal when the host refuses (remote, WSL, win32,
managed-account mismatch, or structured chat switched off)
Deduplicate the create envelope. Renderer and mobile each assembled the
`agentSession.create` params by hand; the fingerprint has to be computed over
the same fields the host recomputes, so both now build it in one shared
`structuredAgentSessionCreateParams`. Mobile's Codex-only launcher becomes
`createMobileStructuredAgentSession(client, worktreeId, agent)` and reuses the
shared display-name map; two copies of a random-UUID fallback collapse into one.
Answer grouped Claude questions. A Claude AskUserQuestion carrying more than one
question — or one multi-select question — is emitted with the real content in
`body.questions` and the flat `options` left EMPTY, so mobile rendered a card
with nothing to tap and the turn stalled with no way out. Codex never emits this
shape. The phone has room for one question at a time, so the group is answered
as steps and submitted once, reusing the shared
`encodeAgentSessionQuestionAnswers` / `isValidAgentSessionQuestionAnswers`
rather than a second encoding.
Prompt responses move into `useMobileStructuredPromptResponses` because grouped
questions carry a multi-step draft the rest of the session does not touch, and
the session hook was at the 300-line cap.
Pin the mobile capability list against the host's parser bounds: it fails closed
to NO capabilities when the array exceeds 64 entries, which would look exactly
like an old client.
Re-pin mobile-session-route-parity: the create-actions edit drops one runtime
string literal and changes one nested function body. Ablated to confirm that
file is the sole cause.
* fix(mobile): derive the grouped-question draft instead of clearing it in an effect
The React Doctor gate flagged the session-change reset as a state adjustment
after a prop change, which renders the stale draft for a frame. Store the
session the answers were collected in alongside them and check it on read, so a
session switch drops the draft during render with no effect at all.
* test(mobile): pin that grouped steps key apart when the questions read identically
Claude can ask the same text twice in one group (once per file, say). The view
keys the question card by its projected content, so identical wording must still
key apart or step 1's checkboxes would be submitted as step 2's answer.
* fix(mobile): harden grouped Claude question answers
* fix(mobile): retry transient structured support probes
* fix(mobile): preserve grouped prompt response compatibility
* fix(mobile): preserve tokenless duplicate choice identity
* fix(mobile): point the launch tests at the generalized create API
The rebase onto #18697 brought its definitive-refusal tests in cleanly, but they
call the pre-rename createMobileStructuredCodexSession, and mobile tsc excludes
test files so nothing caught it. Retarget them and give the agent-copy test a
code that is actually in the definitive allowlist - agent_session_refused now
correctly stays unknown, so it never reached the failure copy it asserted.
* test(mobile): re-pin route parity after the rebase onto main
Main moved its own runtime-string pin to 547; this branch drops the 'codex'
literal from the create-actions gate. Ablated against main's pins to confirm
that file is the sole cause before re-deriving.
---------
Co-authored-by: Merge Sim <sim@local>
The sonnet examples were added for a test cohort. Orchestration must not
choose a model on the user's behalf: pass --model only when the user named
one, otherwise inherit the configured agent default.
* Recover branch compare on visibility change
Add recovery mode that reuses cached branch comparison data when the window regains focus instead of clearing results and forcing a refresh. This preserves the diff display during operations like rebasing that may cause the window to go to the background.
* Retry failed branch comparison results
Cached branch comparison results with error status are now excluded
from the cache-hit check, ensuring they are retried rather than
silently reused. This fixes missing diffs during rebasing.
* Decouple branch compare recovery from refresh kinds
Recovery is now a dedicated callback invoked independently on visibility changes, rather than a refresh kind. This allows pending recoveries to queue during in-flight requests, improving handling when the window regains focus during rebasing or other operations.
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 19 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$962 | $\color{#cf222e}{\Huge{\mathbf{−}}}$136 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$826 |
| Prod | 18 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$295 | $\color{#cf222e}{\Huge{\mathbf{−}}}$116 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$179 |
<!-- /orca-pr-loc -->
## Symptom
Live 2026-09-05 (Orca 1.4.198 client, Ubuntu host): both relay processes `kill -STOP`ped for 20 s, then `-CONT`. The client redeployed while the host was frozen. Its fresh daemon lost the socket bind (`Socket path already in use`) but had **already rewritten** `relay-<id>.sock.credential`. The surviving daemon kept its in-memory credential, so every later `--connect` got `Endpoint credential mismatch; closing socket`, then `Grace started … timeoutMs=0 … ptys=1, clients=0` every ~20 s, forever. Only a manual `kill -TERM` cleared it. Receipts: `review-archive/orchestration-v3-pr16904/smoke-receipts-t012b/E16,E17,E18,E24`.
Three independent defects kept the wedge alive; each is fixed at its own seam.
## Fix
**1. The relay daemon owns credential publication (race-free under two concurrent starters).**
`relay-daemon.ts` binds the socket first, then publishes via the new `src/relay/relay-endpoint-credential-publication.ts`: adopt a valid pre-existing file (older clients still pre-write), else mint 32 random bytes and write temp+rename at 0600. A start that loses the bind exits inside `listen()` and never reaches the file. Why this option and not restore-on-loss or a client-side write: the only process that can *prove* ownership is the one whose `listen()` succeeded, and that proof is atomic with the bind. The client-side pre-write (`ssh-relay-endpoint-credential.ts`) and the launch-command `chmod 600`/`icacls` are removed on POSIX and Windows. The racing test also exposed that macOS reports a mid-bind collision as `EEXIST` rather than `EADDRINUSE`; `relay-socket-ownership.ts` now treats both as "held or stale".
**2. The client distinguishes "no daemon" from "daemon present but not answering", and never rewrites.**
A credential refusal is now typed on the wire: the daemon replies `orca-relay-handshake-credential-mismatch` (same frame type, no new opcode) and the bridge exits **43**; `waitForSentinel` maps it to `RelayCredentialMismatchError`, which the takeover treats as handshake-refusal evidence exactly like exit 42. A relay that holds the endpoint but **never refused** (the stalled-host shape: kernel backlog accepts the probe, handshake gets no answer) is now `RelayEndpointUnresponsiveError`, routed to the relay-lost backoff instead of the terminal Reset Relay path. Silence is not a decision (`docs/reference/ssh-execution-boundary.md`).
**2b. Deploy honours the verdict.** The 40 s live run exposed that the `--connect` catch block in `deployAndLaunchRelay` predates the incumbent probe and swallowed both verdicts as "probe failed, launch fresh", so a fresh daemon was still launched over the live one (it lost the bind by luck, which is exactly the collision in the incident). Held and Unresponsive now propagate; the session backs off on Unresponsive and surfaces Reset Relay on Held. Red-first in `ssh-relay-deploy-incumbent-verdict.test.ts`.
**3. The daemon cannot be wedged by a rotated file, because nothing can rotate it.**
The credential lives in the content-hashed relay dir, and after (1) the only writer is the daemon that owns the socket, so the "file changed under a live daemon" state the incident depended on is no longer reachable in-product. The credential is therefore fixed for the daemon's lifetime, as a plain secret should be. A hand-edited file is refused with the typed reply until restored (tested). Startup adoption of a pre-written file applies an owner-only + same-uid rule (review finding): anything else is replaced by a fresh mint. An earlier revision of this PR also re-read the file on mismatch and adopted it; that was removed as unreachable machinery that turned the credential into a per-handshake file-ownership check.
**3b. Fail closed between bind and publication.** A client that arrives after `listen()` resolves but before the credential is set is refused, not admitted as `unproved`. Nothing can be delivered in that window today; the guard makes the boundary structural instead of an event-loop ordering fact. Red-first in `relay-reconnect-listener-credential-gate.test.ts`.
**Wire compat.** New optional handshake reply only; an old `--connect` hits `Unknown handshake type` and exits 1 pre-sentinel, which it already treated as a generic failure. New daemon adopts an old client's pre-written file; new client still passes `--credential-file` so an old daemon reads it as before. Absence of exit 43 is never used as evidence.
**Also.** `terminal create` on a reconnecting SSH host now says what to do instead of a bare `No PTY provider for connection "<id>"` (prefix preserved; the renderer matches it).
## Tests (red first)
- `src/relay/subprocess.test.ts`: two `--detached` starts race one socket + credential file → exactly one reaches the sentinel, loser exits 1 with `Socket path already in use`, file valid + 0600, a `--connect` reading it reaches `relay.status` and reports the winner's pid. Red before (both starters died: daemon required a pre-existing file), green 6/6 after.
- `src/relay/relay-endpoint-credential-publication.test.ts`: mints after bind; adopts a pre-written 0600 file; replaces a pre-written 0644 file with a fresh mint; refuses a stale credential with exit 43 while still serving the real one, and keeps refusing a rewritten file until it is restored.
- `src/relay/relay-reconnect-listener-credential-gate.test.ts`: a client in the bind-to-publish window is refused and never attached; after publication the right credential is accepted and a wrong one refused; a daemon launched without a credential file is not gated. Red without the guard.
- `ssh-relay-deploy-incumbent-verdict.test.ts`: live-but-silent incumbent → `RelayEndpointUnresponsiveError`, refused → `RelayEndpointHeldError`, and in neither case is `--detached` launched; a failed `test -S` probe still launches fresh. Red 2/3 without the deploy change.
- `ssh-relay-deploy-helpers.test.ts` (exit 43), `ssh-relay-endpoint-takeover.test.ts` (refused → Held even with no `lsof`; silent → Unresponsive, nothing unlinked or signalled), `ssh-relay-session-terminal-error.test.ts` (Unresponsive → `onRelayLost`, not terminal). Deploy/namespace/native-deps tests updated to assert the client writes **no** credential.
## Live proof
New `tests/e2e/ssh-docker-relay-stall-credential.spec.ts` (claimed in `run-ssh-docker-e2e.mjs` and PR source routing), two cases: `kill -STOP` every relay pid in the container, send input during the freeze, hold **20 s** (the incident's duration, which races the mux liveness timeout) or **40 s** (past it for sure), `kill -CONT`; assert status back to `connected`, same pty, same daemon pid, same credential inode and content, relay.log did not shrink (a relaunch truncates it) and has zero `Endpoint credential mismatch` / `Socket path already in use` lines, in-stall input delivered at most once.
Run output (local, fixture image `orca-e2e-ssh-relay:3a864c665ba2cefd`, `ORCA_E2E_SSH_DOCKER=1 SKIP_BUILD=1 ORCA_E2E_FORWARD_APP_LOGS=1 … --project electron-headless --workers=1`, head `c2c20fd994`; re-run identically on the final head after the credential-lifetime change, 2 passed (1.7m), same annotations, and the bind-to-publish refusal never fired):
```
✓ keeps the same daemon and credential across a 20s relay freeze (38.3s)
relay-processes-stopped: 2 relay-processes-continued: 2
bridge-pids-before-after: 480 -> 480
socket-clients-accepted-before-after: 1 -> 1
in-stall-input-delivered: 1
✓ backs off and reattaches, never relaunching, across a 40s relay freeze (57.5s)
relay-processes-stopped: 2 relay-processes-continued: 4
bridge-pids-before-after: 480 -> 1202
socket-clients-accepted-before-after: 1 -> 3
in-stall-input-delivered: 1
2 passed (1.6m)
```
Client log in the 40 s case shows the new path end to end: `Relay channel lost … reconnect attempt 1/6` → `Socket probe result: "ALIVE"` → `Socket reconnect failed … Relay failed to start within 10s` → `Relay endpoint incumbent: … verdict=live evidence=accepted-connection holders=unenumerable` → `Failed to re-establish relay … A relay still owns … but did not answer the handshake … Orca will retry` → `reconnect attempt 2/6` → `Reconnected to existing relay via socket`. The 20 s case never left the frozen bridge (same bridge pid, one accept), so it exercises the "silence is not death" side of the same race. The 20 s case passed 6/6 across the session; the 40 s case was red on the prior head (`Socket path already in use` + `Startup failed: listen EADDRINUSE` in relay.log from the swallowed verdict) and is green after 2b. Before the fix the same injection produced a fresh daemon that rewrote the credential and a survivor refusing every client.
The `relay-processes-continued` count exceeds `stopped` in the 40 s case because the timed-out client's `--connect` bridge and the loser-side processes are parked behind the frozen listener when `CONT` runs; they exit on their own once it resumes.
## Gates
`pnpm test src/relay src/main/ssh` 332 files / 3884 tests pass · `pnpm typecheck:tsc:node` clean · `check:code-quality:changed` 0 findings · `check:react-doctor:changed` 0 findings · `pr-e2e-gate-contract.test.mjs` 42 pass · no lint disables or max-lines bumps added.
## Noted, not fixed here
- `terminal list` `orphaned:false` / `terminal close` `ptyKilled:true` for a pane whose relay is gone (`orca-runtime-stop-explicitly-closed-tab-ptys.ts`): different seam, `@ts-nocheck` characterization-covered file.
- On a host with no `lsof`, a stalled relay still cannot be enumerated as the holder; it is now retried rather than declared held, but a relay frozen past the backoff budget still ends in the existing "reconnect manually" banner.
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$12401 |
<!-- /orca-pr-loc -->
## ELI5
Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.
## What changed
- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.
## Why
User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.
## Linked issues
Fixes#15180. Fixes#17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.
## Review record
This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.
**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:
- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.
Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).
A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.
A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.
Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).
## Testing
- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).
## Notes
- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
* fix(chat): preserve message copy focus
* fix(chat): scope covered-xterm focus guard to the chat leaf
Chat view mode is a tab flag, but only the chat leaf's xterm is covered. In a
split chat tab with a terminal leaf active, the tab-level guard skipped the
terminal's resume focus and the tab-wide deferred focus then landed on the
covered chat xterm. Decide per pane: resume and window-wake read the active
pane's container, and the surface focus query skips leaves hosting the chat
root.
* fix(chat): close covered terminal focus fallbacks
---------
Co-authored-by: Merge Sim <sim@local>
* test: run localhost SSH terminal and hooks in CI
* test: isolate localhost SSH session fixtures across repetitions
* test: route remote agent hook source changes to localhost journey
* test: record localhost SSH reliability evidence and remaining gaps
* test: route the real SSH session hook authority
All 19 general cells on 4916ed67, selector gen 148 -> 186, 0 serving-process
exits across the roll. Three waves used the no-restart mode=rollback resume
(c13 transient trust-probe 409; c26/c21 post-apply runtime-status 503 shed).
Checklist: 2.3, 4.1, 4.3 relay side deployed; status header 2026-09-06.
* test: restore SSH bulk-open freeze coverage in headed CI
* test: record ten passing headed SSH freeze repetitions
* test: record ten passing headed SSH freeze repetitions
* test: route changed SSH freeze spec only to its dedicated lane
* test: continuously exercise real WSL terminal launch and paste
* test: establish live WSL reader before changing default shell
* ci: pin WSL kernel installer and participation selectors
* ci: route deleted WSL paths and record immutable run evidence
* test: require exactly three WSL repetitions in lane contract
* fix(packaging): include Claude agent SDK at runtime
* test(packaging): cover spaced runtime imports
* fix(packaging): verify every emitted main file for bare runtime imports
The packaged-main verifier read two fixed entry files, but rolldown hoists
modules shared by two entries into out/main/chunks. jsonc-parser is reached
only from a chunk today, so nothing verified it, and the agent-hooks entry
the list names contributes no coverage at all. An import that migrates into
a chunk would silently stop being checked -- the same blindness that let the
missing Claude agent SDK ship.
Scan every out/main/**/*.js entry in the asar instead, keeping the two
required-file assertions as a build-integrity check. Measured against the
shipped 1.4.198 app: 93 entries in 72ms, reporting the absent SDK and
nothing else.
Also tighten the specifier match with a (?<![.\w]) lookbehind. Orca has
three registry methods of its own named require(), two taking a string key,
so a minified registry.require('public-a') otherwise reads as a bare module
specifier and fails packaging with a confusing error -- a risk the wider
file set would have multiplied. The lookbehind drops nothing real: detection
over the shipped bundle is identical with and without it.
* test(packaging): cover the missing packaged main entry assertion
The required-file check had no test, so the refactor that split it out of
the scanning loop could have dropped it silently. Removing the assertion
now fails this case.
* docs(packaging): name the embedded-source-string limit of the main scan
ssh-relay-deploy builds a probe script for the REMOTE host as a string, and
its require("node-pty") / require("@parcel/watcher") survive into
out/main/index.js, where this scan counts them as desktop-main imports. Both
are packaged, so it is benign today, but a remote-only dependency added to
that script would fail desktop packaging with a false message -- and the two
obvious fixes (ship the remote dep, or weaken the guard) are both wrong.
Separating an embedded string from real code needs a parser.
* test(packaging): pin the exact import shape oxc emits for the SDK
The fixture only carried the spaced `import (` variant, so nothing pinned
the form a shipped build actually contains. Use the real emitted shape --
`p??=import(`@anthropic-ai/claude-agent-sdk`)`, no space, backticks, and the
`??=` that precedes it -- and keep the spaced variant on the second entry so
both stay covered.
* fix(packaging): keep the main scan able to see a spread require
The `(?<![.\w])` lookbehind also rejected `[...require("pkg")]`, because the
third dot of a spread satisfies it. That trade is not symmetric: excluding a
member call costs a loud release-build failure if it ever misfires, but
excluding a real specifier is this guard going blind -- the failure mode the
whole verifier exists to prevent. Readmit a dot that ends a spread.
Zero occurrences in the shipped bundle today, so this was latent. The chunk
test's asar mock now also emits directory nodes, because real listPackage does
and extractFile throws on them -- that makes the `.js` anchor's load-bearing
role something the tests can actually catch.
---------
Co-authored-by: Merge Sim <sim@local>
* fix(native-chat): keep saved options on structured route
* fix(native-chat): seed structured session options
* fix(native-chat): preserve create wire compatibility
* fix(native-chat): keep create replayable across an option change
Seeding host-resolved options into the attach fingerprint put a mutable
value into the durable operation identity. A create whose outcome was
unknown, retried under the same operation id after the user reselected a
model, re-resolved different options and hashed to a different
fingerprint — so the ledger refused it as a conflict instead of
replaying. That refusal is not definitive, so no legacy fallback fires
and the launch has no recovery.
Options are the session's initial state, not its identity, and the
reservation still carries them to the record. Excluding them also makes
the digest byte-identical to the pre-change one in every case, not just
when no options resolve.
* refactor(native-chat): name the structured launch option seed
The create-intent resolver narrowed saved options to model/effort with an
inline key literal, inside a file carrying @ts-nocheck — so neither the
key list nor the string narrowing had a typechecked or testable home, and
the repo already expresses this concept as a named shared shape.
Move it to resolveStructuredLaunchSeedOptions beside the persisted
settings it reads, where it is typechecked and unit-tested, and document
why the seed is exactly model and effort: they are the only ids the
picker persists that both providers also accept as strings.
No behavior change. Adds coverage for a non-string persisted effort,
which settings.json can hold and the durable record must not carry.
* test(native-chat): name the structured routing pin for what it asserts
The case drives a saved Codex model and effort through the launch path,
but the structured create intent it asserts on carries no options, so it
pins the route and not the preservation its name claimed. Preservation is
pinned host-side, where the seeding actually happens.
* test(native-chat): pin the empty seed the record cannot carry
valuesByModel is merged over the resolved model, so a stored `model` key
can blank it — the seed then empties out and must resolve to undefined.
Nothing covered that branch, so returning the empty map unguarded stayed
green.
It matters because emitting `{ model: '' }` fails the record's
bounded-string guard, and agent_session_options_invalid is not a wire
refusal code: classifyStoreFailure rethrows it, the client reads the raw
error as an unknown outcome, and the launch strands with no fallback.
Corrects eb33ee6bb3, whose message claimed no behavior change. That
extraction also stopped emitting empty and whitespace-only values, which
is a fix on six input shapes, not a pure move.
---------
Co-authored-by: Merge Sim <sim@local>
Jinwoo-H
Brennan BensonandMerge Sim
Jinwoo Hong
Jinjing
Neil