Commit Graph
5 Commits
Author SHA1 Message Date
Brennan Benson 993183afd7 fix(terminal): every explicit terminal close commits through one main transaction (#22929)
* fix(terminal): every explicit terminal close commits through one main transaction

A renderer save cannot shrink terminal membership once main owns a repo's
topology, so desktop tab and pane closes, CLI split-pane closes and mobile
split-pane closes only became durable when the killed process's exit retired
the surface. A close whose kill failed or threw, or whose exit was never
certified, came back after a reload.

Every close now reaches closeTerminalSurface: the renderer sends an explicit
intent for user and cleanup closes, the CLI and mobile split-pane closes commit
the pane after their stop, and the headless and relayed mobile closes reuse the
same commit. A failed flush keeps the in-memory removal and no longer cancels
the kill. Exit retirement is unchanged.

* fix(terminal): tell the desktop renderer to drop a split pane main closed

A CLI or mobile close of one pane in a split commits the pane in main, but the
desktop kept showing it until reload when no exit arrived to remove it. The
close now sends a leaf-addressed notice: a mounted pane closes by leaf id, and
a parked tab collapses its stored layout. Addressing by leaf makes the notice
and the renderer's exit handling no-ops after each other, which replaces the
numeric pane-id notice that could close the whole tab when the exit won.

* fix(terminal): a pane close never widens into a whole-tab close

A leaf-addressed close fell through to the whole-tab close whenever main's layout no longer
held that leaf as one of several. Main's exit handling retires an exited split pane from the
saved layout, so closing that pane afterwards (the exited-pane overlay's Close, or a CLI close
whose stop delivers the exit first) removed the whole tab, live sibling included, and the
next renderer save could not restore it. A pane close is now a no-op unless its leaf is in a
multi-pane layout.

Also updates two mobile split-close assertions to expect the leaf-addressed notice, and adds a
test that a relayed mobile close of a renderer-listed tab still reaches the renderer's pin guard.

* test(terminal): cover the PTY-handle branch of a CLI split-pane close

The existing CLI split test resolves its handle through the renderer graph, so the branch
that closes a runtime-owned pane by its PTY handle had no test failing without its commit.

* fix(terminal): a CLI pane close with an unconfirmed stop closes only that pane

`orca terminal close <handle>` on one pane of a split used to close the
whole tab, live sibling included, whenever that pane's stop could not be
confirmed (for example an unreachable SSH host). An unconfirmed stop is
unverifiable, not a reason to drop siblings: the close now commits only
that leaf, tells the renderer to drop that leaf, and leaves the owed kill
to the controller's existing SSH pending-kill path.

On a host where no renderer lists the tab, main now also removes the
closed pane from the paired-client snapshot (with its retirement proof),
since no exit may arrive to do it.

* refactor(terminal): one resolver decides whether a pane close becomes a tab close

Every explicit close now states its target as `{kind:'tab'}` or `{kind:'pane', leafId}`; no
optional leaf id silently means the whole tab. Main resolves a close it started in exactly one
place, reading the copy of the tab's panes its layout owner holds (the renderer-published layout
for tabs the desktop renderer lists, main's session layout otherwise). Only `last-pane` escalates,
through the existing tab path so the renderer's pin guard still runs; an unknown pane never widens.

- The CLI and phone paths drop their per-site sibling counts for the resolver.
- The notifier splits into a tab-only close and a leaf-addressed pane close.
- The headless tab closer takes a parent tab id, so a pane row cannot reach it.
- A phone close of one pane on a host with no desktop window now stops and closes only that pane.
- A phone close of one pane with no live process record closes that pane, not its tab.

* fix(cli): an unverifiable stop says the close happened

`orca terminal close` still exits 1 when the process stop cannot be verified, but its message now
says the terminal was closed and names the host's reason, instead of "close failed". It promises
that the kill retries on reconnect only when the SSH relay itself never answered the stop, the one
case a recorded kill order backs.

* fix(terminal): a phone pane close commits even when its kill fails

A paired client's close of one pane threw `terminal_close_failed` before committing anything when
the controller reported the kill failed, so the pane stayed. The kill is now best-effort, as it is
for a whole-tab close: the pane's removal always commits and the failure stays on the PTY's
liveness verdict.

* fix(terminal): a pane close widens only when a copy shows it is the last pane

The close resolver read an owner copy that records no panes as "the tab has
one pane", so a CLI close of one pane of a split, addressed while the
renderer listed the tab before publishing its panes, closed the whole tab.

Every copy now counts only if it records at least one pane, read in the
owner's order with the published rows as the last fallback, and a pane
close widens only when a copy lists that pane as the tab's only one. An
unsplit tab whose saved layout predates its pane still closes: its
published row names the pane.

* fix(cli): promise a kill retry only when the host recorded the kill

The close receipt inferred "the kill retries when the host reconnects" from
the stop reason's text, which a new transport message or a reworded error
would silently break.

An explicit close now records the replayable kill order when its stop goes
unconfirmed, before sending the follow-up kill (whose own failure is
recorded only once its RPC settles), and reports that on the receipt as an
optional `pendingKillRecorded`. The CLI promises the retry only from that
field, so an older host, which never sends it, gets no promise.

* test(pty): justify the controller cast the recorded-stop tests extend

* fix(terminal): parse the close target with typed narrowing

The low-evidence lint gate rejects Reflect.get and broad object parameters,
which failed static analysis. Narrow with 'in' checks instead and cover the
boundary parser's accept and reject cases.

* fix(terminal): a desktop tab close is not refused by a split that bound while it waited

The renderer has already removed and killed a tab it closes, so its close intent now skips the
owner fence phone and CLI closes use. Before, a split pane whose binding was admitted between the
close request and its durable write made main refuse the close, and the tab came back on the next
launch whenever its processes did not exit.

* chore(terminal): note that closedByLayoutOwner goes away once main owns the terminal layout

* test(terminal): reload the close-intent fixture through the SQLite profile store

Main now requires a SQLite profile-state authority for a writable Store, so the save-and-reload
close tests build and reopen their store through the shared SQLite test harness.
2026-09-27 15:29:22 -07:00
b3e0a33fa4 fix(runtime): agent-neutral wait-blocked reasons (#19749)
* fix(runtime): agent-neutral wait-blocked reasons and non-Gemini Antigravity readiness

Reported by a user via the in-app help menu (report "not captured", 1.4.198).

The trust/interactive/update/cwd prompt matchers are agent-agnostic - they
match on dialog wording and never inspect the pane's agent - yet emitted
hardcoded codex-* reasons. Those reached users verbatim in worker receipts
(local-worker-start, federation), two automation surfaces, and raw CLI
output, so an Antigravity user was told they had a Codex problem.

findAntigravityReadyPromptIndex also required the model line to start with
the literal "gemini". Antigravity CLI is not Gemini-only, so a non-Gemini
session never registered as ready, stale trust text was never superseded,
and the pane stayed blocked - which is why dispatch --inject answered
agent_prompt_blocked.

Add agent-neutral reasons additively (codex-* members kept on the wire per
docs/reference/remote-wire-compatibility.md, with a legacy alias for older
hosts) and decide Antigravity readiness structurally: header, then
model/account rows, then the prompt caret.

codex-model-migration-prompt and codex-hooks-review-prompt stay Codex-named
- both key on Codex's own wording.

* fix(runtime): finish the agent-neutral rename, revert the Antigravity readiness rewrite

Review follow-up on this branch. Splits the two halves of the original commit:
the reason rename lands, the Antigravity readiness detector goes back to
merge-base until someone captures a real transcript.

Rename half:

- 'hooks need review' + 'press enter to confirm' inspects no agent, so it now
  publishes agent-hooks-review-prompt. That was the last agent-agnostic codex-*
  emission left, and it is the one the original report was about: a Claude Code
  user hitting a hooks dialog still read "codex-hooks-review-prompt".
- The legacy alias is applied at all three surfaces that render a raw reason,
  not just the CLI. describeTerminalWaitBlockedReason() is the single formatter;
  the worker and federation "Agent startup blocked:" receipts use it too. Kept
  one-directional: nothing consumes agent-* -> codex-*, since an old client
  renders with its own shipped code.
- Restores the compat note deleted at the permission-choices site. The Rule 1
  citation is correct - remote-wire-compatibility.md names this enum by name.

Antigravity half, reverted:

findAntigravityReadyPromptIndex goes back to merge-base (header + a 'gemini'
model line + a lone '>' caret) and antigravity-ready-prompt-index.ts is removed.

Executing both builds against constructed tails, the rewrite read a live
startup dialog as ready. Adding the account row from this repo's own ready-screen
fixture to five silent startup dialogs (sign-in, model picker, theme picker,
privacy notice, update banner) flipped all five from unready to ready; so did any
narration line containing an email address, with no account row at all. Readiness
is what gates typing the task prompt into the pane, so that path types a task
prompt into a live authentication dialog. Merge-base returns unready for all ten.

The rewrite also did not reliably fix the wedge it targeted: with no account row
and a non-Gemini model - a personal or API-key user - it still returns unready.

No real Antigravity transcript exists in this repo. The cursor-agent rules are
derived from captures under src/main/runtime/__fixtures__; Antigravity has no
equivalent, and every attempt so far has been tuned against a hand-written
5-line fixture. A false negative (the agent waits) is safer than a false positive
(we type into an auth dialog), so this ships the known behaviour.

Reverting restores a pre-existing gap, not a regression: a non-Gemini Antigravity
session wedges on merge-base too. Closing it needs a captured ready screen and a
captured dismissed-dialog screen, for a personal/API-key account as well as a
Business one.

Tests:

- Ten ratchet fixtures pin the shapes any replacement detector must refuse - the
  five silent dialogs with an account row, and each with a narrated email. All
  ten fail against the reverted rewrite.
- Vacuous tests rewritten so they fail without the code they cover: the CLI
  alias tests asserted only the absence of a suffix, and the worker receipt test
  asserted the raw token. Tests that are characterization rather than a guard
  now say so on the line above.

---------

Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local>
Co-authored-by: Neil <neil@stably.ai>
2026-09-10 23:44:23 -07:00
Jinwoo Hong 06a607a1d7 feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​12401 |

<!-- /orca-pr-loc -->

## ELI5

Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.

## What changed

- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.

## Why

User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.

## Linked issues

Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.

## Review record

This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.

**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:

- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.

Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).

A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.

A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.

Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).

## Testing

- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).

## Notes

- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
2026-09-06 14:34:03 -04:00
Brennan Benson 7afce2ea41 fix(ssh): stop reporting a confirmed kill when the SSH provider is gone (#14977)
* fix(ssh): stop reporting a confirmed kill when the SSH provider is gone

A detached relay PTY is designed to outlive the provider that addressed it
(it ignores SIGHUP and ships with an unlimited grace), so "the SSH provider
is no longer registered" is lost contact, never evidence the remote process
stopped. Both stop primitives in the PTY controller returned `true` from
that branch, and every caller downstream reported the fabricated success:
the CLI printed "PTY killed.", worker-stop settled the dispatch as stopped,
and — because the stop "succeeded" — the unstopped-PTY gate never ran, so
worktree removal walked straight past a live remote agent.

`kill`/`stopAndWait` now still tombstone the local lease but report an
unconfirmed stop and record why, using the three-verdict vocabulary the
worktree teardown gate already spoke (`live` / `unverifiable` / `exited`),
promoted out of that module into `src/shared/pty-liveness-verdict.ts`.
The close receipt, the CLI wording, worker-stop and the removal gate all
read that verdict instead of inferring an exit from silence.

The same rule fixes the mirror-image defect: the aggregate inventory only
enumerates registered providers, so a dropped relay clears `connected` for
every remote PTY at once. The sweep now separates the provider answering
"absent" (an exit) from no provider being able to answer (lost contact), so
worker-stop stops claiming `exited` from a disconnect.

The `connected` wire field is unchanged in meaning and shape.

* fix(orchestration): apply the same honesty to the federation stop path

The federation host runs its own copy of the worker observation and stop
logic, with the same two defects: `inspectRemoteAttachment` read a dropped
relay's `connected: false` as `exited`, and `federationStop` settled the
dispatch as stopped from a close it never confirmed — relaying a fabricated
success all the way home to the coordinator.

Two guards also had to move so the honest verdict does not become a new
refusal. `federationRead` gated on `status !== 'running'`, which would have
rejected a connected terminal the moment a stop lost contact with it; it now
gates on `status === 'exited'`, which is equivalent for every pre-existing
status given the two guards beside it. Local `workerStop` likewise still
attempts the close when the verdict is `unverifiable` — losing contact is a
reason to report the outcome honestly, never a reason to stop trying.

The show observations now carry the reason alongside the status, so a bare
`unverifiable` is actionable. Both are new optional fields.

* fix(ssh): preserve unconfirmed stop verdicts across consumers

* fix(ssh): use canonical live verdict wording

* fix(ssh): refuse wrong-host teardown verification

* test(orchestration): confirm worker release teardown

* fix(orchestration): negotiate honest worker stop receipts

* fix(agent-teams): fence uncertain teammate respawns

* fix(ssh): avoid duplicate missing-provider teardown

* fix(orchestration): preserve archives across release retries

* fix(ssh): preserve verdicts across synthetic kill exits

* fix(ssh): preserve liveness evidence across teardown

* fix(agent-teams): replace panes only after confirmed stop

* fix(ssh): distinguish host exits from relay loss

* fix(ssh): narrow concurrent inventory verdicts

* fix(orchestration): serve archives after uncertain release

* fix(orchestration): expose unverifiable read liveness

* test(ssh): align liveness assertions with verdicts

* fix(ssh): preserve host scope across inventory failures
2026-08-17 00:11:19 -07:00
Neil 339045b150 fix(runtime): coalesce concurrent host terminal focus (#11841)
Bound exclusive host navigation to a generation-aware latest-wins
single-flight so bulk open and switch fan-out stay responsive on large
remote fleets. Add freeze repro harnesses and navigated settlement.
2026-08-03 02:18:05 -07:00