Commit Graph
221 Commits
Author SHA1 Message Date
0e3b71f605 fix(session): give an SSH workspace one owning partition so its tabs stop round-tripping as deletions (#19572)
* fix(session): give an SSH workspace one owning partition so its tabs stop round-tripping as deletions

`workspaceSessionPartitionHostId` answered differently depending on who asked: the
renderer mapped an SSH worktree's session to the `local` blob, the main-process
runtime read-modify-wrote `ssh:<targetId>`. One workspace's session lived in two
stores and no reader reunited them, so whatever landed on the unread side did not
read as unknown — it round-tripped as absence. The remote-workspace upload is a
`replace-session` patch, which turned that absence into deletion on the host, and
the next pull applied the deletion locally and re-poisoned the snapshot.

Collapse the two answers into one: every non-'local' host owns its partition.
Boot hydration and the export fallback now read the SSH partition, and rows a
shipping build left in `local` are folded back in once, gap-filling only — an
empty tab row is a gap, never proof that anything was closed.

Folder workspaces deliberately keep their existing 'local' routing: boot
discovers SSH partitions from the repo catalog, so an SSH target that owns only a
folder workspace has no partition any reader enumerates. They are still adopted
back out of an SSH partition when a repo does name the host.

Fixes #12721
Supersedes #12722

Co-authored-by: Robert Nisipeanu <github@nisipeanu.com>
Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com>

* test(session): pin the old-client empty-publish skew direction

* fix(session): adopt every workspace the host partition names, not only tabbed ones

Review caught that gating adoption on `host.tabsByWorktree[key].length > 0` traded
the #12721 deletion for a narrower one. The write path routes EVERY worktree-scoped
field to the owning partition, so an SSH workspace with open editor files or browser
tabs and no terminals had all of it dropped on every restart — and unlike terminal
state it cannot be recovered from the host snapshot, which carries terminal fields
only, so an unsaved `dirtyDraftContent` was destroyed outright.

The defect was not a missing field. It was a hand-maintained field list deciding what
the read recovers while the write used the ownership table, so the two could disagree.
Adoption now walks `WORKSPACE_SESSION_FIELD_OWNERSHIP` with an exhaustive switch, and
a new ownership kind is a compile-time decision rather than a silent omission.

Session keys are normalized through the shared `normalizeWorkspaceSessionKeyToWorkspaceId`
so host-qualified visit recency (`ssh:target|worktreeId`) reaches its workspace, and the
regression is pinned by feeding the shipping split's own output back through the real
boot read rather than a hand-built fixture.

Co-authored-by: Robert Nisipeanu <github@nisipeanu.com>
Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com>

* fix(session): stop adoption overwriting rows it was never told about

Three losses, one cause: the reader walks its own description of the
partition layout while the writer walks another, so the two agree on
which ownership kinds exist and not on what a kind means.

- An empty host row replaced a populated base row, destroying an unsaved
  dirtyDraftContent the header comment says must never be destroyed. The
  host holding nothing is not evidence the base is wrong.
- A contested bare id was adopted as if local and ssh:<target> were one
  workspace written twice, which is exactly the id where that premise is
  false. The read already reached that verdict and adoption could not ask
  for it, so it is passed in; contested keys are gap-filled, never
  replaced. mergeWorkspaceSessionsWithHostShadow now reports the real
  contested set, which primaryHostBySessionKey never was.
- Tab-, pane- and file-keyed rows are adopted through the split's own
  indexes, so unified-only tabs come back and the pane key is parsed once.
- A bare lastVisitedAtByWorktreeId key only fills a gap; the split has a
  dedicated branch for that field and the reader had none.

* test(session): pin the tombstone/gap boundary the two readings meet at

An explicit empty tabsByWorktree row means the user closed the last
terminal; adoption reads an empty base row as a gap to fill. Same value,
opposite readings, so the boundary is asserted rather than argued: the
tombstone lands in the owning partition, restores as a present empty row
rather than a deleted key, is declined by the real seeding predicate, is
published as an empty list, and the legacy-transition resurrection
happens once and cannot recur.

* docs(reliability): record the adoption guards and the tombstone boundary in the gate

* test(e2e): read the SSH restart assertions from the partition that owns them

ssh-cold-activation-restore asserted persistence through session.get()
with no host, which is the local partition an SSH worktree's rows no
longer live in. The invariant it means to check is that the state is
persisted where the boot read will find it, so it now unions local and
ssh:<targetId> and stays correct on both layouts.

Confirmed the product invariant separately rather than by the edit: the
behavioural half of both tests - the full app restart, the active
worktree, the eager terminal remount and the PTY-owner reclaim against a
real Docker OpenSSH host - runs after this check and passes. 2 passed in
48.9s.

* test(e2e): read ssh-restart-tab-accumulation from the owning partition too

Same layout-coupled read as ssh-cold-activation-restore: the pre-quit
flush asserted through session.get() with no host. Verified against a
real Docker OpenSSH target - both repeated quit/relaunch cycles keep
exactly the restored SSH tabs, no accumulation and no loss. 2 passed in
52.9s.

* fix(lint): clear the casting gate on the partition adoption

main tightened typescript/consistent-type-assertions to assertionStyle:
never, which the rebase brings onto these added lines. Most of the
round-trip fixtures did not need a cast at all -- three were hiding
wrong-shaped literals (a browser workspace keyed 'name', a unified tab
keyed 'type', a layout keyed 'direction'), now written as the types they
stand for. The adoption reads narrow through an isRecord predicate
instead of casting, which also stops a null entry throwing out of
Object.keys. What is left is dynamic-field writes and unknown-typed IPC
returns, each with its own SAFETY rationale.

* fix(session): give an SSH folder workspace one owning partition boot can find

The partition owner rule already names `ssh:<targetId>` for a repo-backed worktree, but
`getFolderWorkspacePartitionHostId` still answered 'local' for a folder workspace while
main's `RuntimeWorkspaceSessionController.getPreferredHostId` answered `ssh:<targetId>`
for the same key. That is #12723 unfixed for folder workspaces, and once the renderer
started writing `ssh:*` at all it got worse: a save's field-level patch carries only the
rows routed to that partition, so a `tabsByWorktree` write without the folder row erased
the row main had put there.

The reason the renderer could not route there was real - boot discovered SSH partitions
from the repo catalog, which cannot name a target whose only workspace is a folder. So
persistence now answers that directly over `session:list-host-ids`, and boot reads the
partitions that exist rather than the ones a catalog implies. Removing a folder workspace
prunes its rows from the owning partition too, or the census would adopt them back on the
next launch as a workspace the user already deleted.

Adoption now decides from the repo catalog instead of from co-presence. Two partitions
holding one bare `repoId::path` is not evidence of a collision - that is the exact shape
the repair exists for - so the verdict comes from `resolveWorktreeExecutionHost`: a repo id
registered on more than one host is contested and may only be gap-filled, and one the
catalog positively resolves to a different host is residue this partition does not own and
is not adopted at all. Without the second rule a stale partition sorting first won the read
and was then written into the live one. Nothing is deleted either way; the rows stay where
they are.

Finally, a workspace adopted out of a partition now routes back to that partition. Routing
used to re-derive an owner from the catalog, so a boot whose repos had not hydrated moved
the rows it had just reunited back into 'local' and re-stranded them. Contested ids are
withheld from that override, because routing the whole bare id to one host is the loss the
gap-fill prevents.

The publish path resolves each workspace's owner once for the whole publish, shared with the
projection, so the per-target catalog attribution does not repeat it per connected host.

* fix(session): drop a deleted workspace from every partition, not just the local blob

Adversarial review of the previous commit found three ways the partition census - which now
reads whatever persistence holds rather than what the repo catalog implies - keeps rows alive
that nothing should keep alive.

`deleteProjectGroup` pruned only the local blob, so every folder workspace under a deleted
group left its rows in `ssh:<targetId>`; the next boot adopted them back, named that partition
their owner and wrote them there again, forever. `removeFolderWorkspace` had the same hole for
a workspace whose partition its host expression could not name: main never persists a folder
workspace's `executionHostId`, and `RuntimeWorkspaceSessionController` can infer a connection
from the group's repos that the workspace row itself does not carry. Deriving the partition at
delete time is the wrong question - a deleted workspace owns nothing anywhere - so both paths
now remove it from every partition.

The third is on the read side. A contested id is deliberately withheld from the read-source
override so the write cannot carry one host's rows into another's partition, but the routing
that then re-derives an owner answers 'local' for an id the catalog cannot name. Adopting such
a row moved it out of the partition that owns it and into the blob: the two-store split this
change exists to remove. A contested id the assembled session holds no row for is therefore not
adopted at all. Gap-filling stays available for a contested id the session already names, since
that row's own partition is what the write follows. Declining to adopt leaves a row invisible
for one boot; it never deletes one.

Also: the folder-key guard in both catalog attributions was dead, because
`getRepoIdFromWorktreeId` hands back the whole key rather than nothing when there is no `::`.
The verdict was right and the resolution wasted; it now skips by shape. And the two type
assertions the changed-code casting gate rejected are gone rather than suppressed.

* fix(session): park the rows a partition read declines instead of letting the next write erase them

A partition write replaces each field with exactly what the unified session routed there. So a row
the read left out of that session is erased from its own partition the moment any sibling workspace
writes the same one - and with SSH partitions now the owning store, that row is then in no partition
at all. Three separate decisions produce such rows: residue the catalog attributes to another host,
a contested id withheld so the write cannot carry one host's rows into another's partition, and a
workspace the base already holds the live copy of. Declining to show a row was quietly deleting it.

The machinery for this already exists. `attachHostSessionShadow` writes a contested runtime
co-claimant's parked rows straight back into its own slice before the write, so the primary's write
cannot erase them; the ssh partitions simply were not among the slices the contention split
arbitrates. The read now parks everything it is not returning to an ssh partition into that same
shadow, and the existing re-attach puts it back. Leak, never kill - docs/reference/ssh-execution-
boundary.md - and a row no partition holds is unrecoverable.

Second, the contested branch of the tab adoption read `Object.hasOwn` as "the base has tabs here".
An empty list satisfies it, so whenever a legacy id happened to be contested, #12721's empty local
row won over the host's real one - the exact reading the module's own header, and the gate invariant
it is pinned by, say is wrong. An empty row is the gap this repair fills, so it is now treated as
one.

* test(session): pin the empty-base-row gap for a contested id

Mutation testing found the assertion missing: reverting the gate to `Object.hasOwn` left all 39
assertions passing, which makes the fix that reads an empty base tab row as a gap unguarded. The
#12721 shape does not stop being a gap because the id happens to be contested.

---------

Co-authored-by: Robert Nisipeanu <github@nisipeanu.com>
Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com>
2026-09-16 22:24:33 -07:00
Brennan BensonandMerge Sim 8999a00281 refactor(native-chat): give each structured dispatch state exactly one meaning (#20133)
* refactor(native-chat): give each structured dispatch state exactly one meaning

`unknown` meant five different things. Only one of them was genuine
ambiguity.

A transport write that the provider's input pump never took is provably
undelivered -- which is what `rejected` already means. It was recorded as
`unknown` anyway, and a one-entry allowlist then existed solely to teach
Retry that this particular `unknown` was safe to re-deliver.

Collapsing that case into `rejected` deletes the allowlist and turns a
predicate into an invariant: Retry never re-delivers an `unknown`, with no
exception to reason about. The four states now each assert one thing --
`pending` written and awaiting, `accepted` the provider has it, `rejected`
provably did not happen, `unknown` genuinely cannot tell.

A fail-closed guard is the right default here because the asymmetry is
severe: refusing a legitimate retry costs the user a retype, while allowing
an illegitimate one sends the model a second copy of their message.

Also fixed, found while auditing every reader of `rejected`:

- The renderer printed `submission.reason` verbatim, so a broken pipe put
  the internal token `provider_write_failed: broken pipe` on screen in
  destructive red. The journal reason is unchanged -- it is the durable
  evidence and the transport-versus-content discriminator -- but the screen
  now gets copy that names the cause and says the message is safe to
  resend. Content rejections still show the provider's own words.
- The fallback copy "Message was not accepted" read as a content refusal.
  A null reason now yields "Message was not sent.", which asserts only what
  every rejection shares.
- A refused worker-start preamble threw a plain Error out of the dispatch
  path. It now throws `OrchestrationError('dispatch_preamble_undelivered')`
  so a coordinator can tell "we could not send it" from "we sent it and
  something else broke" without parsing prose. Retain/discard behaviour is
  unchanged; only the verdict's legibility improves.

Two behaviours improve as a consequence rather than by design: a provably
undelivered message no longer blocks conversation commands, and no longer
leaves the session reading as "working" in chat and in every session list.

Not addressed here, and named rather than implied: a message left `unknown`
by a dead child or a host restart still has no recourse but retyping. The
restart reconciler that would decide those on evidence is written and has
never had a production caller. Parking the refused entry instead would
reintroduce the head-of-queue wedge removed in #19863, so it is not an
option.

Note for whoever edits `journal-reducer.ts` next: it sits at 297 of its 300
counted lines. The next statement added there needs a split, not a shave.

* fix(native-chat): close two gaps review found in the rejection taxonomy

Both are narrow and both were real.

A journal written before a refused write became `rejected` still holds that
submission as `unknown` with the transport marker. The predicate this change
replaced excluded exactly that shape from provider-echo matching; the
state-only check that replaced it does not, so on replay such a row could
claim the echo of a later, genuinely delivered send of the same text and
attach the delivery to the wrong message. Fail-closed still prevented any
re-delivery, so nothing duplicated — but the wrong submission was credited.
Replay now excludes the legacy shape too.

And the content-versus-transport split had a third case neither side covers:
a local capacity refusal is neither the provider explaining itself nor a
frame that failed to leave. It fell through to the verbatim branch, so
`claude structured dispatch queue is full` reached the screen — the same
class of leak this change set out to fix, one reason short of being caught.
Internal reasons now get copy; only a provider's own words are shown as
written.

Each is pinned by a test that fails with its guard reverted and passes with
it restored.

* fix(native-chat): preserve dispatch refusal across clients

* fix(native-chat): rotate immediately rejected retries

* docs(native-chat): correct rejection taxonomy reference

* docs(native-chat): align mobile retry comment

* docs(native-chat): clarify unknown replay semantics

* fix(native-chat): keep a mobile send's operation id when delivery is unknown

Mobile released the retained operation id whenever a send came back
`unknown`, so the user's next send of the same text went out under a fresh
id. A fresh id has no ledger row, so the host treats it as a first delivery
and dispatches it -- even though `unknown` is the one answer that says the
provider may already have the message. That is the duplicate this branch
exists to remove, reintroduced on the client that has no outbox.

Which case that was matters. Mobile only ever sees `unknown` from ack-loss
(`isRpcDeliveryUnknown`: "the host may have processed it and only the ack
was lost"), because the mapper reported every `ok` result as `accepted`
without reading `dispatchState`. So the rotation fired exclusively where
delivery was ambiguous and never where it was provably refused, which is
the inverse of the rule this branch establishes.

Retaining the id is what makes a retry safe, and it costs no liveness:
`performSend` answers a second request under a recorded id from the journal
and never puts it back on the wire, so a reused id delivers when nothing
landed and replays when something did. Rotating can only ever add a second
copy. The retention stays bounded by the host's admission window, which
`retainStructuredSessionOperationId` already enforces.

`retryUnknown` goes with it: the host ignores it for delivery, and all it
does is skip the cached answer to re-read the same row.

Keeping the id exposes what the rotation was hiding, so fix that too: a
replayed `unknown` comes back `ok`, and mobile called it `accepted` and
cleared the composer as if the message had landed. `dispatchState` now
decides, in one pure function:

  accepted/pending  sent, and the id is spent
  rejected          provably did not happen and terminal in the reducer, so
                    reusing the id could only replay that rejection: spent,
                    and the next attempt is a first delivery under a new id
  unknown           keeps its id

Reading `dispatchState` at all is a pre-existing defect, fixed here because
the false "sent" cannot be removed without it, and scoped to the send path.
`mutate`'s rotation for prompt/option/cancel plans is untouched. The
rejection copy is the desktop's notice, so an internal reason
(`provider_write_failed: ...`) still never reaches a person.

Tests: the hook test that was flipped to assert a rotated id now pins the
opposite -- one id across an ack-loss and two `unknown` replays, each
reported `unknown` rather than `accepted`. The send fixture grew the durable
submission row a real host returns; without it every send test asserted
against a shape that cannot express the bug.

* fix(native-chat): enforce fail-closed structured send replay

* fix(native-chat): align retry and mobile RPC contracts

* fix(native-chat): keep transient admissions retryable

* test(tab-bar): expand nested create menu in harness

---------

Co-authored-by: Merge Sim <sim@local>
2026-09-13 13:46:57 -07:00
Neil fee47fdb09 fix(chat): bound journal replay memory by live history (#20247)
* fix(chat): stream journal replay without retaining obsolete revisions

* fix: page journal replay reads so no SQLite snapshot outlives its statement

- iterateJournalEpochRows fetches one completed LIMIT statement per page
  instead of a lazily consumed .iterate() cursor, so reduction never runs
  inside an open read snapshot and a WAL checkpoint can pass mid-replay.
  Regression test: a checkpoint issued from inside the reducer is not busy.
- The retention test now asserts the applyJournalRow spy observed every
  row, so the 8 MiB bound cannot pass vacuously if the spy stops
  intercepting.
- Reliability gate manifest records the new assertion and the paged
  read design.
2026-09-12 19:39:30 -07:00
Neil 3666c12414 perf(daemon): reuse encoded unsplit NDJSON frames (#20298) 2026-09-12 18:08:42 -07:00
Jinwoo Hong 729491597f feat(desktop): measure relay regions and reconnect after idle cutover (#20106) 2026-09-11 11:43:48 -07:00
Jinwoo HongandOmar Shahine 3b82d8de64 fix(runtime): let connections own host status recovery (#20003)
* fix(runtime): let connections own host status recovery

Verify runtime status after authenticated connection recovery and publish
ordered snapshots to desktop and browser viewers. Consolidate failed-status
retries in the connection owner and remove renderer retry/diagnostics merging.

Adapt sidebar host-state derivation and regression coverage from Omar
Shahine's original fix in https://github.com/stablyai/orca/pull/19163.

Co-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com>

* fix(runtime): show blocked hosts honestly and remove obsolete status options

* fix(runtime): preserve timeout guidance and update IPC test fixtures

* fix(runtime): preserve status evidence and address review gaps

* test(sidebar): assert workspace host icons dimming and recovery tooltips

* fix(palette): require available hosts before adding implicit badges

* fix: retain disconnected host snapshots for new renderers

---------

Co-authored-by: Omar Shahine <10343873+omarshahine@users.noreply.github.com>
2026-09-11 03:19:01 -04:00
Jinwoo Hong 74cc9b5039 feat(desktop): native mobile push integration (2/3) (#19935)
* feat(desktop): integrate native mobile push delivery and lifecycle

* fix(desktop): preserve notification replay policy and review invariants

* fix(desktop): correct notification locale namespace and auto-ack tests
2026-09-10 20:16:56 -04:00
Brennan BensonandMerge Sim 2626e2eca4 Make the structured turn lifecycle row durable so completed durations survive (#19695)
* Make the structured turn lifecycle row durable so completed durations survive

A structured-chat turn used to end by tombstoning its running lifecycle item,
which threw away the only durable record of when the turn ended. Completed
"Worked for" labels therefore depended on the renderer having observed the
turn finish, and vanished on reopen.

The lifecycle item is now revised in place, never tombstoned:
- running, with startedAt, at the provider's turn start
- completed or interrupted, with completedAt, at the provider's terminal frame,
  a user stop, or a child exit the host observed
- unverifiable, with no end, when a cold acquire finds a running row from a
  generation whose exit nobody observed

Both timestamps are the execution host's clock at receipt, captured before the
deferred sink, so the completed value is identical on every client and needs
no client clock. Codex history restore uses the provider's own second-granular
endpoints for turns that predate this change. Desktop and mobile read settled
durations off the journal through one shared selector, and anchor the live
counter on the host start with the client's local receipt so a skewed client
clock never leaks into the label. Locally observed durations remain the
fallback for hosts that still tombstone.

Timestamps live inside the existing turnLifecycle field, which old clients
strip, and every working-state consumer keys on state === 'running', so no
capability negotiation is needed.

* native-chat: avoid stale working status on settled turns

* test: align settled turn status expectations

* Name settled lifecycle rows by their terminal state

An interrupted or unverifiable turn must not read as completed for any
consumer that renders status text raw. One shared helper builds the text for
both providers from the lifecycle state.

* test: deduplicate turn lifecycle suites

Each behavior keeps one test; duplicated harnesses and restated cases go.

* Key lifecycle rows to their user item and record the provider's measured duration

A lifecycle row now names the user item that opened the turn by its provider
key, so clients attribute timing explicitly and fall back to journal order
only for rows from older hosts. A provider-initiated turn with no prompt can
no longer claim the previous prompt's duration.

When the provider measures the turn itself (Codex turn.durationMs, Claude
result.duration_ms) the terminal row records it and clients prefer it over the
host interval, so a turn shows the same number live and after a history
restore. Host receipt times remain the live-counter anchor and the fallback.

* Record a turn as a first-class journal item

The turn record is now its own item kind rather than a status row carrying a
lifecycle field: no text to misuse, and the fold matches the durable turn
record other systems keep. Rows that carry it are stamped journal schema v3;
every other row stays v2, so an older host keeps reading them and latches
read-only at the first v3 row instead of truncating the epoch.

Clients that predate the item would paint an unknown kind as a text bubble,
so the host publishes the legacy status form to any client that does not
advertise agent-session.turn-item.v1, through the same per-client seam
background tasks use. The downgrade is transitional and goes once no
supported release lacks the capability. The shared projection now renders
unknown item kinds as nothing, so later kinds need no gate. One shared reader
handles both forms for old journals and old hosts.

* Preserve observed turn end across settlement retries

* Retain turn attribution for loaded chat history

* Preserve Codex exit receipt across close retries

* Register completed turn duration reliability gate

* Keep earlier turns through a Codex rewind and count a mid-turn attach from the real start

Findings from an independent adversarial review of the typed turn record:

- A Codex rewind adopted the provider's item list as the new epoch, and the
  provider never returns the host's own turn rows, so every duration before
  the rewind point vanished. The host's turn rows are now spliced back beside
  the item each followed, and recovery no longer expects the provider to
  prove rows it never owned.
- The epoch row was stamped with the current schema version, so an older host
  latched read-only at row 1 of every new session, defeating the mixed
  version design. It carries no body and stays at v2; a stored-row test now
  reads SQLite directly, because the reader upcasts every row on read.
- A send Codex folds into a running turn shares the opening prompt's provider
  key, and the alias map credited the duration to the later prompt. The
  earliest submission naming a key now wins.
- The live counter anchored on first sight, so a client attaching mid-turn
  counted from zero. Published frames now carry the host's clock, the reducer
  keeps the last sample with its local receipt time, and both clients anchor
  on how long the host says the turn has run.

* Correct turn duration gate assertion reference

* Respect authoritative unknown native chat duration

* Preserve unverifiable timing across older host upgrade

* Record final completed turn duration reliability evidence

* Fix the CI failures the merge left behind

- A merged import list named the same module twice, which the native code
  quality plugin fails on.
- A running turn is now reported by the host with no duration, so the settled
  map carries an explicit null for it; the hook test still expected the entry
  to be absent.
- main gave the older-page action a cursor with a head-trim guard, so the
  retention test's epoch-only action no longer typechecks; it now passes an
  unbounded sequence, which is what the old shape meant.
- The roster comparator moved into the extracted module, leaving its import
  unused in the reducer.

* Split two files back under the line cap after the merge

Merging main put both one effective line over 300, and the cap forbids a
disable or a shave. The wire module's refusal vocabulary moves to its own file
and is re-exported, so its consumers are untouched; the host's four thin
mutation delegates move next to the functions they call.

* Advertise the turn-item capability on every client transport

Local IPC and mobile advertised it; the remote and web transports did not, so a
desktop paired to a remote host, the CLI, and web silently ran on the legacy
carrier forever and the canonical row was never exercised there. The renderer
that paints it is the same build on every transport.

* Update the web auth-frame expectation for the new capability

---------

Co-authored-by: Merge Sim <sim@local>
2026-09-10 14:32:50 -07:00
Neil 373a670aba perf(terminal): skip ordinary output in preview control scans (#19400)
* perf(terminal): skip ordinary output in preview control scans

* test(terminal): cover scan resumption after parsed controls
2026-09-09 03:49:14 -07:00
Neil 1e2ed20b35 perf(terminal): release oversized backing strings behind pending controls (#19396)
* perf(terminal): release oversized backing strings behind pending controls

* test(terminal): record reproducible pending-storage gate evidence

* perf(terminal): own retained control fragments with a fast copy primitive

The pending-control ownership landed with a charCodeAt block copier (10 us
at 4 Ki, 170 us at 64 Ki), so it needed a "copy only when discarded output
dominates the tail" gate to stay affordable. That gate was the whole cost
problem: on adversarial streams it fires every chunk and pays the slow copy
(+45% on 16 Ki ANSI chunks, +81..111% on 194 Ki status chunks), and it also
skipped ownership on fragments too small to be sliced strings anyway.

ownRetainedString replaces it with a Buffer utf16le round trip (0.57 us at
4 Ki, 21.9 us at 64 Ki) and returns anything below V8's SlicedString
kMinLength unchanged. Buffer is absent in the renderer and on mobile, so the
copier is resolved once behind a lone-surrogate round-trip self-check and
falls back to the block copier. With a ~1 us copy the gate is unnecessary:
ownership is now unconditional at all three retention sites and the
adversarial cases land within noise of the un-owned parsers.

The three forced-GC threshold fixtures are replaced by one forced-GC test
for the primitive plus deterministic spy assertions that each site routes
its retained value through ownRetainedString. All fidelity and differential
coverage is kept.

* fix(terminal): escape the NUL in the round-trip probe

A raw NUL byte in the source made git treat the file as binary, so its
diffs and blame were unreadable. Escapes are equivalent at runtime.
2026-09-09 03:40:03 -07:00
Neil 042cc5266c perf(terminal): skip plain text between partial escape sequences (#19393)
* perf(terminal): skip plain text between partial escape sequences

* perf(terminal): take the ESC at hand before searching for one

The unconditional ground-state indexOf regressed dense back-to-back
SGR/CSI streams, where the code unit at the cursor is already the ESC and
the search pays call plus SIMD setup to find it in place. Check the
current unit first and fall back to the native search otherwise.

0.9 MiB dense SGR/CSI medians: 2.04 ms before this PR, 2.56 ms with the
unconditional search, 1.92 ms with the hybrid. Sparse colored logs and
plain text keep the full search win (0.36 / 0.016 ms vs 1.54 / 1.41 ms
baseline). Differential over the full VT alphabet with lone and split
surrogates matched 388,416 cases against both prior implementations with
zero mismatches; the ground-scan work budget now records 16 inspected
code units and 2 native searches.
2026-09-09 03:31:17 -07:00
Neil 23f38ddf7f perf(workspaces): skip unchanged heartbeat subscriber work (#19392)
* perf(workspaces): reuse unchanged heartbeat status projections

* perf(workspaces): skip unchanged title-sync session collections

* test(workspaces): cover constant-size key swaps in status projection reuse

Also point the title-input gate at a tracked source file; the previous
motivatingLink referenced an untracked .agents skill path.
2026-09-09 03:19:55 -07:00
Neil db66ab4ac8 perf(runtime): skip hibernation inventories without completed agents (#19391)
* perf(runtime): skip hibernation inventories without completed agents

* perf(runtime): skip hibernation status scan when no runtime owners

* fix(runtime): require host evidence for workspaces resolved mid-inventory

`runtimeLivenessRequiredWorktreeIds` was sampled before the runtime inventory
await, while the plan is built from the state after it. A workspace that gained
tabs or resolved its runtime owner during that window was therefore absent from
the required set, so the planner did not demand fresh host evidence for it and
fell back to client PTYs — client bookkeeping answering for the execution host.

Union the post-await targets into the required set inside `snapshotFromState`.
Union rather than replace: the set only ever grows, so the planner can only skip
more workspaces, never authorize a hibernation it would previously have refused.
An absent inventory stays a skip; nothing reads it as an exited PTY.

Extract the coordinator test fixtures so the regression lives in its own file
without pushing the coordinator suite past the 800-line test cap.

* fix(types): annotate hibernation fixture mock exports for declaration emit

TS2883: the inferred `Mock<Procedure>` types of the fixture's exported
`vi.fn()` bindings reference `Procedure` from a transitive `@vitest/spy`
path that cannot be named.

* chore: keep local-file-sink-memory test formatting as on main

The merge commit's pre-commit hook reformatted a file this branch does not own.
2026-09-09 03:17:20 -07:00
Brennan BensonandMerge Sim 3a801d213d fix(orchestration): worker-start settles readiness on observed turn start, not write acceptance (#19423)
* fix(orchestration): worker-start settles readiness on observed turn start, not write acceptance

A dispatched PTY worker whose agent wedged at startup (six codex workers on
2026-09-07) was reported 'ok: true, state: ready, stage: input_accepted': the
preamble write was acknowledged with observationTimeoutMs: 0 and nothing ever
verified a turn began. The corpse and the healthy worker produced identical
receipts.

worker-start now runs the existing second-stage prompt observer
(observeTerminalAgentPrompt) after acceptance, inside the 30s window the
client RPC grace already budgets for (orchestration-worker-start-prompt-budget):

- turn observed (or provider ack for structured sessions) -> ready
- permission prompt -> ready; positive liveness, surfaced in the receipt
- provider without a turn-start signal -> ready; observation: unsupported
- observation supported and nothing started -> worker state start_unknown,
  response state outcome_unknown with nextCommands. Honest 'unverifiable',
  never a death claim: the capability and terminal are kept, and
  worker-report settlement already reconnects a start_unknown worker that
  recovers and reports.

Also fixes the effect-verb lie that misdirected the first diagnosis of this
incident: agent-first worktree creation labeled its own brand-new agent
terminal 'reused_agent_terminal' (a role test picking a lifecycle verb) on
both the local and federation paths. It now says 'created'; readers keep
accepting the retired verb for rows persisted before the rename.

* fix: preserve worker authority through start observation

---------

Co-authored-by: Merge Sim <sim@local>
2026-09-08 22:53:48 -07:00
Neil c3a415487b perf(terminal): avoid quadratic status-marker scans in output bursts (#19373)
* perf(terminal): reuse forward OSC status terminator searches

* test(terminal): pin cached OSC terminator reuse across BEL frames

Document that forward match reuse requires a monotonic search offset and cover a distant ST held across many intervening BEL frames.
2026-09-08 19:41:42 -07:00
Jinwoo Hong 8f78c28248 fix(orchestration): fence worker release on mobile keystrokes (#19337)
* fix(orchestration): fence worker release on mobile keystrokes

A settled worker's terminal stayed ownership_state='owned' unless a takeover was
recorded, and the only recorder was orchestration.workerTerminalUserInput, which
only the desktop/web xterm input signal and the native-chat composer call. Mobile
input arrives as terminal.send / stream input frames instead of a report, so a
phone user typing in a settled worker's pane never fenced anything: worker-list
kept recommending release and worker-release closed the PTY under them.

Give the host one definition of "a human typed into this terminal" and route every
lane through it. The mobile input floor claim is that definition and already exists
on both byte lanes: it is taken only for deliberate phone input, never for the
emulator's own query replies, and never for an agent's `orca terminal send`, which
names itself a desktop client and so is indistinguishable from a keystroke at this
layer. Settling that claim after an accepted write now records the takeover through
the same code the RPC reporter uses, throttled to one write per pane per 30s so a
keystroke does not pay for an immediate transaction. The record lands on the runtime
that owns both the terminal and the orchestration database, so SSH-hosted and remote
workers behave exactly like local ones.

No mobile change: mobile already sends client.type (mobile/src/terminal/terminal-send-request.ts:24).

* fix(orchestration): ask the database, do not remember, whether a pane is fenced

The keystroke throttle armed on the attempt rather than on the outcome, so a
zero-row or thrown record poisoned the pane for 30s. A phone keystroke during
the worker-start readiness wait lands before prepareStartingWorkerAuthority
creates the owned resource; a real keystroke seconds later was then suppressed,
the worker settled, and workerRelease closed the terminal under the phone user.
A SQLITE_BUSY on the first write did the same, with no retry.

The cache was the defect, not its arming condition. Its precondition is the set
of owned resources on the pane, which changes underneath it, and any cache keyed
on ownership identity would have to read the database to learn that identity --
which is the whole question. So the input lane now asks: a read using the same
predicate the write uses answers "is anything still fenceable here?" without
taking BEGIN IMMEDIATE, and only then is the write attempted. Ordinary typing
costs a lookup instead of a write lock, a failed write is retried by the next
keystroke, and a takeover writes once per ownership epoch rather than once per
window, because the flip to user_owned removes the pane from the candidate set.
Sharing the predicate keeps the probe from drifting from the writer.

Adds the two escape cases as permanent regressions, drives the mocked send
through the real RuntimeTerminalWriter, and asserts a mobile takeover lifts the
settled-worker resume fence, which no test covered.

* refactor(orchestration): let the database dedupe the takeover, drop the read probe

The probe was meant to keep keystrokes off BEGIN IMMEDIATE, so it had to earn
that with a number. Measured against a real WAL database it costs more than the
write it avoids: at 25 live workers the probe is 0.19ms and the no-op write is
0.10ms, because the probe runs the same candidate selection with each statement
taking its own read snapshot instead of sharing the transaction's. It is a
compensating mechanism with negative value, so it goes, along with the database
method and the predicate extraction it needed.

owned -> user_owned is one-way and scoped to a resource, so the database is
already the dedupe: every deliberate human write attempts the transition, the
second attempt matches no row, and the fence sweep runs only on changed > 0.
Nothing is remembered between keystrokes, so no state can outlive the ownership
it described -- a keystroke before the worker's authority attaches, a write the
database refuses, and a re-dispatch onto the same pane all resolve against the
rows as they are at that instant. An attempt costs about 0.1ms at typical fleet
size and 0.34ms at 100 live workers, on mobile writes only.

Replaces the write-count test, which asserted the old mechanism, with the
invariant: many keystrokes settle into one takeover and one fence sweep. Adds
the re-dispatch case, where a pane's next worker is fenced on its own merits.

* refactor(terminal): name the provenance rule the takeover fence hangs off

The fence rode the mobile input floor claim, with only a comment tying the two
together. The floor is arbitration -- who may write next -- while the fence needs
provenance -- who produced the bytes. They agree today, so anyone reweighing the
floor would have moved the fence without noticing.

isDeliberateHumanInput states the provenance rule on its own terms, and both byte
lanes decide with it when they open a write: the claim carries the verdict beside
the handle, and settlement records the takeover only when a human produced the
bytes. No behavior change -- afterWrite is wired only where the predicate already
answers true -- and the rule is now pinned by its own cases, so a future
arbitration change has to answer this question again rather than inherit it.

* test(orchestration): prove the unary lane classifies a metadata-less phone

A phone build older than client.type is recognised only by its pane's mobile
driver, which the unary lane passes as the provenance evidence. Nothing proved
it did: replacing that argument with false left all 17 tests green while a
shipped phone silently stopped fencing worker release. The new case drives a
clientless send on a mobile-driven pane and fails under that mutation.

The stream lane now passes false outright. Its isMobile is read off the same
client object it carries, so the metadata-less phone cannot reach it, and
passing the flag suggested a legacy path that does not exist there.

Also states what the per-keystroke cost scales with. A pane owning no resource
misses the pane_key index and falls through to a scan of owned resources, so the
figure is tens of microseconds at realistic worker counts rather than a flat
0.1ms, and it grows with rows that are never released.

* fix(terminal): let provenance alone decide the takeover, on every accepted write

A phone older than client.type sends no client metadata, and both stream
initializers derive isMobile from that metadata alone, so such a subscription
reported false and took the stream lane's uninstrumented branch: provenance was
computed and then never consumed. Bytes from a real person landed through both
frame adapters and the resource stayed owned, so workerRelease closed the PTY
under them. The unary lane already fenced that population off the pane's mobile
driver, which is the host's standing reading of clientless input, so the two byte
lanes disagreed at the destructive boundary.

The predicate was still subordinate to floor plumbing: it could only be consulted
where a floor client id existed. Now the accepted-write callback attaches on both
lanes regardless of whether a floor was reserved, and humanInput alone decides
recording; a write holding no claim commits nothing. Arbitration keeps its own
condition around reserveWrite, where it belongs, and the unary lane's duplicate
outer provenance filter is gone. The stream lane reads clientless provenance from
the pane's driver, the same policy the unary lane uses.

The claim holder is now TerminalInputWrite, carrying the verdict beside an
optional floorClaim, so the structure says what the doc said: a write may fence
without holding the floor.

Regressions drive both real frame adapters, clientless direct delivery, and the
paired-web desktop negative. Metadata-only provenance fails 3 on the stream lane
and 1 on the unary lane; gating the callback on a reservation fails the same 3.

* fix(runtime): resolve retained handles before mobile input provenance

A renderer reload clears transient handles while retaining runtime-owned
PTY identities. Legacy mobile provenance saw no leaf, then sendTerminal
restored the same handle and delivered an unfenced key. Normalize through
getLivePtyForHandle at the shared live-leaf resolver entry so classification
and writes agree, preserving existing leaf generation/incarnation checks.

Caller audit:
- terminal-send-method: driver, query-reply authority, lock and floor checks
  now resolve the retained PTY before sending.
- terminal-input-delivery: legacy mobile classification and exact-PTY
  binding now see the same target as the writer; equality checks remain.
- terminal-multiplex-subscribe-resolution: retained PTYs resolve directly
  without a spurious missing-terminal wait.
- terminal-lifecycle-methods resize and terminal-viewport-methods display
  mode, restore-fit and updateViewport retain their original PTY target.
- inspectTerminalProcess: avoids false terminal_gone after reload while
  preserving provider inspection and incarnation fences.
- getLivePaneKeyForTerminalHandle and getOrchestrationDispatchAuthority:
  unaffected because both already call getLivePtyForHandle first.
No wire/schema changes, host fallback, process-death inference, or Git
workspace assumptions; SSH providers keep ownership of execution evidence.

Validation:
- Unmodified round-3 reviewer probe: reproduced 2/2 failures, then 2/2 pass.
- Unmodified round-2 reviewer probes: 13/13 pass.
- Checked-in takeover suites: 24/24 pass. Removing only the resolver call
  fails both new reload cases; source restored afterward.
- RPC orchestration + terminal, aggregate runtime handle registry,
  handle incarnation, mobile tab mount, stale geometry, and reload probe:
  2027 passed, 1 skipped (89 files).
- tc:node and check:code-quality:changed pass; background launch enabled.

* test(rpc): require unconditional terminal afterWrite callbacks

Update exact sendTerminal expectations for the round-2 accepted-write
contract. Preserve beforeWrite expectations, absence of reserveWrite,
byte payloads and call-count checks; require afterWrite to be a function.

Reproduced the requested two-file run: 5 failed, 31 passed. The full RPC
suite exposed the same stale shape in ACK budget/overflow, desktop resize
(including its later retry), and agent-prompt fallback assertions. Update
those too, for 11 assertions across six test files. No production changes.

Validation: ORCA_BACKGROUND_LAUNCH=1 full src/main/runtime/rpc suite:
264 files passed; 2292 tests passed, 1 skipped. Changed-code quality and
staged oxlint/React Doctor/oxfmt checks passed. Ran lint-staged --no-stash
manually to honor checkout safety rather than its default backup hook.

* fix(mobile): report worker takeover outside terminal byte delivery

New phones announce accepted real user input through the existing worker
report RPC, addressed by terminal handle. Share a per-client/per-handle
30-second gate with one bounded retry; report through the same RPC client
as the input. Cover live commits and dictation via their shared sender,
accessory keys, gestures, buffered submit, paste and accepted native chat.
Query replies, attachment heals, triage and diff-review sends do not report.
Phones predating this build do not fence release.

Remove byte provenance and takeover callbacks from host delivery. Restore
both lanes' pre-PR floor-claim plumbing and the original options assertions.
Keep the host recorder uncached with its conditional resume-fence sweep.
No DB schema or stream change; terminal is an optional report address.

Retain the shared resolver recovery independently of takeover: the new
SSH inspection test fails without it during renderer reload. Other callers
still benefit for subscription, resize, viewport and exact-PTY binding;
unary driver/lock checks see the retained PTY. Pane routing and dispatch
authority already recover through getLivePtyForHandle and are unaffected.
Existing leaf generation checks and first-PTY adoption remain unchanged.
No other input-plumbing hunk is retained relative to the PR base.

Replace byte-takeover tests with handle-addressed local/SSH report and
unknown-handle tests, plus real unary/stream writes asserting zero SQL
prepare/exec calls. Mobile send-site integration covers reports, exclusions,
rejected writes and gate counts. Desktop report tests are unchanged.
Register replacement coverage in the settled-worker release manifest.

Validation (all background): host/RPC/runtime 3541 passed, 2 skipped;
mobile session/terminal 2045 passed; node and mobile typechecks, changed
quality, mobile oxlint, reliability manifest and max-lines ratchet passed.
All five requested mutations fail assertions; resolver revert also fails
independent inspection. Staged checks run manually with --no-stash.
Final src diff against PR base: 5 files, +165/-13 (previously +839/-85).

* fix(runtime): allow the takeover report from mobile-scoped tokens

The mobile RPC allow-list gates every phone request before dispatch and the
reporter swallows a refusal, so without this entry every phone shipped
unfenced. Pin it beside the report tests, and pin the once-per-takeover
fence sweep the replaced byte-lane suite used to assert.

* fix(mobile): a no-op takeover report does not arm the gate; Stop reports too

A key during worker startup reports before the resource is owned; caching
that zero-change reply for 30 s suppressed the report that would have fenced
the worker once it attached. Native-chat Stop is deliberate input and now
reports on an accepted Escape.

* fix(mobile): takeover gate ignores the host answer, like desktop

Reopening the gate on a zero-change reply made every accepted key on an
ordinary terminal an RPC plus a host write transaction (round 6: 100 for
100). The startup window it closed is unreachable: the agent has no prompt
to accept input until after its resource row exists. Plain terminals now
pay one report per 30 s window; the native-chat Stop report stays.

Send-site fixture answers the report RPC with a changed count; the draft
test filters to terminal.send calls.

* docs(runtime): say why resolveLiveLeafForHandle re-links before lookup

* chore(i18n): regenerate the runtime-required catalog for the contrast floor strings

* test(orchestration): give the stopping-worker guard fixtures a Run

* test(orchestration): drop fence-sweep assertions retired by the settled-worker policy

* test(orchestration): pin the mid-boot phone takeover that #19608 makes possible

A handle-addressed report during the worker's tui-idle wait now finds the
custody row written at terminal creation, so it flips the pane to user_owned
and worker-release retains it instead of closing it under the user.
2026-09-08 14:48:57 -04:00
Jinwoo Hong 98fdbc4ade fix(orchestration): file federated worker mail under the coordinator Run (#19542) 2026-09-08 04:03:07 -04:00
Neil ed881889c4 fix(terminal): fold-safe CAN/SUB and double-ESC handling in partial-escape tail (#19521)
`extractPartialEscapeTail` broke its own fold invariant
(extract(a + b) === extract(extract(a) + b)) in the oscEsc/stringEsc states,
so a PTY read that split there produced a different pending tail than the
same bytes delivered whole — the tail snapshots append after a restore.

Two causes, both in the "ESC did not terminate the string" branch:

- CAN/SUB were routed through `stateAfterEscByte`, which maps them back to
  `esc` instead of aborting to ground. `extractPartialEscapeTail('\x1bPx\x1b\x18X0abc')`
  returned '\x1b\x18X0abc'; the chunk-split fold returned ''.
- A second ESC opened its new sequence at `i - 1` rather than at itself.
  `extractPartialEscapeTail('\x1b] \x1b\x1b^')` returned '\x1b\x1b^' whole but
  '\x1b^' folded. The fold was right — xterm starts the sequence at the second ESC.

The existing fuzz only asserted the fold as `advance(extract(pending), chunk)`,
which is a tautology because every PENDINGS entry is already a tail. Replaced
with a sweep that re-splits the combined stream at every code-unit boundary,
and extended the alphabet (NUL, 0x20 intermediate, CJK) and SEQUENCES with
CAN/SUB and doubled-ESC-inside-string cases. A 1.25M-split fold fuzz over a
VT alphabet goes from 421 failures to 0.
2026-09-07 23:25:46 -07:00
Neil 28d936e030 perf(native-chat): bound journal reads during paged catch-up (#19360)
* perf(native-chat): bound journal reads during paged catch-up

* perf(native-chat): reduce the journal once per catch-up run, not per page

Bounding the SQL read per page left the JS side still O(total items) per
page: every page re-reduced the whole timeline and rebuilt the live-item
map, and the byte-shrink loop rebuilt it again on each halving.

A catch-up run is a synchronous loop with no await between pages, so the
reduced timeline is loop-invariant. `createAgentSessionCatchUpReader`
holds one snapshot for the run and re-reduces only if the journal cursor
actually moved, and the projection's live-item / alias / submission-byte
indexes memoize on the snapshot arrays the reducer rebuilds on change.

Per catch-up over a 8,000-message backlog: 40 timeline reductions to 1,
reduce+project time 24.3ms to 3.5ms, end-to-end 134.6ms to 111.3ms.
2026-09-07 23:24:25 -07:00
Jinjing 31b84cc71f Fix native PTY I/O failures disabling session termination (#19523)
* Fix native PTY I/O failures disabling session termination

I/O failures on write or resize were incorrectly treated as exit evidence,
which disabled further termination attempts and producer flow control.
Separate ioFailed state from dead state; I/O errors suppress operations
but keep kill, forceKill, and signal available. Publish physical exit
before notifying listeners to prevent reentrant cleanup attempts from
accessing the retired native process.

* Fix PTY I/O cleanup tests and config syntax error

- Fixed missing closing brace and comma in reliability-gates.jsonc
- Added clear() operation to PTY mock fixture and test coverage
- Enhanced assertions to verify operation suppression during I/O failures
- Improved kill operation error handling with better promise-chain assertions
- Updated test result summaries in reliability gate documentation
2026-09-07 22:20:35 -07:00
Neil e182930670 test: cover input in five simultaneously flooding SSH panes (#19071)
* test: cover keyboard input in five simultaneously flooding SSH panes

* test: capture pane focus and buffers on flood input failure

* test: capture pane focus and buffers on flood input failure

* test: capture pane focus and buffers on flood input failure

* test: record replay input loss and application fix dependency

* test: record merged replay-input fix in the five-pane flood gate
2026-09-07 19:51:44 -07:00
Jinjing 9fed61e5c2 Persist agents sidebar search visibility as pairing-local preference (#19313)
* Persist agents sidebar search field visibility as pairing-local preferen

- Add `agentsShowSearch` to workspace UI state with default on
- Include in pairing-local fields so preference syncs across clients
- Convert search from menu action to checkbox menu item for explicit toggle
- Update activity thread options menu to reflect checkbox state
- Add localization strings across all supported languages
- Update RPC schemas and preference persistence layer
- Includes readiness validation reports confirming feature is clean

* rm review

* fix documentation
2026-09-07 11:27:40 -07:00
Neil 2ccf35b135 fix: avoid quadratic trimming during fullscreen terminal redraws (#19214) 2026-09-06 23:41:49 -07:00
Neil 1e301ab1df test: cover native Wayland Hangul in isolated CI (#19174)
* test: exercise native Wayland Hangul in isolated CI session

* test: wait for nested compositor socket before selecting IBus

* test: align Wayland IBus discovery with GNOME environment filtering

* test: assert Wayland launch and register native Hangul evidence
2026-09-06 19:02:54 -07:00
Neil 2e8fa3fe9b test: exercise packaged browser compatibility in scheduled CI (#19157)
* test: exercise packaged browser compatibility in scheduled CI

* test: record final packaged workflow participation evidence

* test: expose manual packaged revision and simplify executable check

* test: reject missing package checksum assertion
2026-09-06 17:42:49 -07:00
Brennan BensonandMerge Sim 20eea184cc feat(native-chat): offer the link-action popover for chat links (#19130)
* feat(native-chat): offer the link-action popover for chat links

A plain click on an http(s) link in a native chat transcript opened the
system browser outright, ignoring the link-routing preference the same
link honors in the terminal. Chat now shows the terminal's destination
popover, with the modifier chords routing straight to a destination.

The popover, its request type, the destination policy and the routed open
move out of terminal-pane so both surfaces share one implementation; the
catalog keys keep their original namespace because they carry shipped
translations. Chat resolves its link owner from the session workspace
(runtime, then SSH, unresolved stays unknown) so a remote transcript only
offers Orca Browser when that host's managed browser route is eligible.

The existing toggle now governs both surfaces, so it is retitled; with it
off a chat link still opens on a plain click instead of going dead.

* Fix native chat link popover lifecycle and keyboard anchoring

* test(native-chat): use one store mock for link actions

* fix: update reliability gate for shared link popover tests

---------

Co-authored-by: Merge Sim <sim@local>
2026-09-06 15:52:50 -07:00
Jinwoo Hong 06a607a1d7 feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​12401 |

<!-- /orca-pr-loc -->

## ELI5

Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.

## What changed

- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.

## Why

User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.

## Linked issues

Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.

## Review record

This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.

**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:

- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.

Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).

A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.

A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.

Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).

## Testing

- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).

## Notes

- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
2026-09-06 14:34:03 -04:00
Neil 3be526c5e6 test: cover SSH reattach replay and enable deterministic Codex CI (#19106)
* test: cover SSH replay replies and run deterministic Codex restore scenarios

* test: register replay probe unit command in reliability gate
2026-09-06 11:25:58 -07:00
Neil 4d9e963ffd test: enable localhost SSH terminal and hook journey in CI (#19097)
* test: run localhost SSH terminal and hooks in CI

* test: isolate localhost SSH session fixtures across repetitions

* test: route remote agent hook source changes to localhost journey

* test: record localhost SSH reliability evidence and remaining gaps

* test: route the real SSH session hook authority
2026-09-06 10:05:26 -07:00
Neil f5960cec00 test: enable Docker SSH browser network route coverage in CI (#19095)
* test: enable Docker SSH browser network route journeys in CI

* test: register Docker browser job in token permissions contract

* test: declare SSH client dependency and narrow browser fixture routing
2026-09-06 09:29:56 -07:00
Neil 3631f886a7 test: enable direct and client-hosted SSH browser coverage (#19090)
* test: enable direct and client-hosted SSH browser coverage

* test: record twelve passing SSH browser journey repetitions

* test: distinguish SSH journey evidence from unit runtime budget
2026-09-06 08:25:13 -07:00
Neil 1d2e00819f test: restore SSH bulk-open freeze coverage in headed CI (#19081)
* test: restore SSH bulk-open freeze coverage in headed CI

* test: record ten passing headed SSH freeze repetitions

* test: record ten passing headed SSH freeze repetitions

* test: route changed SSH freeze spec only to its dedicated lane
2026-09-06 07:42:50 -07:00
Neil f952f1ac96 test: run real WSL terminal launch and paste in PR CI (#19072)
* test: continuously exercise real WSL terminal launch and paste

* test: establish live WSL reader before changing default shell

* ci: pin WSL kernel installer and participation selectors

* ci: route deleted WSL paths and record immutable run evidence

* test: require exactly three WSL repetitions in lane contract
2026-09-06 05:24:54 -07:00
Neil d7767fb196 perf(worktree): remove redundant creation and terminal startup work (#18793)
* perf(worktree): remove redundant creation and terminal startup work

* test(worktree): cover optimized creation call signatures

Preserve explicit branch adoption, WSL callback routing and sparse cleanup expectations.

* perf: preserve user Git checkout worker settings

* perf(git): skip malformed remote base probes

* perf(cli): avoid loading other agent hooks for Codex preflight

* fix(build): retain Codex preflight entry for packaged CLI

* test(ssh): wait for replacement PTY before lease recovery input

* test(ssh): verify recovered shell execution and lease ownership

* test(electron): reap isolated macOS crash reporters on teardown

* test: allow either observed self-exit snapshot ordering

* test: capture frozen-host input recovery evidence
2026-09-05 15:08:22 -07:00
Neil d8c4c83063 test(e2e): fence SSH recovery and release exited Electron pipes (#18880) 2026-09-05 13:53:27 -07:00
Jinwoo Hong 436ef827dd fix(browser): present Electron's own user agent so Cloudflare Turnstile clears (#18749)
Orca rewrote every browser session's UA to look like plain Chrome by stripping
the Electron and app tokens. That rewrite is what Cloudflare rejects: a Chrome
UA that ships no client hints reads as a spoof and Turnstile returns 600010,
while the same binary on the same IP clears every challenge with its stock UA.
PR #885 added the rewrite to fix 600010 and was treating a symptom it created;
issue #11518 later found the same rewrite is what broke Google sign-in.

- Keep the stock Electron UA on every partition. The webRequest handler now only
  owns the host-scoped Google auth Firefox switch, which stays unchanged.
- Delete the anti-detection script. Measured on Electron 43: plugins are already
  a real PluginArray, window.chrome exists, and navigator.webdriver is false even
  with the debugger attached, so three of its four premises were wrong, and the
  overrides it installed (instance-level webdriver, non-native Permissions.query,
  stubbed chrome.csi/loadTimes) are themselves published bot signatures.
- Stop attaching a CDP debugger to every browsing guest. Only the auth-UA detach
  listener remains, because a detach clears Chromium's standing UA override.
- Stop sending Runtime.enable into cross-origin iframes when the agent bridge
  auto-attaches. The challenge widget is one, nothing reads iframe Runtime
  events, and the Runtime domain's serialization side effect is the documented
  Cloudflare CDP tell.
- Add a real-Electron test proving the wire identity: stock UA to ordinary
  hosts, Firefox with no client hints to accounts.google.com.

Verified in the dev build: dash.cloudflare.com/login no longer shows
"There was a problem with verification" and scrapingcourse.com's managed
challenge clears, both failing deterministically before.

Fixes #13822
2026-09-04 23:47:15 -04:00
Jinwoo Hong 573537ecd4 feat(cli): make terminal close the canonical workspace teardown (#18073)
* fix(runtime): recover stale session owners and await retirement

* fix(runtime): preserve session hydration and smoke compatibility

* test(runtime): cover empty and unindexed session owners

* feat(cli): make terminal close the canonical workspace teardown

* fix(preload): align ssh termination result type

* test(runtime): assert folder hydration owner

* fix(runtime): fence legacy terminal stop by worktree host

* fix(preload): reconcile ssh result import with main

* fix(runtime): keep same-id sibling hosts out of workspace close

The stale-owner fallback in the session controller re-routed any worktree whose
catalog partition had no tabs to whichever other partition held tabs. Only
`runtime:` environment ids rotate across relay restarts; `repoId::path` legitimately
repeats across hosts, so an SSH workspace close could retire the local copy's
tabs and resume records, or flip owners mid-close and strand the SSH PTY.

Restrict the fallback to runtime hosts, and pin the session partition once per
workspace close so record clearing targets the partition that owned the tabs.

* test(runtime): give the cross-host close fixture a real resume record

* fix(preload): take main's ssh-bridge import order so the merge stays duplicate-free
2026-09-03 03:58:45 -04:00
Neil d4fa091714 perf(terminal): repaint only the rows an agent redraw touched (#18169)
Forced foreground repaints asked xterm for rows 0..rows-1. xterm's render
debouncer unions ranges, so one full-grid request widened every frame to a
whole-viewport `_updateModel` cell walk even when the write changed five rows.
Re-issue the repair over the parse's own dirty span instead, keeping the
whole grid for viewport scroll, alternate-screen flips, and any write whose
span cannot be observed.
2026-09-02 13:22:29 -07:00
Neil f37d2fec97 fix(linux): land the reviewed Linux packaging stack on main (#18100)
* fix(linux): give the CLI one entrypoint by extracting the AppImage once

* refactor(linux): trim AppImage CLI registration seams

* test(cli): assert registration lock serialization

* fix(linux): fence AppImage terminal shim mounts

* fix(linux): accept extracted AppImage runtimes with APPDIR only

* docs(linux): make headless AppImage extraction runnable

* refactor(linux): import bundled launcher directly

* fix(linux): reclaim superseded AppImage payloads and packaged symlinks

Pruning removed 3215 of 3216 files from a superseded generation and always
stranded resources/app.asar, leaking ~105 MB per version update. Electron's
asar shim reports a *.asar file as a directory, so the recursive remove tried
to rmdir a real file and failed with ENOTEMPTY; the .catch(() => {}) hid it.
Reproduced end to end on Ubuntu 24.04: 519M -> 623M across one update, and
519M again once the payload is actually reclaimed.

removeExtractedAppImagePayload holds process.noAsar for the removal, counted
so overlapping removals cannot hand the shim back early, and the prune site
now warns with the path instead of swallowing the rejection. All three
removal sites use it -- staging cleanup and displaced roots leaked the same
way.

Also reclaim symlinks left by a packaged deb/rpm install, which the
extracted-cache-only rule turned into a hard conflict on a deb -> AppImage
migration, and name the remedy in the conflict error.

* fix(linux): bound the CLI registration lock wait

`retries: 1000` caps the attempt count, not elapsed time, so at up to 1s per
attempt an IPC-driven registration could hang ~16 minutes against a wedged
holder with no feedback.

A legitimate holder is bounded by the extraction timeout, so wait that plus
slack and then fail with a message naming the lock file, rather than hanging.
`maxRetryTime` is forwarded verbatim to the `retry` package by proper-lockfile.

* fix(linux): stop re-extracting the AppImage on inode metadata churn

The extracted-payload cache key hashed ctime alongside dev/ino/size/mtime.
ctime moves on any inode metadata write -- `chmod +x`, which every AppImage
user is told to run, plus `chown`, an ACL or SELinux relabel, and a backup
restore -- none of which alter a byte of the payload.

Measured on Ubuntu 24.04: `chmod +x` leaves dev, ino, size and mtime
identical and moves ctime alone, so the key changed and the next launch paid
a full ~519 MB re-extraction and a multi-second stall to rebuild a payload it
already had, then pruned the old generation.

Key on content identity instead. An in-place content change moves mtime and
almost always size; a replacement moves the inode. The existing
replace-in-place test still passes.

* fix(linux): stop CLI commands from falling through to Chromium startup

* refactor(cli): remove redundant command membership check

* test(cli): cover command-named project selectors

* fix(cli): redirect the open-url command before startup

* test(linux): cover AUR serve wrapper flags

* fix(linux): tighten CLI launch detection

* fix(linux): respect CLI flag value boundaries

* fix(linux): strip injected Chromium switches from CLI args

* fix(linux): report a missing display instead of dying in uv_close

* refactor(linux): read display locks without a preflight race

* fix(linux): preserve unverified external displays

* chore: format reliability gate manifest

* test(packaging): split runtime resource checks

* fix(linux): fail serve when no display is available

* fix(linux): do not treat a lockless X socket as a dead display

An X server writes its lock beside its socket and both survive a crash
(verified against Xvfb under SIGKILL), so a socket with no lock was never
left by a crashed server. It is an endpoint published from elsewhere: a
container bind-mounting only /tmp/.X11-unix, WSLg, or a foreign PID
namespace. Declaring those dead made the desktop gate exit(1) on displays
that work, with no workaround, and the serve gate refuse to start.

Liveness now splits by ownership. A foreign DISPLAY trusts a lockless
socket; Orca's own :99 does not, because removeStaleDisplayArtifacts
unlinks the lock before the socket and so manufactures that state itself --
adopting it would resurrect the orphan-socket bug and stop the cleanup from
self-healing. The stale-lock rejection is unchanged.

Also correct four doc statements this behaviour falsified.

* fix(linux): fail closed when a stale socket blocks the Xvfb rebind

Readiness only checked that /tmp/.X11-unix/X99 exists. A stale socket we
could not unlink still exists after our own Xvfb refused to bind, so Orca set
DISPLAY to a dead server and Chromium died in Ozone init.

Measured on Ubuntu 24.04 against the pre-fix build: with a leftover :99
socket and no lock, serve exits 139 (SIGSEGV), the socket inode is unchanged
before and after, and no lock is recreated -- it neither cleaned up nor
respawned. To a user that is a crash, not a misconfiguration.

This is reachable in the documented topology, where orca-xvfb.service has no
User= and runs as root while serve runs as User=orca: /tmp is sticky, so the
orca uid cannot unlink a root-owned socket, rmSync fails, and Xvfb exits with
the display already active.

Readiness now requires the display to actually be live -- our socket plus a
lock naming a running process -- so the same state reports an unusable
display and exits 1 with the existing diagnosis.

* fix(linux): recognise abstract X sockets and inherited Wayland fds

Two display setups this gate could not prove were refused outright, and on the
desktop path that is app.exit(1) with no workaround.

An X server may bind only the abstract namespace (`@/tmp/.X11-unix/X0`), which
leaves no filesystem socket to stat. Abstract addresses are kernel-owned and
vanish the moment the owner exits, so an entry in /proc/net/unix is proof of a
live server -- no lock file needed and no stale entry possible. Verified on
Ubuntu 24.04, where 139 such addresses were present.

WAYLAND_SOCKET is an already-connected fd handed over by the compositor, so
there is no path to stat and WAYLAND_DISPLAY may be unset entirely. Its
presence is the display.

Both are consulted only after the filesystem-socket check fails, so no
existing verdict changes.

* fix(linux): never treat Orca's own display number as a foreign endpoint

Recognising a lockless X socket as live is correct for an endpoint published
from elsewhere -- a container bind mount, WSLg -- because an X server writes
its lock beside its socket and both survive a crash. It is wrong for
VIRTUAL_DISPLAY_NUMBER, because Orca's own teardown unlinks the lock before
the socket and so manufactures that exact state.

The managed branch was already strict, but a caller that sets DISPLAY=:99
explicitly takes the foreign path and skipped it, accepting a dead display
left by Orca's own interrupted cleanup. Route the managed number through the
strict probe on both paths.

Found by an adversarial audit of the asymmetry introduced earlier in this
branch; the documented systemd topology is unaffected because its Xvfb writes
a real lock.

* test(linux): add a packaged-artifact contract for the CLI launch paths

* test(linux): avoid buffered serve readiness detection

* test(linux): signal AppImage serve owner directly

* test(linux): tolerate readiness timeout boundary

* test(linux): add startup margin to shutdown oracle

* ci(linux): give package contracts timeout headroom

* fix(ci): route all Linux packaging contract changes

* test(linux): poll shutdown readiness without tail leaks

* test(linux): bound shutdown cleanup grace

* test(linux): assert on CLI output, not the harness's own control lines

run-cli-case.sh echoes `RESULT status=N case=<name>`, and the two cases named
*-skills asserted `expectOutput: 'skills'`. That substring was satisfied by
the case name in the harness's own line, so 2 of 8 cases asserted nothing
about the command -- gutting `skills` entirely would still have gone green.

Control lines are now excluded before matching, and both cases assert the
rendered help header, which only real help output produces. Verified on an
Ubuntu 24.04 host: 8/8 still pass against a stack-tip AppImage.

Also register the gate in reliability-gates.jsonc, which #15085 added a CI
Docker gate without. Red/green is recorded from a stock release AppImage
failing 4 of 8, three of them at status 133 (SIGTRAP).

* fix(linux): require static AppImage runtimes (#17319)

* test(linux): reject a wrong-architecture native binary at packaging time

Cross-building the arm64 slice on an x64 host silently packed an x86-64
`pty.node` -- the rebuild logged "Forcing native rebuild for linux-arm64" and
shipped the host's binary anyway. Every gate here inspects symbol versions,
which are perfectly valid on the wrong architecture, so nothing noticed.

Observed on a Raspberry Pi 5: the packaged app loaded, then failed with
"Failed to load native module: pty.node", and the launch contract reported
3 of 8 cases crashed rather than naming the cause. Swapping in the aarch64
`pty.node` took the same build to 8/8.

Compare ELF `e_machine` against the slice being packaged and fail with the
offending path. Checked before the glibc pass, because a wrong-architecture
binary's symbol versions are valid but meaningless and would send the reader
down the wrong path.

Release CI builds arm64 on a native runner, so this guards local and future
cross-builds rather than a shipped artifact.

* test(linux): judge per-arch vendored binaries against their own path

The first CI run of the architecture gate failed the x64 package job on
`@parcel/watcher-linux-arm64-glibc/watcher.node`. That binary is arm64 on
purpose: the package ships every architecture and its loader picks the match,
so its presence in an x64 build is correct.

Judge a binary against the architecture its own path names, falling back to
the slice when the path names none. That keeps the case this gate exists for
-- `bin/linux-arm64-*/node-pty.node` holding an x86-64 binary, which is what
shipped to a Raspberry Pi 5 -- while letting multi-arch dependencies through.

Dry-run over the real dependency tree flags nothing for either target arch.

* fix(linux): move deb/rpm update installation outside Orca (#17318)

* fix(linux): complete deb/rpm package metadata

* fix(linux): preserve CLI link during package upgrades

* docs(linux): document local RPM build prerequisites

* fix(linux): move deb/rpm update installation outside Orca

* fix(updater): preserve Linux recovery across stale events

* fix(updater): fence stale downloaded events by active target

* fix(updater): preserve active Linux package recovery

* test(linux): keep workflow order assertion in scope

* test(updater): assert stale recovery stays silent

* fix(updater): preserve Linux package recovery after checks

* refactor(updater): keep Linux marker message with status

* fix(linux): describe the right manual update path for deb/rpm hosts

A remote host installed from .deb or .rpm now reports
manual-service-update-required, and the guidance told the operator to
"update through the service manager that starts this server" -- which is
correct for unsupported-headless-serve but wrong for a package install,
where nothing about the remedy involves the service manager.

Say both, keyed on how the host was installed.

* docs(linux): document orcad update restart safety

* docs(linux): scope restart census omissions

* docs(linux): use absolute service CLI launcher

* fix(serve): validate in-process serve options before startup (#17683)

* fix(linux): stop offering updates a distro-managed install cannot apply (#17918)

Closes #17702.

The resources/package-type marker is authoritative but never checked against
the host, so any repackager that unpacks Orca's .deb -- AUR, Nix, a container
rebuild -- inherits `deb` verbatim. Install feasibility was then computed
after a ~165 MB download, so those users got check -> download -> a card
promising an install command -> a dead end.

Validate the marker against the host: a deb/rpm marker with no matching
package manager in the trusted directories means a package manager owns this
install. This reuses the exact lists and resolver that
buildLinuxPackageInstallCommand already loops over, so a false positive is
impossible by construction -- any host flagged here would have failed with
no-package-manager after the download anyway. The gate only moves that
verdict earlier. Verified across Debian 12, Ubuntu 24.04, Arch, Fedora 40 and
openSUSE Leap: no false positive on a real deb host, correct on every
repackaging host.

The release is still reported, because the user does want to know 1.4.194
exists and to update through their distro; only the download path is closed.
`externallyManaged` is an additive optional field on the existing `available`
status, so older paired clients decode it unchanged. downloadUpdate() refuses
authoritatively, since main owns this verdict rather than the card, and
unwinds any pinned-build state first -- a Linux pinned jump resolves to
'release', and stranding isPinnedBuildActive would silently kill every
background check for the rest of the process.

Note the fix the issue suggests cannot work: electron-updater builds a
PacmanUpdater whose doDownloadUpdate looks for a .pacman asset Orca does not
publish, then dereferences undefined.

* style(cli): restore prettier wrapping on install error copy

* test(linux): re-pin the child-process ratchets and the batch-shim allowlist after the merge
2026-09-02 03:08:01 -07:00
Jinwoo Hong 69fff5eaa3 fix(runtime): defer websocket heartbeat startup probe (#17810)
* fix(runtime): defer websocket heartbeat startup probe

* fix(runtime): defer heartbeat probes until websocket auth
2026-09-01 14:00:11 -04:00
Neil c558d7e083 Activate terminal splits before inherited CWD resolution (#17601)
* perf(terminal): activate splits before cwd resolution

* test(terminal): prove split focus before cwd publish

* fix(terminal): release stale split cwd fence

* test(terminal): add visible split activation latency benchmark

* docs(reliability): clarify split benchmark provenance

* fix: preserve deferred split handoffs across remounts

* fix: fence late deferred split closes

* docs(reliability): record exact split benchmark runs

* test(reliability): fail benchmark on artifact write errors

* test(reliability): attribute split activation phases

* docs(reliability): record schema-v2 split benchmark

* refactor(terminal): collapse duplicated split-handoff and write-queue paths

- Drop the discardDeferredSplitPaneHandoff alias for its identical clear twin.
- Fold the deferred-cwd resolve/reject settle handlers into one applier.
- Extract settlePaneCwdDeferredSpawn for the repeated read-clear-write pattern.
- Share one head-index FIFO primitive between the ordinary and reply queues.

* fix(terminal): stop retaining a promise reaction per acknowledged write

Racing every accepted write against one queue-lifetime cancel promise kept a
reaction record alive until that promise settled: 200k acknowledged writes
retained 88.6MB, now 0.1MB. Give each in-flight write its own cancel, and
split the shared FIFO primitive into its own module.

Also sanitize the split-latency benchmark report at its single serialization
point so shared artifacts no longer carry the machine-local repo path or
unbounded cleanup error text.

* fix(terminal): settle deferred split input when the spawn is abandoned

An abandoned deferred spawn returns before transport.connect(), so nothing
drained the pre-connect buffer: sendInputAccepted's promise never settled and
a paste into that pane hung forever. Clear the buffer on the abandon fence.

Also re-derive the pre-connect retention cap from the clipboard-paste ceiling
rather than the 16MB single-write ceiling; it is held twice per pane across up
to 64 deferred splits, so 5.59M code units guarded the wrong thing.

* fix(terminal): release the deferred cwd fence on a rejected reattach

A daemon createOrAttach can turn an apparent fresh spawn into a reattach; when
that reattach is refused the spawn ends with deferredSplitSpawn/pendingCwd
still set, permanently arming the pre-bind detach refusal. The release no-ops
when a PTY did bind, so it only fires where the fence would otherwise leak.

The stale-generation return above is deliberately left alone: a newer connect
already owns the pane there, and the fence is not generation-scoped.
2026-08-31 16:45:36 -07:00
Jinwoo Hong faaf38ac45 fix(orchestration): submit staged mail pointer while working (#17470) 2026-08-31 14:25:34 -04:00
NeilandBrennan Benson fbe94ceff6 fix: close readiness gaps found by merged-change audit (#17159)
* fix(ssh): fence stale kills and retired pane replay

* fix(ssh): support cancellable interactive authentication

* fix(ssh): await remote catalog before snapshot adoption

* fix(pty): contain Windows ConPTY input failures

* fix(power): avoid redundant macOS display blocking

* perf(editor): narrow markdown override subscriptions

* fix(quick-open): close directory handles after reads

* refactor(linux): remove unused proc socket scanner

* fix(usage): apply flat Sonnet 4.6 pricing

* ci: prime Node next native test cache

* docs(skills): resolve snapshot cleanup data path

* fix(ssh): recover install locks after host reboot

* test(ssh): recognize boot-aware install locks

* test(ssh): prove previous-boot lock recovery live

* test(wire): pin pre-metadata release coverage

* fix(terminal): preserve remote tab ownership through recovery races

* test(runtime): fence replaced terminal handles in agent guard

* fix(ssh): preserve remote snapshot authority across polls

* fix(pty): contain late ConPTY output EPIPE

* test(pty): register Windows exit watcher before kill

* fix: close SSH and tab readiness race gaps

* fix(tabs): retain headless order and placeholder titles

* fix(build): avoid parallel electron-vite config race

* test(windows): avoid MSYS temp path rewriting

* test(windows): avoid killing exited PTY

* fix(pty): avoid late ConPTY input teardown race

* fix(terminal): sync reconnect error ownership after commit

* fix(runtime): use canonical worktree identity comparison

* test(ssh): assert complete cold-hydration baseline

* test(windows): invoke quoted retention fixture via PowerShell

* test(windows): read ConPTY grid through mode con

* fix(terminal): publish PTY replacements atomically

* fix(terminal): infer stale identity on reattach

* fix(terminal): fence stale pane PTY callbacks

* fix(terminal): fence stale pane binds after rebind

* fix(terminal): reject stale pane transport callbacks

* fix(terminal): fence mirrored reattach spawn callbacks

* fix(terminal): replace stale pane PTYs on remount

* fix(ci): size the Windows launcher-compile test budget from measurement

`native-smoke (windows-latest)` fails ~4.5% of runs on
`preserves a multiline argument through the compiled remote launcher`
with "Test timed out in 15000ms" — on unrelated PRs, for reasons that
have nothing to do with them. Across 176 sampled attempts it is the only
red that job produced, and it hit seven different PRs in two days:
#16900, #16904, #16915, #16955 (twice), #16979, #17014, #17085.

The test is six process creations: powershell.exe forks csc.exe, then
the freshly compiled orca.exe forks node.exe, twice. Hosted Windows
runners periodically slow process creation down, and this test amplifies
that far harder than anything else in the job. Comparing the 80 attempts
where it ran under 3s against the 12 where it ran over 12s, its own
median goes 2198ms -> 15917ms (7.2x) while the same file's
powershell-only test moves 556 -> 686ms (1.2x), the cmd.exe and Git Bash
process tests in the neighbouring file move 1.4x, and the other 35 files
put together move 1.5x.

Measured across those 176 attempts: 1881ms to 35438ms, p50 4264ms,
correlation +0.881 with the job's total Vitest duration. 8 of 176 (4.5%)
exceeded the 15s cap; 2 of 176 (1.1%) also exceeded the shared 30s
testTimeout, so deleting the override and inheriting the config is not
enough on its own. 60s clears all 176 with 1.7x headroom on the worst.

This is slow, not hung. Every body here is synchronous spawnSync, so
Vitest cannot interrupt one — the timer fires only after the body
returns and the reported duration is real elapsed time. That is why a
failure reads `× ... 22464ms` under `Test timed out in 15000ms`. The
work finished; the stopwatch was short. Seven reruns at one identical
head measured 2053 / 4680 / 5551 / 8732 / 13506 / 14868 / 21937ms — the
last of those would have been red on code that had not changed.

The 15s came from #8897, which raised this test off Vitest's built-in 5s
default because the job then ran bare `pnpm vitest run`. #8909 landed
3h27m later and pointed the job at config/vitest.config.ts, which is the
real fix for that. The constant stayed behind and has been the binding
budget ever since.

* fix(terminal): fence stale remount reattach ownership

* fix(terminal): reconcile mounted pane identity after replacement

* fix(terminal): fence stale reattach fallback ownership

* fix(terminal): fence deferred SSH reattach ownership

* fix(terminal): fence stale split pane ownership callbacks

* fix(terminal): keep stale spawns from consuming startup

---------

Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com>
2026-08-31 08:17:40 -07:00
Neil 70df6f0224 fix(terminal): mask the agent composer's dim placeholder during a preedit (#17377)
Split out of #17170, which now carries only the xterm composition-overlay work.

Codex and Claude draw an all-dim, full-row ghost placeholder. The opaque preedit
overlay reproduces the committed row tail it covers, so without this the ghost is
repeated to the right of the composing syllable instead of staying masked. The
binding keys off the `.xterm-composition-remainder` class that #17170 adds and
hides it through CSS while a composition owns a structurally verified placeholder
row — bold prompt glyph plus a dimmed model footer below a blank gap for Codex, a
frame line above the prompt for Claude. Arbitrary dim output, shell lookalikes,
and any row carrying typed text keep their tail visible.

readTerminalCursorLineContext moves from src/main/daemon to src/shared because the
renderer now needs the same reader the daemon uses; the move is import-only.

Depends on #17170.
2026-08-30 03:12:22 -07:00
Neil 7f822a73e3 fix(terminal): render the IME caret and give the candidate anchor one owner (#17170)
* fix(terminal): render IME caret without placeholder overlap

* fix(terminal): preserve dim mid-line composition tails

* fix(terminal): keep IME caret visible at row edge

* fix(terminal): harden IME overlay lifecycle and layout

* test(terminal): type final-cell layout mock

* fix(terminal): keep final-cell IME anchor on-screen

* fix(terminal): bind IME masking to composer ownership

* fix(terminal): bound IME placeholder session ownership

* fix(terminal): track latest IME placeholder session

* test(terminal): share IME session event fixture

* fix(terminal): keep both writers of the IME candidate anchor in agreement

`textarea.style.left` has two writers: xterm's patched CompositionHelper and
Orca's terminal-ime-candidate-anchor.ts. The anchor module listens on
terminal.element, so within a composition event it writes after xterm's textarea
listener and reverted the final-column clamp the patch had just applied.

Moving the clamp into the anchor module and dropping the patch hunk does not fix
it, and the rendered e2e caught that: CoreBrowserTerminal.ts:444 drives
updateCompositionElements from onRender as well, so xterm re-asserts the textarea
position on every repaint, with no composition event for that module to hear. The
anchor survived only when no render happened to follow — measured as a flake at the
final column, 1561.28px against a 1557px screen edge, the fully unclamped value.

So both writers now compute the same clamp. The patch keeps it, because it is the
writer on the render path and already holds cursorLeft, maxWidth and the preedit
bounds. The anchor module applies the same one, so its composition-event write no
longer reverts the correction in the window before the next render. Both halves are
individually necessary and both are mutation-tested.

Also restores _getRowRemainderText's expression from main: translateToString(true,
x, line.length) and translateToString(false, x, getTrimmedLength()) are the same
call, since upstream does endCol = min(endCol, getTrimmedLength()) under trimRight.

Adds the two missing tests — one installing both anchor writers in a single rig, one
driving a render under an open composition — plus disposal cleanup and clamp-bound
coverage, and moves the Codex/Claude placeholder mask to a follow-up PR.
2026-08-30 02:23:04 -07:00
Jinwoo Hong 252dbd60ea fix(terminal): restore lossy initial remote snapshots (#17113)
* fix(terminal): restore lossy initial remote snapshots

* test(terminal): strengthen lossy snapshot causal oracle
2026-08-30 03:11:55 -04:00
Jinwoo Hong f572ba34bc feat(browser): address-bar convergence — previews and browser tabs convert in place (STA-5681) (#16998) 2026-08-29 22:38:59 -07:00
Jinjing 73ff003147 test(e2e): cover session upgrade and Windows terminal recovery (#17289)
* coverage report

* rm test coverage

* test(e2e): cover session upgrade and Windows terminal recovery

* fix stub
2026-08-29 22:37:05 -07:00
Neil 7655d20f04 fix(terminal): complete OMP stale cwd recovery (#17154) 2026-08-29 14:34:50 -07:00
Neil 9db319dc06 fix(terminal): recover OMP from stale working directories (#17128)
* fix(terminal): recover OMP from stale cwd

* fix(terminal): harden OMP cwd recovery
2026-08-29 02:27:44 -07:00