mirror of
https://github.com/stablyai/orca.git
synced 2026-10-07 08:02:21 +00:00
52b0d1295eb5b1062fd080d06cded034830f2f32
104
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jinwoo Hong
|
4b1b7178ad |
fix(orchestration): scope @ group addresses to the sender's Run (#19783)
* fix(orchestration): scope @ group addresses to the sender's Run `@all`, `@idle`, and the agent-name groups (`@claude`, `@codex`, ...) resolved against every terminal on the host. A coordinator meaning "my three reviewers" reached 126 agents across every open project, twice in one day, and every unrelated agent burned a turn discarding mail that was never for it. Every group except `@worktree:<id>` now means the live Dispatches of the sender's own Run, each addressed as `dispatch:<id>` so delivery is durable even when the worker terminal is not attached yet. A sender bound to no Run is refused with `invalid_argument` naming `run:<id>` / `dispatch:<id>`; there is no host-wide fallback and the host's terminals are never enumerated for it. `@idle` and the agent-name groups filter within that set by the same terminal status and host-resolved identity as before. `ask --to @group` returns the same code and points at the owning Run mailbox. Federated Dispatches read relayed control mail rather than a local mailbox, so a Run-scoped fan-out skips them with a `recipient_unreachable` warning naming the direct `dispatch:<id>` address. Group addresses are resolved host-side, so no RPC or stream shape changes; an older CLI sending `@all` to a new host gets the Run-scoped meaning. Claude-Session: run-scoped-group-addresses * fix(orchestration): revalidate legacy takeover before the recipient verdict A legacy coordinator taken over while `listTerminals` was in flight reported `runtime_error` instead of `legacy_read_only`: Run scoping made "no live Dispatch in this Run" the first thing the group send could fail on, and that threw before the takeover check ran. Takeover is a precondition, not a commit-time detail — the sender must be told it is read-only whatever else is wrong with its recipient set. Revalidation moves to immediately after the only `await` in the path. Everything below it is synchronous, so the commit-time window it used to guard is unchanged; only the error paths now see it. The legacy partition test gave `term_current_worker` no Dispatch, so under Run scoping it is correctly not a recipient. It now holds a real current-contract Dispatch in the same adopted Run, which is what the test is named for: one `legacy_direct` and one `current_delivery` recipient in one fan-out. Claude-Session: run-scoped-group-addresses * fix(orchestration): address the Run a nested coordinator created, not its parent A nested coordinator is both a worker of its parent Run and the coordinator of the Run it created. `resolveMessageRun` answers with the parent, correctly, because that is where its own `worker_done` belongs — but audience is a different question. Scoping `@all` to that Run sent a nested coordinator's "shared context" to the siblings it was started beside instead of the workers it started, and reported success, so it never learned its sub-workers heard nothing. Before Run scoping the host-wide fan-out reached the sub-workers by accident; this turned an over-broad delivery into a wrong-audience one, the exact failure class the change exists to remove. Group audience now resolves off the Run the sender coordinates, falling back to its Dispatch's Run. A leaf worker coordinates nothing and is unaffected. This is a separate question from `routing.run`, not a second answer to the same one, so `resolveMessageRun` keeps its meaning for point-to-point mail. Also: when every live Dispatch in a Run is federated, the fan-out skipped them all and threw a bare `Error` that discarded the warnings naming those remote workers and how to address each one. The sender was told "no recipients" while three remote workers existed. That throw now carries a code and the skip explanations. Claude-Session: run-scoped-group-addresses * docs(orchestration): say that no group address reaches a coordinator A coordinator is not a Dispatch, so Run-scoped groups never include one. That follows from the rule, but nothing said it, and the old host-wide meaning did include the coordinator — a worker sending `@all` to raise a blocker would be heard by its siblings and by nobody who can act. The guide, the CLI note, and the docs page now say to use `run:<id>` for that, and that a worker which created its own Run addresses that Run's workers. Also restores the `@cursor` case dropped when the group tests moved: a Claude pane titled "Fix the text cursor blink" must not receive Cursor's mail. That hazard was recorded from real titles and `@droid` alone did not cover it. Claude-Session: run-scoped-group-addresses * fix(orchestration): preserve group audience and mailbox identity * fix(orchestration): validate group scope before dispatch routing * fix(orchestration): preserve pane identity and exclude coordinator dispatches |
||
  Brennan BensonandMerge Sim
|
68dd3909c7 |
feat(orchestration): orchestrate native-born structured chat sessions (#18827)
* feat(orchestration): orchestrate native-born structured chat sessions Orchestration resolves every worker through a terminal handle and a pane key backed by a live PTY. A session created directly as structured has neither, so it was not refused by orchestration — it was invisible. A coordinator could not start one, address one, or receive `worker_done` from one. Add a second authority source rather than a parameter channel. A registry maps a session id to the same three facts the PTY path supplies — a bearer handle, a pane key and a host scope — and the four runtime getters consult it before giving up on `ptysById`. `orchestration.send` and `verifyDispatchCapability` are untouched: authority stays host-derived and the CLI still cannot assert who it is. PTY handles short-circuit on the handle prefix, so the terminal path is unchanged. Mail travels as a session turn instead of as bytes, on a sibling lane that keeps the PTY lane's outstanding-run, waiter, reserved-type and batch rules. Orchestration's database stays the source of truth; the send is best-effort, exactly as the byte write is, and mail is consumed only on a proven-accepted dispatch. Delivery waits for the session to be between turns, because one provider refuses a mid-turn start outright and the other cannot acknowledge one inside the ack window. Security properties, each pinned by test: the pane key's leaf is random and persisted rather than derived, since `check` is identity-gated and accepts a caller-supplied pane key; the handle is a random bearer token; the child env carries no pane key, which would otherwise flow into hook pipelines that assume a PTY leaf; hook attestation stays closed for structured handles; and process continuity comes from record lineage, never the runtime fence, which the host bumps during its own crash recovery. Also remove the "Orchestration paused" notice, which gated only on dispatch status and rendered over bridge chat where orchestration always worked; refuse the implicit-sender fallback when a worktree has more than one candidate leaf instead of guessing; and collapse the archive kinds to one named type with a compile-time assertion that the capture set cannot drift ahead of the storable set. * fix(orchestration): answer the structured idle gate from the reduced timeline The structured pointer gate read a bounded 40-item tail page. A settled turn is tombstoned rather than rewritten, so an idle worker with any real history carries no turnLifecycle item at all and the "full page, no lifecycle item" guard read it as busy forever: every nudge after the worker's first substantial turn parked on a settle edge that had already passed, and the preamble tells workers not to poll. The attention gate had the mirror bug — a prompt older than the tail window was missed and the nudge was delivered into a session blocked on a human. Both facts now come from `journal.snapshot()`, the fully reduced timeline, via a new narrow `readGateFacts` host read; the policy module stays pure and still projects through the shared helpers the chat view reads. Also: - Park `session-not-attached` on the journal edge, so mail that arrives during a transient detach is redriven by the re-attach reset instead of sitting unread. - Resolve a structured worker's provider from the durable agent-session record when the registry entry was rehydrated, so a restarted Codex worker is no longer reported and archived as Claude. - Clear `structured_pointer_operations` in every `orchestration reset` scope. - Drop the per-chat-pane dispatch-status store subscription left behind by the removed paused notice, and re-pin the two terminal-pane ratchets it moves. - Hoist the identical pointer batch selection out of both delivery lanes into `selectOrchestrationPointerBatch`. - Refuse the pre-graph-ready focus-based guess for `requireUnambiguous` callers, matching the ready path. - Move the host teardown phase list into the teardown module it belongs to, which is what keeps the host inside its max-lines budget. * fix(orchestration): discard a structured worker session whose create settled unknown `commitStructuredAgentSessionCreate` answers `agent_session_operation_unknown` when `attach` SUCCEEDED and only the tab publish failed, so `created.ok === false` is not proof that nothing exists. The worker start read it that way and skipped `discardCreatedSession`, leaving a live provider child that took no hold, has no `bindingsByDispatchId` entry and no published tab — the outer `releaseStructuredWorkerSession` no-ops without a binding, and a session that never had a holder never starts the eviction clock, so nothing in the runtime ever retires it. A throw out of the commit half is past `attach` for the same reason; the pre-commit half refuses rather than throwing. Cleanup now asks whether the create MAY have committed, via the existing `isDefinitiveAgentSessionCreateRefusal` predicate. Also: - Strengthen the pre-ready `requireUnambiguous` test so it actually pins the guard: the snapshot now carries a focused terminal, so deleting the `? [] :` ternary turns the test red instead of leaving the refusal to the ambiguous `listTerminals` fallback. - Correct the guard's justification comment, which cited `orchestration check` as covered. `check` resolves through the `--terminal` scope and still guesses; the guard covers the implicit `--from` sender, and a structured worker is covered by the `ORCA_TERMINAL_HANDLE` baked into its child. * docs(orchestration): stop two structured-worker comments claiming guarantees the code does not give The send-time owner re-check reads `target.refusal`, the snapshot the resolver already admitted, so `decideStructuredPointerDelivery` can only agree with the resolve-time answer and `owner-not-settled-native` is unreachable from that call site. What actually fences an owner that moved is `expectedRuntimeFence`, which a handoff bumps. Say that, so nobody later drops the fence trusting a re-check that is structurally a tautology. `discardCreatedSession` was credited with retiring "a published background tab that no dispatch owns". It hides the DURABLE tab reference and closes the session; the live tab snapshot keeps the row, so the background tab this start published stays on screen until the app restarts. Same for stop and release. The comment now describes what the two calls do — including that both are no-ops on a session that was never attached, which is what makes the non-definitive-refusal path safe to reach unconditionally. * fix(orchestration): retire a structured worker's chat tab when the worker settles Starting a structured worker always publishes a real `agent-session:<id>` tab, but every settlement path only called `setSessionTabVisibility(sessionId, false)` plus `host.close(sessionId)`. That clears the DURABLE restore index and leaves the LIVE snapshot untouched, so stop, release and the half-started discard all left a dead "Claude Chat" / "Codex Chat" tab in the worktree's tab bar for the rest of the app session — five dispatches, five dead tabs — and opening one re-attached the released session, respawning a provider child outside orchestration's hold accounting. The snapshot-pruning half of `closeStructuredAgentSessionTab` is extracted into `structured-agent-session-tab-retirement.ts` and exposed on the runtime as `retireStructuredAgentSessionTabFromSnapshot`, so the user-initiated tab close and the three settlements share one implementation instead of a second copy. The settlement side is best-effort BY CONSTRUCTION: it runs only after the close is already proven, calls the runtime method optionally, and swallows any throw. It talks to no renderer, so the startup release reconciler can call it too. Nothing here can turn a proven stop into `release_unknown`. * fix(orchestration): stop a structured worker's nudges, archive and liveness from lying Five defects in the structured-worker lanes, each with the same shape: a check that answered from something other than what it claimed to measure. - The pointer lane gated a WORKER's `dispatch:` mailbox on its RUN's outstanding delivery. Delivery rows exist only for a `run:` address, so that row belongs to the coordinator — and a coordinator holds one for exactly as long as it is acting on received mail, which is when it replies to its workers. The gate is gone; there is no coordinator mailbox in this lane to protect. - `dispatch-rejected` now parks on the journal edge. A rejection consumes no mail and nothing else redrives the mailbox, so an unparked pointer left the worker idle on durable mail until unrelated mail happened to arrive. - The released journal archive bounded forward — keeping the HEAD — before capping newest-first, so a long worker's archive ended at its early exploration and dropped the answer it was released for, under a warning that said the oldest messages had gone. One newest-first pass now, and the warning is true. - The durable pointer operation id was reused on a matching BODY fingerprint, and the body names only the unread count. Two unrelated same-size batches collided, the host replayed its ledger answer as `accepted` with no turn sent, and the lane marked the new mail delivered. Reuse is keyed on the batch's message ids. - `worker-read` on a structured worker hardcoded `terminal: 'running'` and emitted no `liveness`, so a runtime that could not see the session reported the worker as alive. It now carries the observed verdict, as the PTY branch does. Also: the live journal cursor is an index into a re-derived tail window, so the page's oldest item joins its source identity — a slid window now answers `source_changed` instead of silently resuming past the items it skipped. And a stop that reached no host reports `processAction: 'none'`, after installing the host the way release already does. * fix(orchestration): stop a released structured archive claiming a close that never landed `worker-read` on a released structured worker hardcoded `liveness: 'exited'`. The archive is frozen BEFORE the close, so it proves nothing about the provider child, and the read is served for `release_state` in `releasing` / `unknown` too — the two states that exist precisely to record a close that did NOT land. A coordinator that read `exited` from a `release_unknown` worker would start a replacement over the same worktree while the original child was still attached, which is the outcome docs/reference/ssh-execution-boundary.md rule 2 exists to prevent, and it contradicts the release receipt's own "the structured session close was not proven" text. The verdict now comes from the resource row the read already holds: only a settled `released` row is `exited`, everything else is `unverifiable` — which the existing mapping renders as `terminal: 'unknown'`, the same way the live branch does. * fix(orchestration): stop a structured worker-start reporting a preamble it never delivered Two ways a structured `worker-start` handed the coordinator a receipt that did not describe the worker it got. `sendStructuredWorkerPreamble` threw only on a refusal and on `rejected`, so a submission that settled `unknown` fell through as success: the start pushed `dispatch_input: accepted` and marked the dispatch ready. `unknown` is not rare — `dispatchSafely` converts ANY thrown adapter call (provider child gone, transport dropped, ack window missed) into it, and `performSend` still returns ok. The worker then has no task spec while its coordinator blocks in `check --wait --types worker_done` until timeout. This PR's own mail lane already states the rule — "`pending` is not yet an acknowledgement; only `accepted` may consume mail" — so the preamble now applies it too, and raises `operation_unknown` for the states that prove neither delivery nor failure, which is the code `failWorkerStartWithReceipt` turns into the `outcome_unknown` receipt whose nextCommands send the coordinator to look. `rejected` stays a proven failure. `--structured` also accepted `--model` / `--effort` and dropped them: structured session creation takes no launch preferences, while `launch.receipt.effective` echoes whatever was requested either way, so `--model opus` ran on the workspace default and the receipt still said `opus`. Refused now, for the same reason `--terminal` refuses them, and the spec note records that refusal along with the new-child/new-top-level one it never mentioned. Tests: the refusal guard had no coverage at all, and `structured-mailbox-pointer-host` — where the full-timeline gate read lives — had none either; reinstating the bounded tail there left the whole repo green. Both are covered now, and the vacuous "never selects an exact provider session" case is re-pointed at the absent `ORCA_PANE_KEY` that actually keeps that selector shut. * fix(orchestration): let a structured worker actually reach the Orca CLI, and stop four settlements lying A structured worker's provider child runs `orca orchestration ...` exactly like a PTY worker's agent does, but it was handed the ambient PATH. On packaged Linux the CLI installs as `orca-ide` so it never claims GNOME Orca's /usr/bin/orca (#7904), so bare `orca` execs the screen reader and the worker can never read mail, reply or send worker_done; on packaged macOS/Windows the bundled launcher is only reachable from the app's own resources dir. The PTY lane already solves this inside `buildPtyHostEnv`; that block is now its own module and both lanes call it. Also: - a worker start that fails AFTER its session exists now discards the session, so a failed start stops stranding a dead chat tab that the durable restore index republishes on every launch; - a structured worker's resource reconciles to `released` after settlement forgot its identity, instead of answering `unverifiable` for the life of the DB; - `closeAttempted` is set only once a close is issued, so a tab-visibility failure can no longer report `closed_agent_terminal` for a running child; - `forgetSession` prunes only what the settled worker parked, not every sibling whose target momentarily fails to resolve; - release settles with an explicitly empty, warned archive when the journal is unreadable AND the session is proven exited — closing the chat tab is routine, and `archive_failed` there wedged release on evidence that could never arrive; - the new migration test uses mkdtemp and cleans up, so it stops failing Windows CI and leaking. * fix(orchestration): merge the duplicated release-receipts import The release-completion module imported ./orchestration-worker-release-receipts twice, which trips import/no-duplicates in audit:code-quality:native. The changed-file gate does not load that config, so only whole-tree CI saw it. * docs(runtime): note that a background structured tab re-publish is a no-op The activate:false branch for an already-published session returns without writing the snapshot or emitting, so it cannot re-surface a client whose mirror lost the tab. Orchestration is safe from this only incidentally. * feat(orchestration): make the worker mode the user's own default, not a flag `worker-start --structured` was an explicit opt-in that REFUSED --on, --terminal, --model/--effort and worktree-creating placements. The flag, its spec entry and the `structured` RPC param are gone: the mode now follows the user's setting for new agent tabs, so a local claude/codex worker is a structured chat session whenever the user's own default says agent tabs open as one. A setting is a preference, not a demand, so none of those combinations refuses any more. A dispatch that cannot be structured starts an ordinary PTY terminal worker and the receipt names the mode that ran and why, so the fallback is never silent: - a remote --on, an existing --terminal, a new-child/new-top-level worktree and --model/--effort are decided from the request; - the agent, TUI launch customization, Codex-on-Windows and the runtime capability are decided by the shared launch route; - WSL, remoteness and the Windows start-time gate are settled by the executing host's own agentSession.createSupport, asked once the worktree resolves and before anything is created, so a refusal is a terminal worker rather than a failed start. The decision is the renderer's, lifted rather than copied: `resolveAgentLaunchRoute`'s structured half and the settings predicate now live in shared/structured-native-chat-launch-route, which both surfaces call, and the TUI launch customization test moves to shared beside it. `getClientSettings` gains the two native-chat default booleans it was missing. No security invariant moves: the structured worker registry, bearer handle, persisted pane key, the absence of ORCA_PANE_KEY from the child env, hook attestation and lineage-derived process incarnation are untouched. * fix(orchestration): stop the worker mode leaking into the agent contract The mode a worker runs in is a runtime implementation detail. An agent should be taught the same verbs, run the same commands and read the same receipts whether it is a structured chat session or a PTY terminal — otherwise a settings-driven fallback silently changes what the agent can do. The real leak was `canDispatchSubWorkers`, which was forced false for a structured worker. That was not a wording choice: `worker-start` resolved `--from` through `showTerminal`, which needs a live PTY or renderer leaf, so a `structworker_` coordinator genuinely could not dispatch. Rather than withhold the capability, the one fact the command needs from `--from` — its worktree id — now comes from `getOrchestrationDispatchAuthority`, the same authority the pane-key and process-incarnation getters already answer structured handles from. Sub-dispatch is gated on depth alone, identically for both modes. `showTerminal` itself is deliberately NOT taught structured handles: it returns a ptyId, a leaf id and a pane runtime id, and synthesising those for a session with no PTY would hand every caller of a public terminal verb something that looks writable and is not. `inspectWorkerTerminal` already returns `terminal: null` for exactly that reason. Also neutralised three agent-visible refusals that named the worker's kind: a `worker-read --source terminal` on a worker with no terminal now names the sources that do work, and both archive refusals say "transcript output" rather than "structured chat output" (the PTY `transcript_pin` branch said "structured" too). New tests pin both properties: the two preambles are byte-identical once the handle and per-dispatch ids are normalised, and a structured coordinator starts a worker with `showTerminal` rejecting. * fix(orchestration): stop claiming a structured worker was checked for a prompt worker-show reported observation.agentWait: null for every structured worker. The field's own contract says null means Orca looked and found no wait, and absent means it never looked — and nothing looks here: a structured worker parks on a journal question item, which no terminal prompt scan can see. So null was a false negative on the one field a coordinator is explicitly told to read, and it was mode-dependent: the same worker as a PTY would have reported the wait. Absent is both the honest value and a state a PTY worker already reaches (an older host, an unreadable pane, a probe that did not answer), so it discloses nothing about which mode ran. * docs(cli): stop the worker-start spec pointing a caller at the worker kind The note said "the receipt mode field names the mode used and why", which is an instruction to read a field no verb behaves differently for — the one thing the mode was not supposed to become. It now says what a caller actually needs: the dispatch always starts, the options passed are the ones honoured, and every worker is driven the same way. The receipt still carries the mode for operators and telemetry; nothing tells an agent to look at it. * perf(orchestration): coalesce the structured redrive edge Every journal batch is a redrive candidate, because a settled turn is tombstoned rather than rewritten — there is no completed row to watch for. That is free while nothing is parked on the session, but once mail IS parked each batch re-resolved the dispatch, queried unread mail and read the host's gate facts, only to re-park because the turn was still running. A turn streaming tool calls paid that per batch. The edge now coalesces on a 300ms quiet window with a 2s starvation cap, so a streaming turn costs a handful of evaluations instead of one per batch and a settled turn still nudges promptly. Delivery semantics are untouched: the gate, the accepted/rejected/unknown handling and the retain rules all still run exactly as before, just fewer times. Nor is this the path fresh mail takes to an idle worker — that is `deliverForHandle` at enqueue time, which this does not touch — so the common case gains no latency. The mechanism is the session.tabs notify coalescer, generalised into `keyed-trailing-edge-coalescer` and called by both rather than duplicated; the session.tabs windows stay where they were, since 50ms is right for a spinner title and far too tight for a journal stream. Disposal drops the pending timer rather than flushing it, on the existing subscription disposer that every settlement already reaches, so a redrive can never fire for a session no dispatch owns. * fix(orchestration): deliver direct peer mail to a structured worker, and let a peer read it Two agent-to-agent verbs had no answer for a worker that IS a structured agent session, and both failed quietly. Mail addressed to a worker's own bearer handle — how agents mail each other outside a dispatch — fell between the lanes. The send stored durably and reported success, `getLiveTerminalPaneKey` resolved the recipient, and then neither lane claimed the mailbox: the structured resolver answered only `dispatch:` addresses, and the PTY lane refuses a structured handle outright. Nothing errored and nothing logged, so the worker never reacted and the peer waiting on a reply hung. The resolver now also answers a bare worker handle, preferring that worker's active dispatch so peer and coordinator nudges share one operation-ledger budget. A worker BETWEEN dispatches is still nudged, under a session-scoped key: a dispatch says nothing about whether delivery is safe — the idle gate and the lease fence do — and its own `check` reads exactly the direct mailbox the mail is sitting in. The dispatch caller key is left byte-identical, because the ledger is keyed on (callerKey, operationId) and reshaping it would re-mint nudges already in flight as second turns. `terminal read` had no structured branch, so the only peer-accessible read verb answered `terminal_handle_stale` for a live worker; `worker-read` is closed to a peer, which holds neither coordinator standing nor a dispatch id. It now serves the session's journal, projected to LINES and paged by the same reader the PTY tail uses, so the result stays a plain RuntimeTerminalRead and nothing an agent reads discloses which kind of worker answered. Bounding and dispatch-capability redaction are the archive path's, reused rather than rebuilt. A session that is not attached refuses with the existing not-attached code rather than returning an empty tail, which would read as "this worker has said nothing". `terminal.show` still refuses a structured handle. This is read-only on purpose: synthesising a ptyId/leafId/paneRuntimeId would hand every public terminal verb something that looks writable and is not. * fix(orchestration): stop three PTY-only probes answering for structured sessions Three defects, one shape: a probe that enumerates PTYs or resolves a pane was standing in for a question that is not about panes at all. `worktree rm` destroyed a live structured worker. `killAllProcessesForWorktree` sweeps the renderer graph, the provider session list and the local pty-registry, and a structured session is registered on none of them — so all three counted zero, nothing errored, and removal deleted the checkout out from under a running provider child, which kept running with its `cwd` gone while the dispatch still reported the worker live and exact. A fourth sweep now asks what the other three cannot: membership by `location.workspaceId`, which covers a plain chat session as well as a dispatched worker, and liveness by the same `live`/`unverifiable`/`exited` observation the rest of the structured surface uses. It REFUSES a destructive removal rather than auto-closing, on the same bargain and the same `--force` escape hatch as the unstopped-PTY gate — this is the verb that deletes a user's work, and a running agent is exactly what they would want to be told about. Force closes the sessions properly instead of orphaning a child. Best-effort reconciliation callers are excluded: they repair state, delete nothing, and must never be failed closed. Twelve coordinator verbs failed for a structured worker running as itself. `isLiveTerminalHandle` validated `ORCA_TERMINAL_HANDLE` with `terminal.show`, a PTY verb whose leaf lookup misses for a session that never had a pane; the pane remint that would have recovered it needs `ORCA_PANE_KEY`, which a structured child deliberately does not carry, so every one of them died on `no_active_sender_terminal` — including the ones the worker's own dispatch preamble tells it to run. The identity question gets its own probe, `terminal.resolveIdentity`: a handle and a boolean and nothing writable. `terminal.show` still refuses a structured handle, because synthesising ptyId/leafId/paneRuntimeId would hand every public terminal verb something that looks writable and is not. The PTY half is byte-for-byte today's check, `getLiveLeafForHandle` included, so its `rendererGraphEpoch` re-check still runs — that check is the whole reason the sender is validated at all, and a cheaper probe would have quietly started passing stale post-reload handles. A host that predates the method answers `method_not_found` and the client falls back to `terminal.show`, which is correct for that host: one without the identity probe has no structured workers to miss. `dispatch --inject` reported `no_agent_detected` for a structured worker, because `isTerminalRunningAgent` reaches `getLiveLeaf`, throws, and the catch returns false. A structured session IS the agent; there is no foreground process to recognise, so it answers before the PTY probes rather than through them. Also: a Run whose coordinator is structured now gets its `run:` mail. Both lanes declined and neither logged — the PTY lane because the owner is structured, the structured lane because the mailbox was not `dispatch:` — so each half believed the other owned it. The PTY lane's reasoning (a coordinator blocks in `check --wait`, where a waiter preempts pointer delivery) does not transfer: a structured coordinator is a chat session whose turn ends. Its `run:` deliveries take the `hasOutstandingRunDelivery` gate the PTY lane applies for exactly that mailbox, and only for that mailbox. The test that would have caught the twelve drives the CLI with `ORCA_TERMINAL_HANDLE=structworker_…` and no `--from`. Every existing orchestration CLI test passes `--from` explicitly, so the resolver a real worker goes through was never exercised — which is why the suite stayed green while the preamble failed on its first line. Two files crossed their line ceiling and are split rather than waived: `worktree-teardown.ts` sheds its two PTY-surface sweeps and the deadline arithmetic they share, and `orchestration.test.ts` — which sat exactly on 800 — sheds the two caller-identity suites this change rewrote. * fix(orchestration): arm the takeover signal for structured chat input `worker-release` closed a structured session a user had taken over, losing work mid-conversation, while `orchestration-worker-specs.ts:106` promised "Never closes … user-taken-over terminals". Every guard was already correct and simply never armed. `reportWorkerTerminalUserInput` has exactly one call site — the real-user-input signal on a PTY connection — so structured chat input never reached `orchestration.workerTerminalUserInput`, `markWorkerTerminalUserOwned` never ran, ownership stayed `owned` instead of `user_owned`, `retainedReason` never returned `user_takeover`, and `stopStructuredWorker` proceeded. The durable flag is reused as-is rather than given a parallel mechanism: it exists precisely so a restart, an SSH drop or a renderer remount cannot erase a takeover. Addressed by SESSION, never by pane key. A structured worker's pane key is a random identity credential — anyone holding it can read and consume that worker's mailbox, and session ids are embedded in tab ids in plain text — so it stays in main and the runtime resolves the session to it. Handing it to a renderer to echo back would make it learnable by anyone who can see a chat pane. The RPC gains an optional `sessionId` alongside `paneKey`; a host that predates it rejects the call, and the report is already best-effort with a catch, so that host degrades to exactly today's behaviour rather than failing a send. The signal fires from the composer send hook and only past `accepted`: the outbox dispatcher retries, and orchestration's own pointer nudges never pass through the composer at all — so neither can be mistaken for a user takeover. * fix(orchestration): reach structured workers through group addresses `orca orchestration send --to @all` — and `@idle`, `@claude`, `@codex`, `@worktree:<id>` — silently skipped every structured worker. Recipients came from `listTerminals`, which enumerates leaves and PTYs, and a structured session is on neither. The exclusion happened BEFORE per-recipient resolution, so the `SendRecipientWarning` machinery never ran: the caller got exit 0 and a receipt naming the workers that did resolve, and a broadcast "stop work" or "base moved" reached the PTY workers and nobody else. With every worker structured it degraded to `terminal_not_found`, which reads as "the group was empty". Fixed at the group-resolution site rather than inside `listTerminals`. That result is published to paired mobile and remote clients and to consumers that assume a summary carries a `ptyId` or is writable, so widening it is its own change under `docs/reference/remote-wire-compatibility.md`. Group addressing reads exactly three fields off a recipient, and `RuntimeTerminalSummary` already satisfies them structurally, so the resolver widens to that smaller shape and nothing here invents a `worktreePath` or a `branch`. Candidates are liveness- gated on the same observation the rest of the structured surface uses — mail addressed to a settled worker would be stored for a lane that will never deliver it — and once a worker IS a candidate, the existing per-recipient warnings cover it, so an unresolvable one is reported rather than dropped. `@idle` needed more than enumeration: `getAgentStatusForHandle` reaches a PTY probe that throws for a handle with no pane, so a structured worker would have been enumerated and then silently dropped from the one group address that selects on status. It now answers from the session's journal — and off the FULL reduced timeline, never a bounded tail. Settlement tombstones the running turn's lifecycle item rather than rewriting it, so on any page-sized read a long tool-calling turn looks identical to an idle session; `@idle` would then broadcast into a running turn, which Codex answers with `turn already running` and Claude queues behind. An unreadable session answers null, never idle. `terminal list` and `worktree ps` still omit structured workers; that is the wire-visible half and is deliberately not in this change. * fix(orchestration): refuse rather than guess when a chat session has no identity An ordinary structured chat session — not a dispatched worker — is spawned with no `ORCA_TERMINAL_HANDLE`, because `structuredWorkerChildIdentityEnv` early- returns for any session outside the worker registry. `orca orchestration check` then fell through to `terminal.resolveActive`, which picks the focused tab's active leaf or the first leaf in the worktree. It returned a valid handle, so nothing errored — and `check` is destructive by default, so it consumed another pane's oldest unacknowledged batch and marked it read. The rightful worker never saw that mail. `requireUnambiguous` does not fix this, only narrows it: it refuses when MULTIPLE leaves could be meant, and with exactly one terminal pane in the worktree the guess still resolves — to a sibling. "One terminal pane plus one chat tab" is a normal layout, so the common case stayed broken. The pinned test is that case. So the child now carries `ORCA_STRUCTURED_SESSION`, and every remaining route that would GUESS an implicit terminal refuses on it with an error naming the flag to pass. The marker names NOTHING — no handle, no pane key, no session id, no token — which is the whole reason it is safe: it cannot be replayed, cannot impersonate, and cannot flow into the hook-attestation, agent-row or mobile-projection pipelines the way a pane key would. That makes it a different decision from withholding `ORCA_PANE_KEY`, not a reversal of it. It also grants no CLI reachability, so packaged builds keep exactly today's exposure. The comment at `orca-runtime-adopt-terminal-orphans-from-inventory.ts` that justified the guess — "a structured worker is covered instead by the `ORCA_TERMINAL_HANDLE` its child is spawned with" — was true only for dispatched workers and false for every other structured session, a population this branch creates. It now says which case it covers and which case it does not. * fix(orchestration): stop two surfaces lying about a worker with no terminal `orca terminal <verb>` answered `terminal_handle_stale` for a structured worker's handle. Nothing went stale: the session is live and simply has no terminal, and it never had one — so callers acted on a false claim and went hunting for a remint that cannot exist. The refusal now carries its own code and names the structured equivalents (`orca terminal read`, `worker-read --source transcript`, `orca orchestration send`), so an agent that lands there learns what to run rather than what failed. A PTY handle that really did go stale keeps the old error, and so does a session this runtime no longer owns — that handle IS dead. `terminal.show` stays non-resolving: synthesising a ptyId/leafId/paneRuntimeId would hand every public terminal verb something that looks writable and is not. `orchestration-worker-specs.ts` promised "the same verbs, the same handle, and the same worker-read sources", and all three clauses were false for a worker with no terminal. A spec agents read must not carry a false promise, so it now states the limitation and the alternative that always works. Note this had to be reconciled with an invariant this branch already holds: the worker MODE must stay opaque, or a coordinator starts branching on something no verb it runs behaves differently for. So the note says "not every worker has a terminal" and points at `--source auto`/`--source transcript` WITHOUT naming a kind — the same mode-neutral wording `readStructuredWorkerOutput` already uses when it refuses `--source terminal`. Both properties are now pinned by tests, so neither can be restored by breaking the other. * fix(orchestration): close the review findings on the structured parity work Four defects and two follow-ups from the delta review. The `worktree rm` refusal was a dead end in the desktop UI. Its message matched no matcher in `classifyWorktreeForceDeleteReason`, and an ordinary desktop delete already passes `force=true` for the dirty-file skip, so classification returned null unconditionally: the toast showed raw CLI wording with no Force Delete button, and a user with a live chat session was stuck unless they knew to reach for the CLI. That is the #11960 shape `shared/worktree/removal.ts` documents, so the refusal now has its own prefix, matcher, `WorktreeForceDeleteReason` and toast copy, classified BEFORE the `force` guard and nulled once the waiver is spent — exactly how `unstopped-pty` is handled, with matcher and hint kept in the same file as that contract requires. The copy says Force Delete will close a running conversation rather than borrowing the "could not confirm" wording, because Orca watched these sessions stay attached; there is no doubt to waive. Structured `terminal read` cursors were unsound and are now refused. The PTY cursor indexes an append-only completed-line buffer with a monotone count; a session journal is a BOUNDED tail re-projected on every read, so a saved index addressed different lines as the journal grew — and `truncated` could never fire to say so, because it tests `cursor < oldestCursor` and `oldestCursor` was always 0. A poller got wrong or duplicated lines under `truncated:false`. Separately, a streaming turn's lines counted as completed with `partialLine` hardcoded empty, so a mid-turn cursor consumed a half-written line whose growth was never redelivered — the `"hel"`/`"hello"` hazard the PTY reader guards against. The journal does have stable item identity, but `terminal.read`'s cursor is a number on the wire and cannot carry it, so a cursor read now refuses and names `worker-read --source transcript`, which already has that contract including `source_changed`. No cursor space is advertised either: `nextCursor` is null and the cursor fields are absent, rather than claiming an index the next read cannot honour. The header claim that all four fields kept their meanings was true of the shape and false of the invariants; it now says which ones hold. Two fixes had no test at their real seam, which is the same failure that produced this whole set — the runtime tested directly, the seam tested by neither. The group-addressing test hand-composed the recipient list itself, so deleting the composition at the call site left it green; it now drives `sendGroupMessage` with no PTY terminals at all. Nothing referenced `isLiveStructuredAgent`, so the `dispatch --inject` fix had no red-then-green at all; it now has one driving `RuntimeTerminalAgentPresence.isRunning`. Both were ablated and confirmed red. Folder-workspace removals sweep and kill PTYs without `requirePhysicalStop`, so the structured sweep no-opped there and left a live session bound to a workspace about to be forgotten. They now close best-effort under an explicit `closeStructuredSessions` flag, kept separate from `requirePhysicalStop` because the two questions differ: that one asks whether a stop must be PROVEN before files are touched, and it is what licenses a refusal. These paths do not refuse — the root is shared so no checkout vanishes under the child, and one of them is a never-throw forget a refusal would wedge. Reconciliation sweeps set neither and still close nothing. Also: the force close is raced against the same sweep deadline every PTY surface is bounded by, so a wedged provider close reports the timeout instead of hanging `worktree rm --force` forever; and the refusal now prints a count and the providers instead of raw session ids, which our own marker rationale treats as one tab-id hop from a credential. * test: pin structured-session close on the folder-workspace removal path The folder and orphan removal callers now pass closeStructuredSessions so a live structured session is closed best-effort rather than left bound to a workspace Orca has forgotten. These three exact-args characterizations describe that call and had not been updated. * fix(orchestration): stop the structured worker-read cursor misdelivering silently `worker-read --source transcript` for a structured worker fingerprinted only the oldest item's id, so `source_changed` fired when the window slid off the front and could NOT fire when the page's contents changed under a stable oldest item — which is the normal case, because the journal is a reduced, mutable timeline. A `running` tool item gains its `[tool result]` at its original sequence once later items exist, the 60ms delta coalescer revises a message in place, settlement can rewrite an item smaller, and a pending approval projects to null until it resolves and then appears in the MIDDLE of the array. Two silent failures followed, both returning ok. Omission: a caller handed a coalesced `hel`, resuming past it, never received the revision to `hello world` — the same defect we refused to ship on the terminal read path, already shipped here. Duplication: a resolved approval inserted ahead of a saved index, which was still accepted, so the caller re-read content it already had. The blast radius is the coordinator polling loop, the verb's primary consumer. The anchor is now the oldest item PLUS every item whose projected message sits below the caller's position, by id and revision. `createWorkerOutputSourceIdentity` already takes an arbitrary string array and the cursor is already opaque base64url carrying its own position, so neither the wire shape nor the `source_changed` contract changes. Prefix-scoped rather than whole-page deliberately: fingerprinting every item on the page would flip the identity every 60ms with the coalescer window during an active turn, making the cursor unusable exactly while the worker is working — that trades a silent bug for a useless verb. Tail growth the caller has not read cannot invalidate; any change to what it already holds does. Position-dependence is safe because `p` rides in the same opaque payload as the identity, and the returned cursor is stamped with the identity of its own end, which is precisely what the next read recomputes. The frozen archive keeps a constant identity: no item can be revised under a caller there, so it has no prefix to fingerprint. Both silent shapes are pinned across a page boundary with the journal mutating between reads — a static-journal test passes either way. Two ablations at the real call site: reverting to the oldest-item-only anchor turns both red, and widening the prefix to the whole page turns the tail-growth case red, which is what proves the scoping is real in both directions. * docs(orchestration): stop the structured terminal-read refusal recommending a dead end The refusal told a peer to "page it with `orca orchestration worker-read --source transcript`", which is wrong three ways and this file said so itself: its own header explains that this verb exists BECAUSE `worker-read` demands a dispatch id and coordinator standing "a peer does not have" — and then the refusal sent that same peer there. The verb it named is also a window index over the same bounded page, so it is not a paging answer even for a caller who can reach it; under load it now answers `source_changed` on most polls, which is better than the silent hole it had before but still not what the sentence promised. The refusal now says what actually works — the tail is bounded and newest-last, so poll it and diff — and names no alternative, because there is none. That is the honest framing: a durable cursor is not achievable here at all, rather than blocked on the wire shape. The journal is a reduced, MUTABLE timeline: an item's projected text changes at its original sequence after later items exist, the delta coalescer revises repeatedly, settlement can rewrite an item smaller, a pending approval renders as nothing and then as something, and `sequence` resets on epoch rollover. No index, numeric or opaque, survives that. So the docstring's "pagination with a real anchor lives on `worker-read --source transcript`" is gone too — there is no real anchor there — and the file now records why no windowed alternative should be built later: a broken cursor fails UNSAFE, as a silent hole in a poller's output, while diffing a bounded tail fails safe as a harmless re-read, and a second paging-shaped verb would invite the PTY assumptions this one cannot honour. The test asserted the old advice, so it now pins the contract instead: the refusal explains the working approach and must never name `worker-read`. `worker-read --source transcript` remains a good bounded snapshot for a coordinator reading a worker it dispatched; only the "or page it with" clause was false. * fix(i18n): add the missing worktree-removal agent-session refusal string The structured-session removal refusal introduced a translate() key with no en.json entry. Nothing local catches that: typecheck passes, and the full suite passes, because a missing key falls back to its inline default at runtime. Only verify:localization-catalog fails on it, which is why CI's static analysis reddened on a branch that was green everywhere else. Fallback wording mirrors the sibling unstoppedPtyLive string, since the two refusals differ only in what is still running and what Force Delete does to it. * test(codex): expect the no-identity marker on an unregistered structured child The refuse-rather-than-guess marker landed after these expectations were written, and all three assert exact env equality on the unregistered path — the one branch that now carries ORCA_STRUCTURED_SESSION. One of the two files was added by this same branch, so this is a self-inflicted drift; the other predates the branch and was broken by it. The marker's presence is still pinned positively by structured-worker-child-identity-env.test.ts and the CLI's orchestration-structured-session-no-identity.test.ts, so relaxing these three exact-equality checks loses no coverage of the security property. * fix(orchestration): require exit evidence before settling structured close --------- Co-authored-by: Merge Sim <sim@local> |
||
|
Jinwoo Hong
|
06a607a1d7 |
feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->
| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$12401 |
<!-- /orca-pr-loc -->
## ELI5
Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.
## What changed
- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.
## Why
User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.
## Linked issues
Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.
## Review record
This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.
**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:
- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.
Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).
A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.
A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.
Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).
## Testing
- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on
|
||
 Neil
|
51eed5a1bc |
feat(cli): report SSH host platforms (#18896)
* feat(cli): report SSH host platforms * feat(cli): include SSH connection status * fix(cli): preserve unknown SSH connection state |
||
|
Neil
|
95eed52801 |
fix(cli): report which hosts a worktree listing covered, and stop the cap starving remote ones (#18417)
`orca worktree list` returned zero of 24 SSH worktrees at the default limit (#18104). Rows are resolved repo by repo, so every SSH repo's rows land contiguously at the end of the fleet order — the 24 remote rows sat at indices 496-520 of 521 and a plain `slice(0, 200)` never reached them. The omission was not fully silent: text output printed `truncated: showing 200 of 521` and JSON carried `totalCount` / `truncated`. What was missing is that the omission was *categorically every remote host* — no host column, no `hostScope`, nothing to distinguish "200 of 521" from "one host is entirely absent". Per docs/reference/ssh-execution-boundary.md, a listing that does not name its scope reads as absolute. Adopt the mechanism `terminal list` already has rather than inventing a second one: - `RuntimeTerminalListHostScope` becomes an alias of a shared `RuntimeListingHostScope`, now also carried (optional, so old hosts are unaffected) on `worktree.list` and `worktree.ps` results. - `src/shared/host-balanced-listing-page.ts` round-robins the row cap across hosts and returns the survivors in the caller's original relative order, so the page stays a subsequence of the unbounded listing and nothing downstream re-sorts. An uncapped listing is returned unchanged. - `worktree list` / `worktree ps` text output gains a `host=` column and the same trailing `scope:` line `terminal list` prints. Third defect, same mechanism: `hostScope.omittedHostIds` is built from the runtime's own bookkeeping, so it names `runtime:` ids for servers that are no longer paired — 6 of 9 in the recorded QA run hard-error when queried. Since `hostScope` is *the* documented way to complete a partial listing, that makes the mechanism unreliable for its intended use. Annotate rather than filter. Dropping an id would shrink what the listing admits it did not cover, and the boundary doc requires a listing to name its gaps — the gap is real whether or not this machine can name the host that owns it. `src/cli/omitted-host-scope-selectors.ts` resolves each omitted id against this machine's pairing store and the runtime's SSH-target registry and attaches the exact flag that reaches it, or `null` marked "not selectable from this machine". This is a client-side annotation: nothing new goes over the wire, it answers "can I select it" and never "is it up", and the SSH round trip is only paid when an `ssh:` host was actually omitted. No `--host` filter was added; the host column plus scope line covers the reported need without a new selector axis. |
||
|
Neil
|
9bed758e36 |
fix(cli): reject runtime selectors on host list and environment list (#18405)
`orca host list --environment m4air` was not ignoring the flag — it was applying it to half the answer. `shouldIgnoreRemoteSelection` never pinned the `host` family, so the SSH-target lookup was routed to m4air while paired servers were still read from this machine's own pairing store, and the handler stamped the envelope `_meta.runtimeId: "local"` regardless. The result was one listing describing two hosts: the openclaw row silently disappeared, which reads as "m4air has no SSH targets". `environment list --environment X` had the pin but no guard, so the flag vanished with no signal at all. Reject rather than route. `host list` answers "what can this machine target and with what flag"; its paired-server half comes from a client-local store and cannot be routed at all, so any routed answer is necessarily half-substituted — rule 1 of docs/reference/ssh-execution-boundary.md. `environment list` is entirely client-local, so there is no other host to ask. This matches the `account` and `artifacts` precedent, the only two pinned families that already paired the pin with a rejection guard. - pin the `host` family so an ambient ORCA_ENVIRONMENT cannot produce the same two-machine listing with no flag to reject; `runtimeId: "local"` is now true - extract the duplicated `rejectRemoteSelectionFlags` from account.ts and artifacts.ts into src/cli/remote-selection-flag-rejection.ts - `environment show` / `environment rm` / `environment add` are untouched: there `--environment` and `--pairing-code` name the row to act on, not a route |
||
|
Jinwoo Hong
|
573537ecd4 |
feat(cli): make terminal close the canonical workspace teardown (#18073)
* fix(runtime): recover stale session owners and await retirement * fix(runtime): preserve session hydration and smoke compatibility * test(runtime): cover empty and unindexed session owners * feat(cli): make terminal close the canonical workspace teardown * fix(preload): align ssh termination result type * test(runtime): assert folder hydration owner * fix(runtime): fence legacy terminal stop by worktree host * fix(preload): reconcile ssh result import with main * fix(runtime): keep same-id sibling hosts out of workspace close The stale-owner fallback in the session controller re-routed any worktree whose catalog partition had no tabs to whichever other partition held tabs. Only `runtime:` environment ids rotate across relay restarts; `repoId::path` legitimately repeats across hosts, so an SSH workspace close could retire the local copy's tabs and resume records, or flip owners mid-close and strand the SSH PTY. Restrict the fallback to runtime hosts, and pin the session partition once per workspace close so record clearing targets the partition that owned the tabs. * test(runtime): give the cross-host close fixture a real resume record * fix(preload): take main's ssh-bridge import order so the merge stays duplicate-free |
||
|
Jinwoo Hong
|
d1350735ef | fix(orchestration): explain invalid send message types (#17487) | ||
  OrcaWinandm4air
|
a4bf9bb1e8 |
fix(settings): allow Escape to close from controls (#17516)
docs(orchestration): clarify terminal worktree selection Co-authored-by: m4air <m4air@m4airs-Air.localdomain> |
||
|
Brennan Benson
|
cc6b600e21 |
Fix orchestration CLI recovery, settled-Dispatch mail, and guide defects (#16919)
* Fix orchestration CLI recovery, settled-Dispatch mail, and guide defects Five reported orchestration CLI defects, verified individually before fixing. Two were real code defects, one was a docs error, one was correct as-is, and one was correct on both ends except for its recovery wording. - Mail addressed to a settled `dispatch:<id>` was accepted and silently dropped. Local sends bypassed the settlement check the federated branch already had, so the caller was told success for a delivery no worker would ever read. Reject with `dispatch_inactive` and name the Run mailbox to use instead. - A lost mutation response offered no read-only way to ask whether it took effect. `--retry-request` does dedupe correctly, but the recovery guidance emitted a query command only when the payload carried a dispatch id, which is exactly what a lost response lacks. Add read-only `orca orchestration request-show --request <id>` over the durable receipt ledger, and always emit a read-only step before the keyed retry. - The bundled `orca-cli` guide documented `check --unread --inject`, a flag the parser rejects. Correct it to `--format` and add a ratchet that runs every orchestration invocation in the bundled guides through the real CLI parser. - `check --json` is one stdout document and its keepalives are stderr-only; the reported `Extra data: line 2` came from merging the streams. Document the contract rather than changing the wire. - A rejected lifecycle message is loud on both ends already, but the rejection never named the flag that supplies the missing capability. Name it. * Harden orchestration mutation recovery guidance |
||
|
Brennan BensonandMerge Sim
|
b8d5b0486e |
Fix opening HTTPS URLs from headless runtimes (#17467)
* fix(runtime): open headless browser URLs on paired client * fix(pty): tolerate runtimes without browser relay probe * fix(browser): require automation-capable client host * fix(browser): map client URL opener in sidecar --------- Co-authored-by: Merge Sim <sim@local> |
||
|
Brennan Benson
|
419e3b4496 |
Fix terminal reads that flatten composer drafts into output (#16711)
* fix(terminal): separate composer drafts from read output Rendered screen reads treated cursor-line suggestion overlays as PTY output. Detect composer-owned text from cell attributes and cursor context, remove it from tail, and expose it as structured draft metadata. * fix(terminal): handle wrapped composer overlays * fix(terminal): preserve draft wrapping and tail alignment * fix(terminal): preserve composer wrap boundaries * fix(terminal): preserve draft continuations with middle dots * fix(terminal): recognize configurable Codex status lines |
||
 erish
|
5bcbafff53 |
docs(cli): document worktree rm branch cleanup (#16167)
Document that Git worktree removal may also delete the checked-out local branch, while clarifying that --force does not force branch deletion and that Orca retains branches whose changes cannot be proven merged. |
||
|
Brennan Benson
|
3fca1d1648 |
fix(linear): unbound list-issues by default, surface truncation, bind cursor workspace (#15824)
Fixes STA-5076. list-issues capped at 50 by default and hard-clamped at 250, with hasMore buried under result.meta and no stderr warning for --json, so a page that stopped early read as a complete answer. Omitting --limit now walks Linear's pages until they run out (meta.limit is null), and --limit <n> is the only cap, paging past Linear's 250-per-request maximum to reach it. result.truncated sits next to result.issues and is set only when a cap actually held results back; human output prints "truncated: showing N". The read still has to fit the CLI's 60s RPC budget, so a 20s wall-clock deadline and a 200-page ceiling stop the walk early and report truncated with a continuation cursor rather than failing the command. Also: - issued --cursor values bind the resolved workspace, so call -> nextCursor -> call works without --workspace; raw Linear cursors still need one and now carry nextSteps - issued cursors whose payload smuggles back `all` or an empty workspace are rejected at decode, since either would widen the read past the bound workspace - JSON issue rows carry priorityLabel (none/urgent/high/medium/low), matching orca linear priority set - truncated and priorityLabel are optional on the wire, so a host that predates either is not read as "complete"; readers fall back to meta.hasMore - the truncation line prints the rows actually rendered, so a remote result with no meta.returned cannot print "showing undefined" |
||
|
Neil
|
ef096d539d |
fix(terminal): refuse a cursor on a screen read, and correct the source docs (#15563)
Review follow-up on #15380. The RPC accepted `cursor` and `screen` together. The CLI refuses the pair, but terminal.read is reachable without it, and honoring both answered with rendered lines carrying the stream's pagination metadata — two frames of reference in one payload, which is the confusion `source` exists to remove. The guard beside it, withVisibleSnapshotFallback, already declines to substitute rendered lines when a cursor is present; the screen path now agrees, at the RPC boundary where every remote caller passes. Nothing could previously send both, since `screen` did not exist, so rejecting breaks no existing caller. The command notes and the runtime comment both still described the fallback as `source: stream`, left over from renaming that value to `screen-unavailable` during implementation. The spec text is surfaced through `orca help` and the agent-context schema, so a caller following it would test for a value the code never emits. Both now describe all four states, including that an absent source means the host predates the field. |
||
|
Neil
|
9d1dfc314f |
fix(cli): resolve host names across both kinds, and stop ssh: answering empty (#15449)
* fix(cli): resolve host names across both kinds, and stop ssh: answering empty `--host ssh:<id>` was never validated. An unknown target filtered to nothing and returned ok:true with an empty list — the same silent wrong-machine answer that unknown `runtime:` ids gave before they were rejected. And because SSH target ids are machine-generated (`ssh-<timestamp>-<random>`) while the name anyone actually knows is the label, this fired on the ordinary spelling rather than a rare typo: every human-typed SSH name missed. The two kinds of remote machine are also reached on different axes. A paired Orca server is a connection (`--environment <name>`); an SSH target is a machine the connected host reaches (`--host ssh:<id>`). A caller only knows "the machine called X", so naming X on the wrong axis was the common failure and produced either an empty answer or a dead-end "unknown environment". Now: `ssh:` resolves labels as well as ids and rejects an unknown target with the known ones listed; `runtime:` accepts the environment name as well as its id, matching --environment, and canonicalizes to the id so stored host ids still compare; and when a name misses on one axis but exists on the other, the error says which and gives the exact flag. Candidates ride along in error.data so an agent can recover without parsing prose. `orca host list` is the discovery surface that was missing entirely — nothing in the CLI listed SSH targets, so a caller told to use one had nowhere to look. It prints this machine, the SSH targets registered on the connected host, and the paired servers, each with the selector to use. * fix(cli): give --environment the same cross-kind hint, and validate the ssh host on setup-create Two gaps a follow-up survey found in the first pass. `--environment openclaw` still dead-ended with a bare "Unknown environment" while an SSH target by that name sat right there — the inverse of the case just fixed, and the direction the report actually hit. The store's own error cannot carry the hint: translateStoreError forwards code and message and drops data. So the selector is resolved before the client is built, where the payload survives. Only the explicit flag is asserted eagerly; an ambient ORCA_ENVIRONMENT stays lazy, because failing local-only commands over stale background config would be a regression. `project setup-create` records independent metadata and, unlike the other setup paths, is not covered by the runtime's ssh rejection — so an unknown target persisted a row pointing at a machine that does not exist. It now resolves the host. `local` and `runtime:` still pass through untouched: this is also the provisioning path, where a runtime host legitimately may not exist yet when its metadata is written. `setup-existing-folder` and `setup-clone` deliberately keep the unresolved id. The runtime rejects every ssh host for those operations regardless of whether it exists, so resolving first would answer "no such target" and imply the command would have worked with the right id. * fix(cli): refuse an ambiguous host name instead of resolving the first match Name lookup took the first match while the environment store itself refuses an ambiguous name rather than guessing. That put the guess back, in the selector whose entire purpose is to stop a command reaching a machine the caller did not choose — and it applied to both spellings: two SSH targets sharing a label, and two paired servers sharing a name. Both now resolve to nothing and report every candidate with its id, so the caller picks. An exact id still resolves past a colliding name, since an id is never ambiguous. Also pins the property that makes accepting a name safe at all: `runtime:<id>` is a persisted token that lands in ProjectHostSetup.hostId and is embedded in generated setup ids, so the name is canonicalized to the id before anything downstream sees it. A test now asserts a name never reaches the wire. * fix(cli): fall back to the older ssh listing so an old host is not read as having no targets Hosts predating ssh.listTargetSummaries still answer ssh.listTargets, and both are served by the same summariser. Swallowing the method_not_found made such a host indistinguishable from one with no SSH targets registered, which would reject a target id that is valid there — a new-client/old-host regression on a path that previously passed the id through unvalidated. |
||
|
Neil
|
3ffab9a6b3 |
feat(terminal): read the rendered screen with terminal read --screen (STA-4792) (#15380)
* feat(terminal): read the rendered screen with `terminal read --screen` (STA-4792) `terminal read` returns accumulated pty output with escape sequences stripped. That is the right answer for "what happened over time" and the wrong one for "what is on screen": any program that repaints a line comes back as stacked fragments, so one `clear` typed key by key reads as `cclclecleaclear`, and a prompt that draws a space by moving the cursor loses it. Nothing in the output said which question had been answered, so it was used as rendering evidence and produced false conclusions. The runtime already knew how to render — it replays the byte stream through a headless emulator — but only as a fallback for blank reads, alternate screen, and never-attached ptys. A normal attached terminal never reached it. `--screen` asks for it directly. Every read now reports its source, which also surfaces the pre-existing snapshot fallback that until now swapped rendered lines into an ordinary read with no indication. `screen-unavailable` distinguishes "asked for a screen, none could be rendered, here is the stream" from a stream the caller asked for, and an absent source means the host predates the field. Because an older host strips the unknown param and answers with its ordinary read, `--screen` against one fails with that explanation rather than passing the stream off as a screen. `--screen` and `--cursor` are mutually exclusive: a screen is the current frame and has nothing behind it to page. * refactor(terminal): stamp the screen source where rendered lines enter the read Inferring it from tail array identity worked but made a load-bearing contract out of reference equality; any later path spreading the read would silently mislabel. Rendered lines only enter through one builder, so it stamps there and anything still unlabelled is the stream. |
||
|
Jinwoo Hong
|
79be5b7fde |
feat(orchestration): report a worker blocked on a human prompt (STA-4513, STA-3714) (#15261)
* feat(orchestration): report a worker blocked on a human prompt (STA-4513, STA-3714) A lane parked on an approval, trust, or permission prompt looked exactly like a lane that was thinking or inside a long tool call. On origin/main, driving a real cursor-agent through Orca: surface running `sleep 60` awaiting approval worktree ps agents[].state working working terminal show / list no such field no such field terminal wait --for tui-idle satisfied: true satisfied: true worker-show no agent state no agent state The runtime already fuses hook state, OSC title, and matched prompt text into a `permission` verdict inside getTerminalAgentStatus — it was reachable only from the renderer, and it was blind to cursor-agent approvals. Two gaps, one boundary. Exposure: getTerminalInteractiveWait publishes that same fusion, minus the async foreground probe, as `agentWait` on `terminal show` and on `worker-show`'s observation. It carries the evidence that proved the wait (hook, prompt-text, or title) so a coordinator can weigh it. Null means no proof; a missing field means the host predates it — absence is never read as "not waiting". Detection: cursor-agent's hook set has no approval event and beforeShellExecution fires identically for auto-allowed commands, so its rendered menu is the only authority. Matched on the key-bound choices rather than the prose, requiring two, and self-clearing when the follow-up input line returns. Its live spinner title is exempted from the staleness rule that clears startup modals, because cursor keeps spinning while it waits. Falls out of routing it through the shared verdict: `dispatch --inject` into a cursor pane on an approval now refuses with agent_prompt_blocked instead of typing the preamble into the dialog. Fixtures are captured verbatim from cursor-agent 2026.08.11-e8db854 driven through Orca; the same case matrix was replayed live against a built runtime. terminal list stays untouched: its rows would each need a full tail scan, and STA-4694 owns the one-call-per-run aggregate. * fix(orchestration): only call a Cursor approval live while it owns the screen Independent review found the approval detector trusted one dismissal string, so any later output that did not contain cursor's follow-up line left the menu reading as a live wait. Reproduced: a tail of the real menu followed by two lines of ordinary output returned agent-approval-prompt, which fails tui-idle and refuses prompt injection on a healthy lane. Replaced with the structural property the string was standing in for: a live dialog owns the bottom of the screen, so the last choice may sit at most one line above the end of the retained tail. That tolerates a status footer or a partial line mid-redraw without admitting scrollback, and it drops the vendor prose. Being bottom-of-screen is also the dating this reason needed, so it no longer requires waitBlockedAt. A tail restored from terminal history carries none, and a lane parked on a prompt emits no bytes — so before this, an Orca restart made exactly the lane both issues are about go quiet for good. The startup modals keep the timestamp rule: their text lingers in scrollback with nothing to say whether it was answered. Also from review: - worker-show and federationShow reuse the verdict showTerminal already computed rather than rescanning the tail, so the two can no longer disagree. - The worker-show test now drives a real runtime, real PTY tail, and the real detector; it previously mocked getTerminalInteractiveWait, so it would have passed with detection permanently returning null. - The guard claim is now asserted against the guard: a blocked pane rejects both assertTerminalAgentSendable and sendTerminalAgentPrompt, and a working pane still passes. - Added a non-local (connectionId) pane case, since the verdict is derived from retained tail and title state on every host. * fix(agent-status): stop a hook wait from outliving its agent A third reviewer caught that the hook branch proved agent ownership from the pane title alone, while the shared verdict it claimed to reuse also probes the foreground process. A shell that takes a pane back usually sets something like `user@host: ~/repo`, which no title rule recognizes, and a hook row stays fresh for AGENT_STATUS_STALE_AFTER_MS — so a dead agent could be reported as waiting on a human for half an hour. Hook evidence now goes through getTerminalAgentStatus, which is the only thing that can answer whether an agent still owns this PTY. The two prompt branches skip it: a matched prompt is on the pane's screen now, so it proves itself. That makes the probe cost fall exactly where correctness needs it, and getTerminalInteractiveWait async, which only showTerminal had to absorb. Also trims the comments the same reviewer flagged as longer than the repo's rule. * test(agent-status): pin that a dead pane stops reporting a human wait A fourth reviewer noted the approval menu sits at the bottom of a dead pane's tail forever, and that no test covered process exit with no trailing output. The snapshot already refuses an exited pane, and worker-show gates agentWait on proven identity — this pins both so neither can drift into reporting a worker that needs intervention as one that needs an answer. * fix(orchestration): never report an unchecked worker as not waiting Automated review caught that the three worker paths which return before the wait is ever evaluated — unattached, missing, and identity_changed — then had their undefined coerced to null by the emitters. A worker whose process was replaced was reported as `agentWait: null`, which reads as "Orca looked and nobody is waiting" when Orca never looked. That is the false negative this field exists to remove. The field is now emitted only when it was evaluated, so a present null is a claim about the pane and an absent one means nobody looked — because the host predates the field, or the worker's identity could not be verified. The CLI and the worker-show note say that rather than blaming an old host. Covered on the context-only path, where the regression test fails against the previous behavior; the supervised and federated emitters take the identical one-line change. Also trims the two test-file headers to one statement of purpose. * fix(agent-status): tighten the Cursor menu match and stop guessing on unknowns Fourth review round, three findings, each reproduced before acting. Matching each choice marker with an independent lastIndexOf let text outside the menu carry the anchor. An agent narrating "next time I'll suggest Run Everything" after the menu was answered pulled the match down to the bottom of the screen and revived it. The match is now confined to the last lines of the tail, and a choice is a line that ends in the key that picks it — prose writes the same words but not the same shape. The one line of slack under the dialog went with it. It was a guess; every capture of a live dialog ends on its last choice, and one line is exactly enough room for that narration. A redraw caught mid-flight now reads as no wait until the next poll, which is the safe way to be wrong. The hook branch awaited a foreground probe that reaches a PTY controller which may be a remote host, so a wedged probe stalled every caller of showTerminal — a path that never probed before. It is bounded now, and a timeout leaves the wait unevaluated rather than claiming there is none. Which is the same distinction the previous commit only fixed one level up: getTerminalInteractiveWait itself turned an unreadable pane into `null`, so showTerminal published "looked, nobody waiting" for a pane it could not read. It returns undefined there, showTerminal omits the key, and worker-show's text output prints unknown rather than rendering it the same as none. * fix(agent-status): bound the wedged probe's cost and stop matching prose keys Fifth review round. No correctness defects in the shipped behaviour this time; two robustness holes and the documentation of the contract. The bounded probe abandoned the wait but not the request, so a coordinator watching a wedged remote host added one live probe on every poll. It is single-flighted per PTY now, the way the leaf-absence probe already is. The trailing-key rule that separates a menu row from the agent narrating a choice was written as a character class, and any lowercase run up to twelve characters satisfied it — "…suggest Run Everything (as before)" passed. Spelled out as key names instead, which also lets the glyph forms of those keys through. The contract wording said an absent agentWait meant an old host or an unverifiable identity. It also covers an unreadable pane and a probe that did not answer, and a reader diagnosing an old peer from that would be wrong. Corrected on the type, the worker-show note, and in docs/reference/remote-wire-compatibility.md, which had no entry for a field whose absent and null states mean different things. Also strengthens the worker-show agreement test, which compared the terminal and observation payloads without asserting either held the expected wait, so it passed when both were absent. |
||
|
Neil
|
4cc7e7859a |
fix(cli): route --host runtime:<id> to that server instead of answering locally (#15364)
* fix(cli): route --host runtime:<id> to that server instead of answering locally `--host` was only ever a local filter over whatever runtime the CLI happened to connect to, so `--host runtime:<id>` silently answered for (and mutated) the local machine. A real environment id and a made-up one were indistinguishable: both returned ok:true with an empty list and the local runtimeId in _meta, and `project setup-clone --host runtime:<id>` cloned into the caller's own machine. Resolve the flag before the client is built: unparseable host ids and runtime ids that no paired environment owns are rejected, and a known runtime id selects that environment as the connection (conflicting with --pairing-code or a different --environment is an error). Once routed, a host filter also accepts the runtime's own `local`-stamped rows, since both spellings name the machine we are now talking to. * fix(cli): close --host routing gaps found in review - Conflict-check an ambient ORCA_ENVIRONMENT, not just the --environment flag. `ORCA_ENVIRONMENT=staging orca ... --host runtime:<prod-id>` silently routed to prod while the flag spelling errored. An ambient pairing code still loses to the explicit flag, because it cannot be resolved to an id to compare. - Attach the known environment ids to the unknown-id error as `error.data`, so a --json consumer can retry without parsing prose, and say outright that runtime:<id> matches ids only and never environment names. - Fix four command examples that documented `--host runtime:gpu`. `gpu` is an environment name, so every one of them would now be rejected; use an id. - Cover the routed connection on `worktree create` and `automations create` (the mutating paths), the `--environment X --host local` filter-only case, and assert error.code/error.data rather than only substrings. * test(cli): pin execution-host-flag to the deferred error-class import index.ts now loads execution-host-flag.ts on every invocation, making it the sixth module on the --help path. It imports RuntimeClientError from ./runtime/types today, but nothing enforced that; switching it to the barrel would silently drag zod/ws/tweetnacl back onto --help, which is exactly what this guard exists to prevent. Verified the assertion fails when the import is flipped to the barrel. |
||
|
Jinwoo Hong
|
0bedeea642 | fix(orchestration): expose unsupervised dispatch lanes (#15105) | ||
|
Jinwoo Hong
|
fa9b20cb41 | feat(skills): reland private bundle sharing safely (#14934) | ||
 Jinjing
|
763b1febeb |
Revert "feat(skills): add private bundle sharing (#14401)" (#14913)
This reverts commit
|
||
 Jinwoo HongandE2E Test
|
757fae28d7 |
feat(skills): add private bundle sharing (#14401)
Co-authored-by: E2E Test <e2e@test.local> |
||
|
Brennan Benson
|
78d5920446 |
fix(orchestration-cli): point dropped mutations at --retry-request (#14586)
* fix(orchestration-cli): guide dropped mutations to idempotent retry * test(orchestration-cli): preserve read-only drop message * fix(orchestration): harden mutation replay identity * fix(orchestration): preserve replay across remints * fix(orchestration): defer local mutation identity |
||
|
Brennan Benson
|
537864a248 |
Fix Codex hook trust before manual shell launches (#14326)
* fix codex hook trust before shell launch * fix packaged cli preflight dependency * fix codex shell preflight safety * fix Codex shell preflight settings and startup safety |
||
 Jinwoo HongandJinwoo-H
|
45c1cb979a |
fix(orchestration): release context-only dispatches (#13376)
* fix(orchestration): release context-only dispatches Refs #13005 * test(orchestration): align PTY readiness timeout --------- Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com> |
||
|
Jinwoo Hong
|
c991bb27d3 | Add account-backed artifact sharing (#13012) | ||
|
Jinwoo Hong
|
b0ba51831c | Add per-worker model and effort overrides (#12851) | ||
|
Brennan Benson
|
0ce108d935 |
fix(browser): add native-UA session profiles (#12608)
* fix(browser): add native-UA session profiles * test(browser): add Google sign-in UA probe * fix(browser): preserve native profile UA identity |
||
|
Brennan Benson
|
39c3c58d55 |
perf(runtime): gate terminal.list visual layouts (#12450)
* perf(runtime): gate terminal.list visual layouts and stop the false writable claim visualLayouts is ~31% of a large terminal.list payload (44,208 B of 137,412 B on a live 134-terminal remote runtime) and has exactly one consumer: the human-readable CLI formatter. Gate it behind an includeVisualLayouts request param that defaults to included, so pre-flag clients are unaffected, and have every --json/internal caller opt out. Also drop the record-backed builder's writable, which was a verbatim copy of connected. terminal.show now states writability explicitly as exactly what terminal.send's PTY gate enforces. * test(runtime): type the payload-size fixture arrays for tsc * fix(runtime): preserve terminal list compatibility * test(runtime): guard terminal list optimization * fix(cli): preserve agent access to terminal layouts |
||
 Brennan BensonandOrcaWin
|
f4b2b782b5 |
feat(orchestration): coordinator-driven release of settled worker terminals (STA-905) (#12355)
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
Jinjing
|
1c8908b791 |
Fix orchestration gate authorization to scope by Run binding (#11802)
* fix(orchestration): gate methods route calls to the caller's Run with `f Gates are Run-scoped state; every gate command now resolves the caller's active Run (via pane binding or explicit --from flag) and authorizes within that Run's scope. Settled adopted work no longer requires --takeover-legacy, and the legacy coordinator fence respects both binding-based and attestation-based proof of authority. * fix(orchestration): gate methods route calls to the caller's Run with at Gate and run methods now verify that declared terminal handles match the caller's attested identity, preventing spoofing of other coordinators. Extracted shared `resolveRunScope` to enforce one authorization rule across all orchestration mutations. Added comprehensive regression tests for #11745. |
||
  
|
650dd48ec9 |
feat(cli): add orca account add / account list for headless hosts (Claude + Codex) (#9177)
* feat(cli): add `orca account add` / `account list` for headless hosts The desktop "Add account" UI is disabled when the renderer drives a remote runtime (isRemoteAccountScope === kind:'environment'), so a headless server reached from a remote desktop/web client has no way to register managed Claude accounts. Add a host-local CLI path that reuses the existing capture logic: - ClaudeAccountService.addAccountFromConfigDir(): register a managed account by capturing credentials from an already-authenticated CLAUDE_CONFIG_DIR instead of spawning the interactive browser login (extracted persist/rollback helpers shared with the existing add flow) - RPC accounts.addClaudeFromConfigDir, bridged via OrcaRuntime; rejected for mobile device tokens (host-local only) - `orca account add` runs `claude login` in the user's own terminal into a temp CLAUDE_CONFIG_DIR, then registers it via the local runtime; `orca account list` lists managed accounts Switching (select) already works from a remote client; only adding was blocked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cli): support Codex in `orca account add` / `account list` Mirror the Claude headless-account CLI for Codex: - CodexAccountService.addAccountFromHome(): register a managed Codex account by importing auth.json from an already-authenticated CODEX_HOME, reusing a shared persist helper extracted from doAddAccount (no interactive login spawned here) - RPC accounts.addCodexFromHome + OrcaRuntime.addCodexAccountFromHome bridge, rejected for mobile device tokens (host-local only) - `orca account add --agent claude|codex` (default claude); `orca account list` now renders both Claude and Codex managed-account blocks Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: cover headless account-add capture paths (Claude + Codex) - ClaudeAccountService.addAccountFromConfigDir: registers a managed account by capturing an authenticated CLAUDE_CONFIG_DIR; rejects and rolls back when the dir has no .credentials.json - CodexAccountService.addAccountFromHome: imports auth.json from an authenticated CODEX_HOME into a managed account; rejects when auth.json is missing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: address CodeRabbit review on headless account-add flows - CLI login spawn uses a shell on Windows so `.cmd` agent shims resolve without ENOENT (args are fixed literals, no injection risk) - Claude capture skips the `.credentials.json` precheck on macOS, where creds live in the Keychain and captureAuthFromConfigDir reads them - Claude add rollback is best-effort: a failed rematerialization no longer skips managed-auth cleanup or masks the original add error - Codex persist restores the prior account/selection if a post-write sync or rate-limit refresh fails, so a failure can't leave a dangling managed account - Codex sync passes the account's selection target (correct runtime for WSL) - Add JSDoc to the new public service methods and CLI functions Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(cli): harden headless account capture * fix(cli): correct account command flag surface and interrupt cleanup - `account` commands no longer accept or advertise the browser `--page` flag; `supportsBrowserPageFlag` allow-listed them by omission, so `orca account list --page x` was silently accepted and `--help` rendered a browser-only option - account specs declare GLOBAL_FLAGS, so `--help`/`--json` render in the Options block like every other command - `--agent` on `account add` documents the account provider instead of the terminal TUI-agent meaning inherited from the shared flag table - a SIGINT/SIGTERM during the interactive login now removes the temp login dir (and restores the macOS Keychain item) before exiting 130; Node terminates without unwinding `finally`, which stranded live OAuth credentials on disk * perf(cli): stop `account list` forcing a provider usage refresh `accounts.list` awaited refreshAccountsForMobile(), which runs fetchAll({ force: true }) — bypassing both the poll throttle and the per-provider Retry-After gate — then O(N) serial per-account round trips. `orca account list` renders only emails and the active ids, so all of that work was discarded. The RPC now takes `refreshUsage` (default true, so mobile and web keep the forced lane) and the CLI opts out. Older hosts declare `params: null` and ignore the field, so a newer CLI degrades to the previous behavior rather than failing. Also documents on `account list` that `--environment` does not retarget it, matching the host-local behavior of shouldIgnoreRemoteSelection. * fix(cli): survive repeated and hangup signals during account add withInterruptCleanup latched cleanup behind a boolean, so a second signal got an already-resolved promise and its process.exit fired while the first cleanup was still inside a Keychain call (3s each) — the temp dir's OAuth credentials and the swapped macOS Keychain item both survived. Memoize the cleanup promise so every signal awaits the same run, and register with `on` instead of `once` so a second Ctrl-C cannot fall through to Node's terminate-immediately default mid-cleanup. Handle SIGHUP too. This flow exists for headless/SSH hosts, where the most likely interrupt is the connection dropping, which hangs up the login's terminal and previously ran no cleanup at all. Warn when the interrupt lands after sign-in completed: the runtime finishes the add independently of this process, so exiting 130 silently would tell the user it was cancelled when the account may exist. Reject a valueless `--agent`; the parser turns it into boolean true, which silently ran a full OAuth login for Claude when the user asked for another provider. Also lock two behaviors the refactor changed but left uncovered: a WSL Codex add must sync the WSL runtime lane rather than the default host lane, and rename the account-spec help test to describe the Options block it actually asserts rather than the usage string it never reads. * fix(build): bundle the main modules the account CLI imports electron-vite cleans out/main and emits only its declared entries, and `build:desktop` runs it after `build:cli`, so the tsc-emitted copies of `claude-accounts/keychain`, `codex-cli/command` and `win32-utils` were deleted before packaging. Both `orca account add` and `orca account list` then died at require time with "Cannot find module '../../main/claude-accounts/keychain'" — reproduced against a real `--serve` host. `agent-hooks/managed-agent-hook-controls` already carried an entry for exactly this reason; these three were missing. Adds a parity test so any future CLI import of a `src/main` module fails in CI rather than at a user's shell after packaging. * test: cover the desktop add-path behavior this PR changes Both changes ride in the persist/rollback helpers the existing GUI add flow shares with the new headless path, and neither had coverage: - Claude: rollbackAddAccount now guards forceMaterializeCurrentSelection- ForRollback, so a rejecting rematerialization no longer replaces the real add error nor skips safeRemoveManagedAuth. Asserts the original error surfaces and the throwaway auth dir is gone. - Codex: the desktop add now passes the account's selection target to syncForCurrentSelection, matching reauthenticate and select. Asserts the host target alongside the existing WSL assertion. Both fail when the corresponding change is reverted. * fix(cli): close the remaining account-add interrupt and preflight gaps The round-1 interrupt fix detached the signal handlers before running the finally-path cleanup, so the very window it was meant to protect — the two serial 3s `security` calls plus rmSync on the success/error path — was still covered only by Node's terminate-immediately default. Both review lanes reproduced it independently. Await cleanup first, detach in a nested finally, and stop a cleanup failure from replacing the error that actually explains why the add failed. Do not burn the interactive login when the runtime is unreachable. The RuntimeClient is lazily constructed and the first call was the registration RPC itself, so "Requires the Orca runtime to be running" was discovered only after the user completed a full OAuth round trip. Preflight with the now-cheap `accounts.list { refreshUsage: false }`. Reject `--environment` / `--pairing-code` on `account add`. shouldIgnoreRemoteSelection pins account commands to the local runtime, so `orca account add --environment homelab` silently registered the account on the laptop instead of the headless host it names. Survive a daemon that cannot spawn `claude`. `allowFailure` is honored in onClose but not onError, and unlike the GUI flow nothing has run `claude` in the daemon before this point — so a launchd/systemd daemon with a minimal PATH hard-failed an add the user had already signed in for, even though identity resolves fine from the config dir's oauthAccount. Also align the `--agent` help description with the global flag column. * fix(cli): reject runtime selectors on `account list` too `orca account list --environment homelab` was accepted and silently listed the LOCAL machine's accounts, because shouldIgnoreRemoteSelection pins account commands to the local runtime. Documenting that in --help does not reach someone who already typed the flag, and answering with the wrong host's accounts is the specific wrong answer they would act on. `account add` already errors; this makes the new command group internally consistent. The other groups in shouldIgnoreRemoteSelection keep their existing silent-ignore behavior — changing those is not this PR's job. * test: harden account-add signal tests and cover cleanup failure - Identify the handler under test by set difference instead of `process.listeners(sig).at(-1)`. Vitest installs its own once-wrapped SIGINT teardown, so the positional lookup could grab the wrong listener; the helper also asserts exactly one new listener was added. - Mock rmSync while keeping the real implementation by default, so the temp-dir assertions elsewhere stay honest. - Cover that a cleanup failure in the `finally` does not replace the error explaining why the add failed. Fails when that guard is removed. Completes the review loop's final round; the loop died on an API error before it could commit this, and its `import()` type annotation would have failed oxlint. * fix(cli): harden interactive account add * test(cli): make account cancellation coverage portable * fix(cli): preserve merged skills runtime modules --------- Co-authored-by: Dominik <marketing@gavaplast.sk> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Brennan Benson <79079362+brennanb2025@users.noreply.github.com> |
||
  Sebastián Castañoandscastanoh21
|
676ef7fab8 |
feat(cli): add orca skills install and orca skills update for headless skill setup (#9201)
Adds `orca skills install` and `orca skills update` so skills can be set up without the GUI — SSH hosts, containers, CI. Previously `orca skills` had only `list` and `get`, so there was no headless path. **Agent targeting is scoped explicitly rather than delegated to detection.** The `skills` CLI decides which agents to install into, and with `-y` and zero detected agents it takes `targetAgents = validAgents` — all ~75. That is not a corner case for a headless CLI: a fresh SSH box or container with no agent installed is the normal starting state. Measured on a bare host, the unscoped command created **52 top-level agent directories and 54 junctions** (one real payload in `~/.agents/skills`, the rest links) on Windows, and 52/53 on macOS. The CLI now passes `--agent` derived from Orca's own detection, mapped to the `skills` key namespace, plus `universal`. Supplying `--agent` makes `runAdd` use it directly and never call `detectInstalledAgents()`, so the fan-out branch is unreachable. On a bare host it now refuses with `No coding agent detected on this host` and exit 1, creating nothing. Same command with scoping: **1 directory, 0 junctions.** `universal` alone would under-install — Claude Code is not in that set, and 19 of 28 mapped keys write agent-private homes `universal` never touches. `--agent '*'` is the bug itself. The mapping is hedged three ways: `null` for any agent whose key could not be confirmed, `satisfies Record<TuiAgent, …>` so a new Orca agent is a compile error, and a test pinning every mapped key against the CLI's own valid list. Fixed during review — two holes that each restored the full fan-out through a different door: - `--agent ','` trimmed to nothing, which skipped the refusal *and* emitted no `--agent`. - `--agent -y` passed an emptiness check, and the vendor CLI silently drops `-`-leading values, re-emptying its list. The real invariant is argument *shape*, not emptiness, and it is now enforced at the choke point in `buildAgentFeatureSkillInstallArgs`, so no caller can emit `-y` without a usable target. `*` remains allowed — asking for every agent explicitly is a choice, not an accident. Verified with 51 hostile inputs through the built binary, each recorded argv replayed through the vendor's own parser. Also fixed: the `ORCA_CLI_CWD` refusal now runs before target resolution (it was quoting the wrong host's agent list), and `--dry-run` is refused in a forwarded shell rather than printing a command naming the wrong machine. Validated on a real Windows host across PowerShell 7, PowerShell 5.1, cmd.exe and Git Bash: `.cmd` shims route through `cmd.exe` and `.exe` shims spawn directly (proved with instrumented shims, not inferred), the ENOENT path produces an actionable error rather than a silent failure, and `skills update` genuinely restores a corrupted skill byte-for-byte. Known, not addressed here — both upstream behaviours this only forwards: a partial install failure exits 0, and "no installed skills found" exits 0. Both are invisible to the headless callers this feature exists for. Co-authored-by: scastanoh21 <scastanoh21@gmail.com> |
||
|
64a1269409 |
perf(orchestration): bound mutation ledger and run pages (#11432)
* perf(orchestration): bound mutation ledger and run pages Co-authored-by: Orca <help@stably.ai> * fix(orchestration): close retention pagination gaps * fix(orchestration): preserve unpaginated run listing Co-authored-by: Orca <help@stably.ai> * fix(orchestration): reject malformed run cursors --------- Co-authored-by: Orca <help@stably.ai> Co-authored-by: Jinjing <6427696+AmethystLiang@users.noreply.github.com> |
||
|
Neil
|
d0f341ad69 |
fix(computer-use): make modifier clicks interruption-safe (#11451)
* fix(computer-use): make modifier clicks interruption-safe * fix(computer-use): pace modified Windows multiclicks * fix(computer-use): address modifier safety review |
||
|
OrcaWinandOrcaWin
|
363e478909 |
fix(orchestration): preserve active workers across updates (#11271)
* fix(orchestration): preserve active workers across updates * test(ssh): model absent legacy adoption * test(orchestration): align compatibility contracts * fix(windows): escape updater PowerShell booleans * fix(windows): restore stock uninstall process check * fix(orchestration): keep recovery off renderer startup barrier * fix(orchestration): harden legacy recovery migration * fix(orchestration): close recovery review gaps * fix(orchestration): complete legacy worker cutover recovery * fix(orchestration): preserve legacy workers across updates --------- Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com> |
||
|
Brennan Benson
|
3baffb49ff |
fix(runtime): refuse SSH hosts in project setup instead of acting locally (#10799)
* fix(runtime): refuse SSH hosts in project setup instead of acting locally projectHostSetup.clone and .setupExistingFolder threaded executionHostId all the way down but never used it for routing: cloneRepo runs a local mkdir plus a local gitSpawn, and addRepo probes the path with existsSync/statSync. An `ssh:` host therefore cloned and validated on the *local* machine and then registered the result as living on the SSH host. It only failed loudly here because the remote path did not exist locally. With a plausible destination the clone succeeds and writes a setup record pointing at the wrong machine. Nothing legitimate sends `ssh:` to these RPCs: the renderer maps every ssh host (including ephemeral-VM `ssh:runtime-ssh-*`) to the desktop IPC path, which dispatches to addRemoteRepoFromPath/cloneRemoteRepo, and the IPC handler symmetrically rejects `runtime:`. Only the CLI can reach here with `ssh:`. Fail closed until the RPC learns to route through the SSH providers. * test(runtime): make the SSH guard test observe the corruption it names The test asserted `gitSpawn` was never called and no repo was registered, but neither assertion could fail. `/home/brennan` is unwritable on macOS, so the pre-guard clone died at `mkdir` before reaching `gitSpawn`, and `/home/brennan/orca` failed `isGitRepo` before reaching `addRepo` — the exact side effects under test were unreachable either way. `rejects.toThrow` also aborted the test before those lines ran. Use a real temp destination and a real temp git repo, await both calls via `.catch`, and assert the side effects before the wording. With the guard disabled the test now fails on `gitSpawn` being called once with a real `git clone`, and on a repo registered stamped `executionHostId: 'ssh:openclaw'` — the silent local-clone-recorded-as-remote defect itself. `gitSpawn` is stubbed so a regression records the call instead of hitting the network. Also document the SSH restriction on `project setup-existing-folder`, which the guard now rejects. `setup-clone` already carried that note; its sibling did not. |
||
|
OrcaWin
|
cd05f2ff93 | Implement robust orchestration primitives and connected-server workers (#9925) | ||
 Rod Boev
|
108a2ad41b | fix(cli): relativize absolute --path for file open and file diff before the runtime RPC (#9429) (#9824) | ||
 Jinjingand5Hyeons
|
4a9affd6e5 |
fix(emulator): iOS ax via plain-JSON serve-sim helper (supersedes #10007) (#10029)
* Revert "Enable accessibility tree (`ax`) command on iOS emulator sessions (#10007)"
This reverts commit
|
||
|
Jinjing
|
43ae014a64 |
Enable accessibility tree (ax) command on iOS emulator sessions (#10007)
* Enable accessibility tree (`ax`) command on iOS emulator sessions Fetch the accessibility tree from serve-sim's /ax endpoint, which requires an active session but provides the same UI snapshot capability as Android's uiautomator output. Derive the endpoint from the stream URL when not explicitly provided by the helper, and route through the bridge to pass session context to the backend. * Add ax command routing and backend integration tests Tests verify accessibility tree routes through EmulatorBridge, Android backend ignores iOS-specific ax URLs, and ax endpoints are derived from serve-sim stream URLs. |
||
|
OrcaWin
|
34c160442f | Fix headless Linux serve pairing readiness (#9785) | ||
|
Brennan Benson
|
f1c84d3858 | refactor(cli): split oversized command modules (#9775) | ||
|
Brennan Benson
|
a10a2ba53c |
feat(linear): add MCP-style save issue (#9670)
* feat(linear): add MCP-style save issue * fix(linear): harden save issue parity * fix(linear): close save issue contract gaps * docs(linear): bundle project discovery with save issue |
||
|
Brennan Benson
|
87af1c8673 |
feat(linear): add complete issue relations (#9674)
* feat(linear): add complete issue relations * fix(linear): harden relation reads and writes * fix(linear): classify ambiguous relation writes |
||
|
Brennan Benson
|
42a4f017b4 | feat(linear): add MCP-compatible issue listing (#9672) | ||
|
Brennan Benson
|
be066fe8e9 | feat(linear): expose issue activity history (#9667) | ||
|
Jinwoo HongandOrca
|
319ae4e9ea |
fix(terminal): make whole-tab close durable (#8958)
Co-authored-by: Orca <help@stably.ai> |
||
|
Brennan Benson
|
31f643ca42 |
Add version-matched skill guides to the CLI (#8624)
* Add version-matched bundled skill guides * Clarify skill freshness rollout PRs * Add canonical skills show alias * fix(skills): address guide review feedback * fix(skills): make guide commands cross-platform * fix(skills): apply the ORCA convention to the emulator guides Review follow-up: the emulator guides still instructed literal `orca emulator ...` in sh fences with no Linux disambiguation, so on unmanaged Linux they could launch the GNOME screen reader — the exact failure the executable-selection preamble prevents. Both emulator guides now carry the preamble and ORCA placeholder across fences, tables, and prose, and the cross-platform safety test covers all four converted guides. Also replaces computer-use's "unless a block names a shell" carve-out, which contradicted its own POSIX example, with the unconditional placeholder rule. |