mirror of
https://github.com/stablyai/orca.git
synced 2026-10-18 16:03:17 +00:00
Add orchestration runtime restart resilience design plan
Document the key reliability gaps and recommended solutions for handling runtime loss and app restart in orchestration workflows: - Treat message waits as replayable database queries with reconnect support - Close the read-before-response window via cursor/receipt protocol - Persist coordinator identity on dispatches for stable routing - Add startup reconciliation to preserve live workers and fail only definitive exits - Extend idempotency to mutations like worktree.create Recommends incremental approach: recovery semantics (I0), transport resume with replay (I1), identity-complete dispatches and startup reconciliation (I2), and durable idempotent mutations (I3). Status: Brainstorm/recommended design, not an implementation plan of record.
This commit is contained in:
Jinjing
@@ -0,0 +1,348 @@
|
||||
# Orchestration resilience across runtime loss and app restart
|
||||
|
||||
Status: brainstorm / recommended design, not an implementation plan of record
|
||||
|
||||
Date: 2026-07-17
|
||||
|
||||
## Executive summary
|
||||
|
||||
- Treat a blocking wait as a replayable database query, not as a durable socket or a durable in-memory waiter. `messageWaitersByHandle` should remain ephemeral.
|
||||
- Make `orchestration check --wait` reconnect across a runtime generation change, preserving its original recipient identity, filters, cursor, and overall deadline.
|
||||
- Close the current response-loss window: `check` marks rows read before its terminal RPC frame is known to have reached the client. Add a sequence cursor/receipt protocol so reconnects can replay a response that was committed but not observed.
|
||||
- Give lifecycle sends a client-generated event ID and retry only the ambiguous transport failures. The runtime should deduplicate and reconcile the lifecycle projection transactionally.
|
||||
- Persist the coordinator identity on every dispatch, including its pane key and execution-host scope. Exit escalation must target that dispatch owner, not only `getActiveCoordinatorRun()`.
|
||||
- Add a startup reconciler after terminal/provider hydration. Preserve a dispatch when its pane is live, fail it exactly once when exit is definitive, and leave it recoverable when SSH/WSL/provider state is unknown.
|
||||
- Resume or explicitly interrupt persisted `orchestration.run` rows on startup; never leave a `running` row with no in-memory `Coordinator` loop.
|
||||
- Extend idempotency to CLI mutations such as `worktree.create` with a durable operation journal. The existing 60-second in-memory `clientMutationId` cache is useful precedent but does not cover desktop CLI calls or process restart.
|
||||
- Ship the protocol and skill recovery playbook first, then stable routing/reconciliation, and defer a detached worker outbox until evidence shows the bounded retry plus startup reconciliation still loses completions.
|
||||
|
||||
## 1. Problem statement
|
||||
|
||||
The incident is not one failure called “runtime unavailable.” It exposes several different ambiguity windows that need different guarantees.
|
||||
|
||||
### 1.1 The wait connection is disposable, but the CLI treats it as the operation
|
||||
|
||||
`src/cli/runtime/transport.ts` rejects immediately with `runtime_unavailable` when the socket closes before a terminal frame. That is correct transport behavior: the client cannot invent a response. The problem is that `src/cli/handlers/orchestration.ts` makes one `client.call('orchestration.check', ...)` and has no recovery loop, so a coordinator interprets loss of one connection as loss of the supervision operation.
|
||||
|
||||
Server keepalives and the widened client timeout solve idle connection expiry; they cannot solve process death. The durable object is the SQLite inbox, not the socket and not `messageWaitersByHandle`.
|
||||
|
||||
### 1.2 A response can be lost after the message has already been consumed
|
||||
|
||||
The current `orchestration.check` handler in `src/main/runtime/rpc/methods/orchestration.ts` does this in order:
|
||||
|
||||
1. reads unread messages;
|
||||
2. reconciles lifecycle side effects;
|
||||
3. marks the rows read;
|
||||
4. returns to the RPC dispatcher, which later writes the terminal frame.
|
||||
|
||||
If the runtime dies between steps 3 and 4, the caller receives `runtime_unavailable`, but a retry of the normal unread query returns nothing. The task may already be correctly completed by lifecycle reconciliation, while the coordinator never observes the `worker_done` summary. Reconnecting only to the unread inbox is therefore necessary but insufficient.
|
||||
|
||||
### 1.3 A lifecycle send has two ambiguous restart windows
|
||||
|
||||
When a worker runs `orchestration send --type worker_done` during restart:
|
||||
|
||||
- the connection may fail before SQLite insertion, in which case no durable completion exists; or
|
||||
- insertion and reconciliation may commit, but the response may be lost, in which case retrying can insert a duplicate because message IDs are currently generated by the runtime.
|
||||
|
||||
The lifecycle projection is mostly idempotent after the first completion, but the duplicate message remains user-visible and there is no durable acknowledgement tied to a client event ID.
|
||||
|
||||
### 1.4 Process and orchestration lifecycles can diverge
|
||||
|
||||
Daemon-backed PTYs can survive an app relaunch, while update installation explicitly calls `killAllPty()`. If the worker survives, a restart must not burn a failure or redispatch over its uncommitted work. If the worker dies while the runtime is absent, `failActiveDispatchOnExit` never observes the exit and the persisted dispatch can remain `dispatched` indefinitely.
|
||||
|
||||
Heartbeat age cannot decide between these cases. A runtime outage creates a heartbeat gap even for a healthy worker, and SSH/provider unavailability creates an “unknown” state rather than proof of exit.
|
||||
|
||||
### 1.5 Stable authority exists, but stable routing is incomplete
|
||||
|
||||
Schema v6 records `dispatch_contexts.assignee_pane_key` and `messages.sender_pane_key`. `lifecycle-reconciliation.ts` correctly uses pane identity for completion/heartbeat authority after handle remint.
|
||||
|
||||
Recipient routing still uses `messages.to_handle`, coordinator runs store only `coordinator_handle`, and a dispatch does not record the coordinator that dispatched it. A reconnect should initially reuse the exact old handle so it can read the old inbox. Longer term, messages and dispatch ownership need a recipient/coordinator pane identity so a reminted handle can find the same durable inbox without relying on stale environment variables.
|
||||
|
||||
There is a related hole in `failActiveDispatchOnExit`: it looks up the active dispatch by the current handle, without passing the leaf pane key. An exit after handle remint can therefore miss a dispatch whose authority row still has the old handle.
|
||||
|
||||
### 1.6 Automated coordinator state can become a zombie
|
||||
|
||||
`orchestration.run` persists a `coordinator_runs` row and starts a module-scoped `activeCoordinator`. After process restart, the row can still say `running`, but the JavaScript loop is gone and the module reference is null. A new run is rejected because `getActiveCoordinatorRun()` sees the old row. Persisting a run row without a restart policy is worse than an explicitly interrupted run because it looks active while doing no work.
|
||||
|
||||
### 1.7 A mutation can succeed while its response fails
|
||||
|
||||
The observed `worktree create` result is a standard ambiguous commit: the worktree exists, but the CLI saw `runtime_unavailable`. Retrying an unkeyed multi-step mutation can create another branch/worktree or hit a confusing conflict.
|
||||
|
||||
There is already partial precedent: `worktree.create` accepts `clientMutationId`, advertises `worktree.create-idempotency.v1`, and uses a 60-second in-memory promise cache in `OrcaRuntimeService`. The desktop CLI handler does not send the token, and the cache itself disappears on runtime restart, so it does not cover this incident class.
|
||||
|
||||
## 2. Threat and failure matrix
|
||||
|
||||
| Failure | Current observable behavior | Main risk | Required invariant |
|
||||
| --- | --- | --- | --- |
|
||||
| Coordinator long-poll socket dies mid-wait because the app crashes or quits | Keepalives stop; socket close becomes immediate `runtime_unavailable`; the CLI exits | The coordinator abandons a healthy dispatch | A wait can reopen against the same durable inbox until its original deadline |
|
||||
| Runtime dies after `check` marks a message read but before the terminal response frame | Retry of unread inbox returns `{count:0}` | `worker_done` summary is silently missed even though the task may be completed | Replay is cursor/receipt based, not limited to `read = 0` |
|
||||
| Worker remains alive through runtime restart | Heartbeats stop during outage; handle may remint; dispatch stays persisted | False stale classification or duplicate redispatch over uncommitted changes | Pane-equivalent live worker preserves the exact active dispatch |
|
||||
| Worker dies with the app or while runtime is absent | No runtime exit callback runs | Dispatch remains stuck forever, or a later manual recovery guesses incorrectly | Startup reconciliation fails the exact dispatch once only after definitive exit |
|
||||
| `worker_done` starts while runtime is unavailable | CLI send fails before insertion | Completed work has no lifecycle record | Lifecycle send retries with the same event ID, or a later recovery conclusively fails/resumes the worker |
|
||||
| `worker_done` commits but its response is lost | Worker sees failure and may retry; runtime may insert a duplicate | Duplicate mail or worker stops without knowing completion was accepted | Duplicate event ID returns the original committed acceptance |
|
||||
| Manual coordinator uses `task-create` + `dispatch --inject` | Worker exit escalation is sent only if an active coordinator run exists | Manual coordinator gets no exit notification | Dispatch records and notifies its own coordinator identity |
|
||||
| `orchestration.run` is active during restart | Persisted row remains `running`; loop/reference disappears | Run becomes an unrecoverable zombie and blocks a new run | Startup resumes exactly one loop or durably marks the run interrupted/failed |
|
||||
| Coordinator and/or worker handle remints | Sender authority survives via pane key; recipient inbox remains on the old handle | New-handle checks miss old mail; exit lookup can miss the worker dispatch | Route by host-scoped pane identity while retaining handles as snapshots |
|
||||
| SSH/WSL/relay is temporarily unreachable after restart | Terminal liveness may be unavailable rather than false | Recovery burns a failure for a worker that is still running remotely | `unknown` never means `exited`; capability/provider state is host-scoped |
|
||||
| CLI mutation succeeds and response is lost (`worktree.create`) | Client reports failure; resulting worktree may be visible | Duplicate side effects on retry or confusing conflicts | Same mutation ID + same payload returns the same durable result across restart |
|
||||
| User intentionally quits Orca | A reconnecting CLI could try to reopen it | Surprising resurrection of an intentionally closed app | Reconnect policy is bounded and explicit about local reopen versus remote waiting |
|
||||
|
||||
## 3. What already works and should be kept
|
||||
|
||||
1. **SQLite is already the right source of truth.** `src/main/runtime/orchestration/db.ts` uses `userData/orchestration.db`, WAL, atomic schema migration, a busy timeout, and persisted task/dispatch/message/gate/run rows. Recovery should build on it rather than add a second runtime-owned database.
|
||||
|
||||
2. **Pane-key lifecycle authority is the correct model.** `lifecycle-reconciliation.ts` accepts reminted handles only when sender and assignee pane identities are equivalent, rejects the wrong pane, suppresses late heartbeats, and requires both task and dispatch IDs. Stable routing should extend this model, not replace it with handle equality or payload knowledge.
|
||||
|
||||
3. **Lifecycle reconciliation is performed on both send and authoritative reads.** A crash between message insertion and send-path reconciliation can still be repaired when a check reads the row. The next design should make this projection transactional/idempotent, while preserving the dual-path safety for migrated or legacy rows.
|
||||
|
||||
4. **Runtime-observed process exit has a useful state transition.** `failDispatch` increments a persisted circuit breaker and returns the task to `ready` until the third failure. Keep that policy, but look up and fail the exact dispatch by pane identity and notify its actual coordinator.
|
||||
|
||||
5. **Transport keepalives, timeout widening, long-poll admission limits, and socket-abort cleanup are sound.** They prevent idle expiry and resource leaks. Reconnection should sit above these mechanisms and preserve their per-connection behavior.
|
||||
|
||||
6. **`read` and `delivered_at` are intentionally separate.** PTY injection does not hide a message from explicit check. A future receipt/cursor must remain a third concern; it should not overload PTY delivery.
|
||||
|
||||
7. **Lifecycle sends skip the `terminal.show` liveness probe.** This avoids blocking the exact mid-restart completion path. Keep the fail-closed sender identity checks and add retry/dedup around the RPC instead.
|
||||
|
||||
8. **Persistent terminal infrastructure already provides the best case.** Daemon PTYs can survive relaunch and be reattached. Orchestration recovery should query that provider-owned truth after hydration instead of attempting to preserve the Electron process.
|
||||
|
||||
9. **The worktree mutation cache is a useful compatibility precedent.** Retrying with `clientMutationId` is the correct API shape. The missing parts are desktop CLI participation and durable storage across runtime generations.
|
||||
|
||||
## 4. Design options
|
||||
|
||||
### Option A: Protocol and skill recovery only
|
||||
|
||||
Change the orchestration skill so `runtime_unavailable` during a supervised wait is a recovery checkpoint:
|
||||
|
||||
- for a local runtime, run/open Orca according to an explicit policy and wait for a new runtime generation;
|
||||
- rerun the same check against the same old terminal handle first;
|
||||
- inspect `task-list --brief`, `dispatch-show`, terminal pane resolution, and Git status before redispatching;
|
||||
- never treat a heartbeat gap alone as worker failure; and
|
||||
- require a worker whose lifecycle send failed to retry rather than claiming completion.
|
||||
|
||||
Advantages: fastest mitigation, no schema change, and it helps current users immediately.
|
||||
|
||||
Limitations: it does not close the mark-read-before-response window, cannot recover a send after the worker process dies, and relies on an LLM following a multi-command playbook. It is a release-zero guardrail, not the reliability boundary.
|
||||
|
||||
### Option B: Reconnecting check plus replayable lifecycle events
|
||||
|
||||
Implement orchestration-specific retry instead of globally retrying `RuntimeClient.call`:
|
||||
|
||||
- `check --wait` retains the original recipient, type filter, and absolute deadline;
|
||||
- on `runtime_unavailable`, runtime-ID change, or `remote_runtime_unavailable`, it refreshes metadata/pairing and reconnects with capped backoff;
|
||||
- intentional timeout and `{count:0}` retain their current checkpoint meaning;
|
||||
- `runtime_busy`, invalid arguments, authorization failures, and incompatible runtimes are not retried as restart failures;
|
||||
- stderr keepalives continue during the reconnect phase so host agent tools do not background the command;
|
||||
- before blocking, the client takes a non-consuming inbox snapshot that returns either pending rows or a high-water sequence; this closes the race between “nothing pending” and registering the wait;
|
||||
- the client then carries `afterSequence` from that observed high-water/last terminal frame, and the server can return newer matching rows regardless of their global read bit; and
|
||||
- lifecycle sends carry a client-generated `eventId`; insertion plus task/dispatch reconciliation is one idempotent transaction.
|
||||
|
||||
Advantages: directly fixes the incident path, works for Unix/named-pipe and remote WebSocket transports, and makes ambiguous send/check retries safe.
|
||||
|
||||
Limitations: a cursor held only by the CLI process does not recover if the entire coordinator PTY/agent dies. Stale recipient handles remain a problem for a fresh coordinator process. Compatibility needs a capability gate because older runtimes strip unknown request fields.
|
||||
|
||||
This is the recommended first code increment.
|
||||
|
||||
### Option C: Runtime startup reconciliation and stable orchestration routing
|
||||
|
||||
Extend durable identity and add a recovery pass after terminal graph and provider hydration:
|
||||
|
||||
- persist `coordinator_handle`, `coordinator_pane_key`, and `coordinator_host_scope` on each dispatch;
|
||||
- persist recipient pane identity/host scope on messages when resolvable, retaining `to_handle` for display and legacy reads;
|
||||
- resolve active dispatches by `assignee_pane_key` plus host scope, update their current routing handle, and preserve them when the PTY/agent is alive;
|
||||
- fail an exact dispatch once when the provider proves the process exited;
|
||||
- retain the dispatch and report `unknown/recovering` when SSH, WSL, relay, or daemon state is unavailable;
|
||||
- enqueue exit escalation to the dispatch coordinator, whether manual or owned by a coordinator run;
|
||||
- persist enough run configuration and a message cursor to resume one `Coordinator` loop after restart; if safe resume is impossible, transition the run to a durable interrupted/failed result instead of leaving `running`; and
|
||||
- surface a read-only `orchestration recover/status` report for manual coordinators.
|
||||
|
||||
Advantages: covers both worker-survives and worker-dies cases, fixes manual coordinator escalation, and removes reliance on stale handle environment variables.
|
||||
|
||||
Limitations: terminal/provider hydration ordering is subtle, especially for SSH. False exit is more damaging than delayed recovery, so the reconciler needs an explicit three-state liveness model (`alive`, `exited`, `unknown`) and exact-dispatch idempotence.
|
||||
|
||||
This is the recommended second code increment.
|
||||
|
||||
### Option D: Detached worker outbox and durable mutation journal
|
||||
|
||||
For the strongest delivery guarantee, write a lifecycle event to a sender-side outbox before attempting the RPC. Replay it until the runtime acknowledges the event ID. Separately, journal mutating RPCs in the runtime database:
|
||||
|
||||
```text
|
||||
operation_id, method, actor_scope, payload_hash,
|
||||
status(pending|succeeded|failed), result, error, timestamps
|
||||
```
|
||||
|
||||
The runtime reserves the operation ID before side effects. A retry with the same ID and payload returns the in-flight or committed result; the same ID with a different payload is rejected. A restart can reconcile a pending operation by its durable business marker rather than blindly rerunning it.
|
||||
|
||||
Advantages: survives both response loss and runtime process death, and generalizes the `worktree.create` partial-success fix.
|
||||
|
||||
Limitations: the outbox owner/location is difficult across native, WSL, SSH, and paired remote clients; replay requires a live process or daemon; retention and credential boundaries need design. Generic mutation journaling is too large for the first orchestration fix.
|
||||
|
||||
Recommendation: build the durable journal first for `worktree.create`. Defer a detached lifecycle outbox until failure injection demonstrates that bounded send retry plus startup process reconciliation is insufficient.
|
||||
|
||||
## 5. Recommended path
|
||||
|
||||
### Increment 0: recovery semantics and diagnostics
|
||||
|
||||
Update the orchestration skill and CLI error text so all coordinators understand:
|
||||
|
||||
- timeout and empty result are normal checkpoints;
|
||||
- runtime loss is a reconnect/recovery checkpoint for a live dispatch, not evidence of worker failure;
|
||||
- after restart, query persisted task/dispatch state and pane-key liveness before redispatch;
|
||||
- reuse the old recipient handle for the first recovery read because current messages are addressed by handle;
|
||||
- do not reset orchestration state or delete/clean the worktree during recovery; and
|
||||
- if a lifecycle send did not receive acceptance, retry it rather than sending `worker_done` “exactly once” at the command-attempt level. The invariant should be one accepted event ID, not one network attempt.
|
||||
|
||||
Add a diagnostic command or report that shows runtime generation, active dispatch, assignee pane/current handle, last heartbeat, provider liveness (`alive|exited|unknown`), coordinator identity, and unread lifecycle rows. It can be read-only in this increment.
|
||||
|
||||
### Increment 1: safe transport resume
|
||||
|
||||
Add a capability such as `orchestration.replayable-wait.v1` and implement:
|
||||
|
||||
1. a non-consuming cursor snapshot plus server query by `afterSequence`, recipient identity, and type filter (a message inserted after the snapshot is immediately visible to the subsequent wait);
|
||||
2. client reconnect for only the known ambiguous transport errors;
|
||||
3. an absolute wait deadline so reconnects do not accidentally extend a 20-minute wait forever;
|
||||
4. lifecycle `eventId` deduplication and transactional projection; and
|
||||
5. failure-injection tests that stop runtime A and start runtime B on the same on-disk SQLite database.
|
||||
|
||||
The old `read` behavior can remain for legacy one-shot checks. The reconnecting path must be able to replay a row that is already read because the prior response may have been lost. The initial snapshot must be non-consuming so losing that response cannot recreate the same bug before a cursor exists. Longer term, replace the global read bit as coordinator-consumption authority with per-consumer receipts/cursors; keep it for backward-compatible inbox UX.
|
||||
|
||||
Do not make `RuntimeClient` retry every mutation. Retry policy belongs to each operation and only becomes safe when that operation has an idempotency key.
|
||||
|
||||
### Increment 2: identity-complete dispatches and startup reconciliation
|
||||
|
||||
Migrate dispatch contexts to persist their coordinator handle, pane key, and host scope. The dispatcher is the owner; `task.created_by_terminal_handle` is useful lineage metadata but is not reliable enough to infer who must receive exit escalation.
|
||||
|
||||
After provider and terminal graph hydration:
|
||||
|
||||
1. load every active dispatch;
|
||||
2. resolve the assignee by host-scoped pane key;
|
||||
3. if alive, update routing metadata and keep `dispatched`;
|
||||
4. if definitively exited, call an exact, active-only failure transition and enqueue one escalation to the dispatch owner;
|
||||
5. if unknown, retain the dispatch and expose recovery state without incrementing the circuit breaker; and
|
||||
6. rehydrate or explicitly interrupt active coordinator runs.
|
||||
|
||||
The current 10-minute stale-heartbeat detector should remain warning-only. Runtime downtime should be excluded from operator wording, and it should never independently trigger redispatch.
|
||||
|
||||
An adjacent `orchestrator-not-poll` branch explored storing `coordinator_handle` on dispatches and event-driven wake ownership. It is useful evidence for the ownership direction, but it diverges from current schema-v6 pane authority and should not be cherry-picked wholesale; the forward migration should store coordinator pane identity and host scope together.
|
||||
|
||||
### Increment 3: durable idempotent mutations
|
||||
|
||||
Start with `worktree.create` because it has a live incident and already has a `clientMutationId` API shape. The journal can live in the canonical runtime store or a focused operations database; unrelated mutation state does not need to be coupled to `orchestration.db`:
|
||||
|
||||
- make the desktop CLI generate and retain a mutation ID for the command;
|
||||
- replace or back the in-memory 60-second cache with a SQLite operation record/result;
|
||||
- capability-gate automatic retries across old runtimes;
|
||||
- bind the ID to a canonical payload hash;
|
||||
- persist the created worktree ID/branch as the result; and
|
||||
- on an interrupted pending row, reconcile the actual worktree/branch before deciding to continue, fail, or require user action.
|
||||
|
||||
Only generalize the journal after a second mutation needs it. “Exactly once” is not the transport promise; the promise is at-least-once requests with one idempotent durable effect.
|
||||
|
||||
### North-star architecture
|
||||
|
||||
```text
|
||||
worker/coordinator CLI
|
||||
├─ stable actor identity: (host scope, pane key)
|
||||
├─ event/mutation ID
|
||||
└─ reconnect loop with absolute deadline + last observed sequence
|
||||
│
|
||||
▼
|
||||
transport (Unix / named pipe / remote WebSocket)
|
||||
└─ disposable connection, keepalive, abort, capability negotiation
|
||||
│
|
||||
▼
|
||||
SQLite durable log + projections
|
||||
├─ messages/events, unique event ID, recipient identity, sequence
|
||||
├─ per-consumer receipt/cursor
|
||||
├─ tasks + dispatches with worker and coordinator identities
|
||||
├─ coordinator run config/cursor
|
||||
└─ selected mutation operation records
|
||||
│
|
||||
▼
|
||||
startup reconciler after provider hydration
|
||||
└─ alive: rebind | exited: fail once | unknown: preserve and report
|
||||
```
|
||||
|
||||
Long-poll waiters remain in memory. On restart, callers recreate them from a cursor against the durable log. This is simpler and safer than attempting to serialize JavaScript promises or “rehydrate waiters.”
|
||||
|
||||
## 6. Concrete acceptance scenarios
|
||||
|
||||
Each scenario should run against an on-disk temporary SQLite DB with two runtime instances where restart matters, not only an in-memory DB.
|
||||
|
||||
1. **Socket dies before any message.** Start `check --wait`, stop runtime A, start runtime B on the same DB, then send `worker_done`. The original CLI command reconnects and returns the completion before its original deadline.
|
||||
|
||||
2. **Response lost after read/reconcile.** Runtime A inserts/reconciles a `worker_done` and marks it read, then closes before writing the terminal frame. Runtime B receives the retry with the old cursor and returns that same message. The task and dispatch are completed once.
|
||||
|
||||
3. **Completion send was never committed.** Stop runtime before the worker send connects. Restart it. The still-running send command retries the same event ID and receives acceptance; exactly one message row exists.
|
||||
|
||||
4. **Completion committed but ACK was lost.** Commit the event and kill the connection before the response. Retry the same event ID after restart. The runtime returns the original row/result and does not duplicate lifecycle effects.
|
||||
|
||||
5. **Worker survives and handle remints.** Persist a dispatch with an assignee pane key, restart, rehydrate the daemon PTY under a new handle, and send heartbeat/worker_done from the equivalent pane. The original dispatch remains active then completes; no failure count is added.
|
||||
|
||||
6. **Worker also dies.** Kill the worker while runtime is absent, then restart. After provider hydration proves exit, recovery fails the exact dispatch once, returns the task to `ready` (or circuit-breaks on the third prior failure), and preserves all worktree files.
|
||||
|
||||
7. **Remote worker state is unknown.** Make the SSH provider unavailable during restart. Recovery does not fail or redispatch. When SSH reconnects and the pane is found alive, the same dispatch is rebound.
|
||||
|
||||
8. **Manual coordinator exit escalation.** Create and dispatch manually without `orchestration.run`, then exit the worker. The escalation is addressed to the dispatching coordinator pane and appears after its handle remints.
|
||||
|
||||
9. **Automated run resumes once.** Restart with a persisted `running` coordinator row. Exactly one loop resumes with its stored cursor/config and it does not double-dispatch. If compatibility prevents resume, the row becomes durably interrupted/failed with an actionable diagnostic rather than staying `running`.
|
||||
|
||||
10. **Stale heartbeat is not exit proof.** Advance time beyond 10 minutes while runtime/provider is unavailable. The dispatch may warn after recovery but is not failed until liveness is definitive.
|
||||
|
||||
11. **Intentional wait timeout remains unchanged.** With no message or restart, the command returns `{count:0}` at the requested deadline; reconnect backoff never extends the deadline.
|
||||
|
||||
12. **Non-retryable failures remain non-retryable.** Invalid type, unauthorized pairing, incompatible runtime, and lifecycle authority rejection return promptly and do not enter reconnect loops.
|
||||
|
||||
13. **Unix, Windows, and remote parity.** Run the wait-restart tests over Unix socket, named pipe, and WebSocket transports. Remote reconnect refreshes pairing/runtime compatibility without conflating the local desktop process with the remote runtime.
|
||||
|
||||
14. **Worktree ambiguous success.** Runtime A completes `worktree.create` then loses the response. Runtime B receives the same mutation ID and payload and returns the same worktree ID. Only one branch/worktree exists; a different payload under that ID is rejected.
|
||||
|
||||
15. **Update install kill path.** When update install kills the PTY and relaunches Orca, startup recovery fails the active dispatch once and notifies its manual or run coordinator. It never reports the killed worker as still heartbeating.
|
||||
|
||||
## 7. Compatibility and rollout notes
|
||||
|
||||
- Add capabilities before the CLI sends replay cursor, event ID, stable-recipient, or durable-mutation semantics. Older Zod schemas may strip unknown fields, which can silently turn a safe retry into an unsafe one.
|
||||
- Keep exact handle queries as the legacy fallback. During the first reconnect attempt on current schema, reuse the original handle; resolving to a new handle too early can skip rows addressed to the old handle.
|
||||
- A pane key must be scoped to the execution host/environment that owns the PTY: native runtime, WSL distro, SSH provider/connection, or relay runtime. Do not assume an opaque leaf ID is globally unique across hosts.
|
||||
- The recovery reconciler must wait for authoritative provider hydration. “Terminal not listed yet” during renderer or SSH startup is `unknown`, not `exited`.
|
||||
- Message/event retention needs a compaction policy once per-consumer cursors exist. Never compact past the oldest live cursor or unresolved lifecycle projection.
|
||||
- Log runtime generation changes, reconnect attempts, event dedup hits, recovery decisions, and operation-journal reconciliation so future incidents can distinguish “not sent,” “committed/response lost,” and “recipient did not consume.”
|
||||
|
||||
## 8. Non-goals and risks
|
||||
|
||||
- Do not try to keep an Electron main process or its socket alive through an app update.
|
||||
- Do not serialize or restore `messageWaitersByHandle`; a waiter is disposable transport state.
|
||||
- Do not promise exactly-once network delivery. Use at-least-once delivery with idempotent event/projection handling.
|
||||
- Do not globally retry arbitrary mutating RPCs. Only retry mutations with a server-recognized durable idempotency key.
|
||||
- Do not automatically reset orchestration state, delete worktrees, clean uncommitted files, commit, or stash during recovery.
|
||||
- Do not auto-fail solely from missing heartbeats, missing renderer leaves, or a temporarily unavailable SSH/WSL/relay provider.
|
||||
- Do not build distributed consensus or multi-coordinator leader election in this project. One persisted coordinator owner/cursor per dispatch/run is enough.
|
||||
- Avoid automatically reopening Orca after an intentional quit unless the command/skill policy explicitly opts in. A bounded reconnect window that waits for a user-initiated restart is safer as the default.
|
||||
- A sender-side detached outbox can leak sensitive message content or cross environment boundaries if its storage scope is wrong. Defer it until ownership, encryption, retention, and replay-agent behavior are specified.
|
||||
|
||||
## 9. Residual open questions
|
||||
|
||||
1. Should local `check --wait` automatically relaunch Orca, or only wait for a new runtime generation and tell the coordinator/skill to invoke `orca open`? Recommendation: bounded wait by default, explicit relaunch opt-in.
|
||||
2. What is the default restart grace? Recommendation: preserve the user’s absolute wait deadline, with capped reconnect backoff and no separate unbounded timer.
|
||||
3. Is the durable receipt cursor per coordinator pane, per coordinator run, or per CLI invocation? Recommendation: per run for `orchestration.run`; per host-scoped pane for manual inbox consumption, with explicit reset/inspection tooling.
|
||||
4. What proves definitive exit for SSH and WSL? Recommendation: provider-authenticated process/session absence after hydration, never a renderer-list miss or transport disconnect alone.
|
||||
5. Where could a detached worker outbox live across native, WSL, SSH, and paired remote execution? Recommendation: do not choose until the execution-host identity and replay owner are explicit; bounded in-process retry is the first release.
|
||||
6. Should update-install shutdown proactively mark active dispatches interrupted before `killAllPty`, or rely on startup reconciliation? Recommendation: record shutdown intent before killing PTYs, then let startup reconciliation perform the exact failure transition.
|
||||
7. How long should event dedup and operation results be retained? Recommendation: longer than the maximum supported restart/retry window and until every relevant receipt cursor advances; begin with conservative bounded retention plus telemetry.
|
||||
8. Should a manual dispatch coordinator be inferred from the task creator or recorded from `dispatch --from`? Recommendation: record the dispatcher; task creator remains lineage only.
|
||||
|
||||
## 10. Evidence map
|
||||
|
||||
- Durable schema, WAL, messages/tasks/dispatches/runs: `src/main/runtime/orchestration/db.ts`
|
||||
- Pane-key authority and lifecycle idempotence checks: `src/main/runtime/orchestration/lifecycle-reconciliation.ts`
|
||||
- In-memory waiters, process-exit failure, push delivery: `src/main/runtime/orca-runtime.ts`
|
||||
- Check/read-before-response ordering and send-path reconciliation: `src/main/runtime/rpc/methods/orchestration.ts`
|
||||
- Module-scoped coordinator loop ownership: `src/main/runtime/rpc/methods/orchestration-gates.ts`
|
||||
- Stale heartbeat warning-only policy: `src/main/runtime/orchestration/coordinator.ts`
|
||||
- Socket-close error and keepalive framing: `src/cli/runtime/transport.ts`
|
||||
- Long-poll timeout widening and remote transport selection: `src/cli/runtime/client.ts`
|
||||
- Single-call check handler and lifecycle no-show-probe comment: `src/cli/handlers/orchestration.ts`
|
||||
- Remote close classification: `src/shared/remote-runtime-client.ts`
|
||||
- Worktree client omission of `clientMutationId`: `src/cli/handlers/worktree.ts`
|
||||
- Existing worktree idempotency capability/cache: `src/shared/protocol-version.ts`, `src/main/runtime/rpc/methods/worktree.ts`, `src/main/runtime/orca-runtime.ts`
|
||||
- Update-install PTY kill path: `src/main/updater.ts`
|
||||
- Daemon PTY warm-reattach behavior: `src/main/daemon/daemon-pty-adapter.ts`
|
||||
Reference in New Issue
Block a user