Files
orca/src/main/runtime/orchestration
Jinwoo HongandOrca ea8ea08116 feat(orchestration): bundled improvements — check-wait, stale-base, preamble+ask, QoL (#1403)
* feat(orchestration): transport keepalive + delivered_at split for check --wait

Implements the four §3 fixes from the check-wait design doc:

- §3.1 Transport keepalive: long-poll RPCs (orchestration.check --wait) emit
  `{"_keepalive":true}` frames every 10s so neither server nor client tears
  the socket down on idle. A `longPoll` admission counter capped at 16 fails
  fast with `runtime_busy` when saturated; an AbortController wired through
  the RPC dispatcher cancels the inner waiter the moment the socket closes.
- §3.2 delivered_at split: push-on-idle now stamps `delivered_at` instead of
  flipping `read`, so the check caller remains the sole consumer of its
  queue. Adds a synchronous idempotent schema migration that hard-fails on
  error.
- §3.3 inbox/check parity: `orchestration inbox --terminal <handle>` and
  `orchestration check --all` agree on the same rows (sequence DESC, no
  mark-read). `check --unread=false` kept for one release as a compat shim.
- §3.4 CLI heartbeat: `orca orchestration check --wait` emits JSON heartbeat
  lines to stderr every 15s so Claude Code's Bash tool sees continuous
  output and doesn't auto-background the subprocess.

Tests: extends runtime-rpc, orca-runtime, envelope-schema, orchestration
method, and formatter suites; adds a subprocess test that spawns the built
CLI and verifies stderr line-flushing, heartbeat cadence, and stdout
cleanliness end-to-end.

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): preamble rules + heartbeat schema

- Preamble (#7, #15, #9): worker_done body ("3-sentence summary" + reportPath),
  BEHAVIOR RULE #1 forbidding AskUserQuestion, heartbeat every 5 minutes with
  taskId+dispatchId payload, AFTER YOU SEND grace window.
- Schema v2 migration: adds 'heartbeat' to messages.type CHECK, adds
  dispatch_contexts.last_heartbeat_at, gated by user_version PRAGMA with
  transactional rebuild + explicit CREATE INDEX to avoid silent perf regress.
- DB helpers: recordHeartbeat (dispatched-only), getStaleDispatches,
  getThreadMessagesFor (thread+handle scoped for ask).

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): coordinator heartbeat + stale detector

Handle incoming 'heartbeat' messages by calling recordHeartbeat keyed on
payload.dispatchId (strict — log-and-skip if missing, no taskId fallback so
a straggler heartbeat from a previously-failed dispatch cannot mask a hung
retry per §5.3.4). On every tick after the 10-minute threshold, emit one
log per stale dispatched row — no auto-fail.

Also threads dispatchId through buildDispatchPreamble so workers can
attribute their heartbeats back to the correct dispatch context.

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): orca orchestration ask verb

Adds a CLI verb that sends a decision_gate message and blocks on the
coordinator's reply, scoped to the outbound message's thread. Group
addresses (@all, @idle, …) are rejected — fan-out questions must use
send --type decision_gate explicitly.

--json emits bare single-line JSON (bypassing printResult) so workers can
pipe `orca orchestration ask … --json | jq -r .answer` without unwrapping
an RPC envelope; human mode prints just the answer. On timeout the verb
exits 1 and returns {answer: null, timedOut: true}.

This is the CLI surface BEHAVIOR RULE #1 in the dispatch preamble points
workers at instead of AskUserQuestion.

Co-authored-by: Orca <help@stably.ai>

* feat(orchestration): QoL bundle — preamble visibility, status enum, dispatch cross-ref, inbox --full

Addresses four items from ORCHESTRATOR_FEEDBACK:

- #5 preamble visibility: `dispatch-show --preamble` regenerates the preamble
  text for a task; `dispatch --inject --dry-run` previews without mutating
  state; `dispatch --return-preamble` echoes the injected preamble in the JSON
  response so coordinators can audit what a worker received.
- #6 status enum validation: CLI rejects unknown `task-update --status` values
  with `invalid status '<x>', expected one of: pending, ready, dispatched,
  completed, failed, blocked` before the RPC's generic Zod message. Valid
  statuses are listed under Notes in `task-update --help`.
- #13 task-list dispatch cross-ref: `task-list --json` now includes
  `assignee_handle` and `dispatch_id` for tasks in status=dispatched via a
  read-only LEFT JOIN on dispatch_contexts. Non-dispatched rows keep their
  legacy shape so existing consumers are unaffected.
- #14 inbox body visibility: `inbox --full` prints body + payload verbatim;
  default output is unchanged (id/from/to/subject only).

No DB migrations; join-only change on dispatch_contexts so the sibling
preamble PR's `last_heartbeat_at` column addition will not conflict.

Co-authored-by: Orca <help@stably.ai>

* fix(worktree): prevent stale-base worktree creation and dispatch

Addresses feedback #16 per DESIGN_DOC_STALE_BASE_FIX.md §0. Four v1
components coordinated by a single shared fetch cache on the runtime:

1. Concurrent-fetch-with-gate in UI create path: `createLocalWorktree`
   fires `git fetch` BEFORE the suffix loop / PR probe / path
   resolution, then awaits right before `addWorktree` so the new branch
   always spawns from a fresh remote tip. Renderer sees a two-phase
   spinner via the new `createWorktree:progress` IPC event. The cache
   is a `Map<repoPath::remote, Promise<void>>` + 30s success-only
   timestamp on `OrcaRuntimeService` (§7.1 — shared with dispatch).
2. Dispatch pre-flight drift guard in `Coordinator.dispatchTask`:
   probes `rev-list --left-right --count` against the target worktree
   and silently returns (preserves `ready`, no circuit-breaker burn)
   when `behind > 20` unless the task spec carries
   `allow-stale-base: true`. Parsing strips the flag so it never leaks
   into the worker's `--- TASK ---` block.
3. Preamble drift section: populated only when dispatch detected drift.
   Workers see `--- BASE DRIFT ---` with the N-most-recent subjects
   they don't have, so they can pull them in before running.
4. §3.3 Lifecycle: `.finally()` evicts Map entries on BOTH success and
   rejection; timestamp is written ONLY on success. Prevents a single
   DNS hiccup from wedging every future create on the repo until
   restart, and keeps the freshness window honest.

Defers the DB `allow_stale_base` column (§0.2) and the create-time
warn toast; both can layer in later without migration.

Tests: 35 new/updated unit tests covering drift preamble, dispatch
refusal, spec-text flag parsing, fetch Map eviction after rejection,
freshness-window short-circuit, and concurrent-caller serialization.

Co-authored-by: Orca <help@stably.ai>

* test(orchestration): seed v2 DB in migration hard-fail test

After consolidating the schema bump, fresh DBs are initialized directly at
v3 via createTables(), so the v2→v3 ALTER TABLE is skipped on new installs
and the prior test's stub never fired. Seed a v2-shape file on disk so the
guarded ALTER actually runs and the "simulated migration failure" stub
propagates as intended.

Co-authored-by: Orca <help@stably.ai>

---------

Co-authored-by: Orca <help@stably.ai>
2026-05-04 12:10:53 -07:00
..