fix(orchestration): complete tasks on worker_done + coordinator UX fixes (#8030)

* fix(orchestration): complete worker tasks and improve coordinator UX

* Fix orchestration lifecycle sender resolution and peek/check compat hand

- Lifecycle sends (worker_done/heartbeat) now use ORCA_TERMINAL_HANDLE
  verbatim, skipping the liveness probe and pane remint that could
  block delivery during restarts or mismatch stale-runtime assignee
  handles.
- --peek now round-trips as {peek:true, unread:false} so older runtimes
  that strip unknown params degrade to non-destructive "all" instead of
  mark-read, with client-side filtering to restore peek semantics and a
  clear error when --peek --wait can't be honored.
- Reject combined read-mode flags (--unread/--peek/--all) before calling
  the runtime.
- Distinguish suppressed (already-consumed) lifecycle messages from
  ignored ones so send doesn't wake --wait waiters for stale heartbeats.
- Fix task summary truncation to avoid splitting UTF-16 surrogate pairs
  and to not misreport whitespace normalization as truncation.

* Add shared helper to abbreviate orchestration task specs for brief listi

- Normalizes whitespace and caps spec length at 160 chars, flagging
  truncation separately from whitespace-only changes
- Truncates on UTF-16 code point boundaries to avoid splitting
  surrogate pairs and emitting malformed strings

* Add pane-key identity to worker_done/heartbeat reconciliation and server

- Records the sender's pane key on messages and dispatch contexts so
  worker_done/heartbeat ownership can be verified by the remint-stable
  pane leaf instead of the terminal handle, which is reissued across
  restarts.
- Rejects lifecycle messages from a genuinely foreign pane while still
  tolerating handle remints, tab break-outs, and older CLIs that lack
  pane identity.
- Moves task-spec abbreviation server-side (orchestration.taskList
  --brief) so full specs no longer cross SSH/relay transports, with a
  client-side fallback for older runtimes; consolidates the shared
  abbreviation helper under src/shared.
- Adds a stderr warning when a pre-peek runtime's --peek response hits
  the 100-row cap, since older unread messages may be missing.

* Isolate ORCA_PANE_KEY in CLI test beforeEach to fix leaked senderPaneKey

Co-authored-by: Orca <help@stably.ai>

* Fix pane-key remint bypassing dispatch mutual-exclusion lock

- Dispatch locking only matched on assignee_handle, so a reminted
  terminal handle (tab break-out) could open a second concurrent
  dispatch on the same pane.
- Add leaf-UUID-based pane key comparison (parsePaneKey) as a
  secondary lock, falling back to exact handle match for legacy
  rows without pane keys.

* Update orchestration skill docs for lifecycle authority and CLI flag add

- Clarify that dispatch lifecycle is tied to taskId+dispatchId verified against
  the dispatched pane, not the terminal handle, since handles can be reminted
  after restart
- Document new `check --peek`/`--all` and `task-list --brief` flags, with
  fallback guidance for older CLIs that reject them
- Note that a valid worker_done auto-completes the task/dispatch, so workers
  shouldn't also call task-update manually

---------

Co-authored-by: Orca <help@stably.ai>
This commit is contained in:
Jinjing
2026-07-12 02:15:11 -07:00
committed by GitHub
co-authored by Orca
parent ee82d66a35
commit 26934b11bf
24 changed files with 1090 additions and 133 deletions
@@ -1,14 +1,14 @@
// Why: subprocess-level test for the CLI heartbeat behavior described in
// Why: subprocess-level test for the CLI keepalive behavior described in
// design doc §3.4. Spawns the real compiled CLI with no TTY, points it at a
// real in-process runtime via ORCA_USER_DATA_PATH, and asserts:
// - the first heartbeat line appears on stderr well under Claude Code's
// - the first keepalive line appears on stderr well under Claude Code's
// ~2 min Bash-tool silence budget (we verify with a shortened interval;
// production uses 15 s via the same code path)
// - ≥3 heartbeats arrive during the wait window
// - stderr is line-flushed (we observe each heartbeat as a separate chunk
// - ≥3 keepalives arrive during the wait window
// - stderr is line-flushed (we observe each keepalive as a separate chunk
// before the process exits — not in one burst at the end)
// - stdout stays a single clean JSON payload (no heartbeats leak to stdout)
// - a `jq "select(._heartbeat|not)"` filter on the merged stream would
// - stdout stays a single clean JSON payload (no keepalives leak to stdout)
// - a `jq "select(._keepalive|not)"` filter on the merged stream would
// yield exactly the final result
//
// This test is skipped if the CLI hasn't been built yet (out/cli/index.js
@@ -66,7 +66,7 @@ async function runBuiltCli(
}
describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
it('emits newline-flushed JSON heartbeats to stderr while waiting', async () => {
it('emits newline-flushed JSON keepalives to stderr while waiting', async () => {
const userDataPath = mkdtempSync(join(tmpdir(), 'orca-cli-sub-'))
const runtime = new OrcaRuntimeService()
const db = new OrchestrationDb(':memory:')
@@ -75,12 +75,12 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
await server.start()
try {
// Why: use the ORCA_HEARTBEAT_INTERVAL_MS escape hatch to shrink the
// Why: use the ORCA_KEEPALIVE_INTERVAL_MS escape hatch to shrink the
// test to ~1 s wall time. Production callers never set this; the
// production default (15 s) is exercised by §3.4's own unit tests
// and by the fact that this same code path runs with the real
// constant when the env var is absent.
const heartbeatMs = 200
const keepaliveMs = 200
const waitTimeoutMs = 1200
const child = spawn(
@@ -99,7 +99,7 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
...process.env,
ORCA_USER_DATA_PATH: userDataPath,
ORCA_TERMINAL_HANDLE: 'term_nobody',
ORCA_HEARTBEAT_INTERVAL_MS: String(heartbeatMs)
ORCA_KEEPALIVE_INTERVAL_MS: String(keepaliveMs)
},
// Why: explicit pipe for all three fds so we can watch stderr
// in real time; no TTY attached (Bash-tool parity).
@@ -125,7 +125,7 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
const stderr = stderrChunks.map((c) => c.data).join('')
const stdout = stdoutChunks.map((c) => c.data).join('')
const heartbeatLines = stderr
const keepaliveLines = stderr
.split('\n')
.filter((line) => line.trim().length > 0)
.map((line) => {
@@ -135,51 +135,52 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
return null
}
})
.filter((p): p is Record<string, unknown> => p !== null && p._heartbeat === true)
.filter((p): p is Record<string, unknown> => p !== null && p._keepalive === true)
// ≥3 heartbeats in a 1.2s window with a 200ms interval
expect(heartbeatLines.length).toBeGreaterThanOrEqual(3)
expect(heartbeatLines[0]).toHaveProperty('elapsedMs')
expect(heartbeatLines[0]).toHaveProperty('deadlineMs', waitTimeoutMs)
// ≥3 keepalives in a 1.2s window with a 200ms interval
expect(keepaliveLines.length).toBeGreaterThanOrEqual(3)
expect(keepaliveLines[0]).toMatchObject({ _keepalive: true, _heartbeat: true })
expect(keepaliveLines[0]).toHaveProperty('elapsedMs')
expect(keepaliveLines[0]).toHaveProperty('deadlineMs', waitTimeoutMs)
// Why: under full-suite load the child process startup may take longer
// than one heartbeat interval. The invariant that matters is that at
// least one heartbeat is observed before the terminal stdout payload.
const firstHeartbeatChunk = stderrChunks.find((c) => c.data.includes('_heartbeat'))
expect(firstHeartbeatChunk).toBeDefined()
expect(firstHeartbeatChunk!.at).toBeLessThan(stdoutChunks[0]?.at ?? Number.POSITIVE_INFINITY)
// than one keepalive interval. The invariant that matters is that at
// least one keepalive is observed before the terminal stdout payload.
const firstKeepaliveChunk = stderrChunks.find((c) => c.data.includes('_keepalive'))
expect(firstKeepaliveChunk).toBeDefined()
expect(firstKeepaliveChunk!.at).toBeLessThan(stdoutChunks[0]?.at ?? Number.POSITIVE_INFINITY)
// Why: line-flushing proof — the *first* heartbeat chunk must arrive
// Why: line-flushing proof — the *first* keepalive chunk must arrive
// strictly before the exit chunk; i.e. we got at least two separate
// stderr data events (heartbeat + final). A single-chunk delivery
// stderr data events (keepalive + final). A single-chunk delivery
// would indicate stderr was buffered until exit.
const lastStderrAt = stderrChunks.at(-1)?.at ?? 0
const firstStderrAt = stderrChunks.at(0)?.at ?? 0
expect(lastStderrAt).toBeGreaterThan(firstStderrAt)
// Stdout: exactly one JSON payload, the terminal result. No heartbeats
// Stdout: exactly one JSON payload, the terminal result. No keepalives
// leak, and the content parses as valid JSON.
const stdoutTrimmed = stdout.trim()
const stdoutPayload = JSON.parse(stdoutTrimmed) as Record<string, unknown>
expect(stdoutPayload).not.toHaveProperty('_heartbeat')
expect(stdoutTrimmed).not.toContain('_heartbeat')
expect(stdoutPayload).not.toHaveProperty('_keepalive')
expect(stdoutTrimmed).not.toContain('_keepalive')
// Why: result should be an RPC success envelope with the expected
// shape. `count: 0` and `messages: []` because the wait timed out
// with no message for term_nobody.
expect(stdoutPayload).toMatchObject({ ok: true })
// Why: the heartbeats-on-stderr design is meant to pair with shell
// filters like `2>&1 | jq "select(._heartbeat|not)"`. jq is
// Why: the keepalives-on-stderr design is meant to pair with shell
// filters like `2>&1 | jq "select(._keepalive|not)"`. jq is
// line-oriented by default, but also accepts pretty-printed JSON
// across multiple lines. What matters here is that every
// heartbeat line on stderr is a standalone JSON object (so jq can
// keepalive line on stderr is a standalone JSON object (so jq can
// match it) and doesn't span multiple lines — assert that each
// heartbeat is a single-line JSON with no embedded newlines.
// keepalive is a single-line JSON with no embedded newlines.
for (const line of stderr.split('\n')) {
if (line.trim().length === 0) {
continue
}
if (line.includes('_heartbeat')) {
if (line.includes('_keepalive')) {
expect(() => JSON.parse(line)).not.toThrow()
expect(line).not.toContain('\n')
}