mirror of
https://github.com/stablyai/orca.git
synced 2026-10-06 16:02:25 +00:00
fix(orchestration): complete tasks on worker_done + coordinator UX fixes (#8030)
* fix(orchestration): complete worker tasks and improve coordinator UX
* Fix orchestration lifecycle sender resolution and peek/check compat hand
- Lifecycle sends (worker_done/heartbeat) now use ORCA_TERMINAL_HANDLE
verbatim, skipping the liveness probe and pane remint that could
block delivery during restarts or mismatch stale-runtime assignee
handles.
- --peek now round-trips as {peek:true, unread:false} so older runtimes
that strip unknown params degrade to non-destructive "all" instead of
mark-read, with client-side filtering to restore peek semantics and a
clear error when --peek --wait can't be honored.
- Reject combined read-mode flags (--unread/--peek/--all) before calling
the runtime.
- Distinguish suppressed (already-consumed) lifecycle messages from
ignored ones so send doesn't wake --wait waiters for stale heartbeats.
- Fix task summary truncation to avoid splitting UTF-16 surrogate pairs
and to not misreport whitespace normalization as truncation.
* Add shared helper to abbreviate orchestration task specs for brief listi
- Normalizes whitespace and caps spec length at 160 chars, flagging
truncation separately from whitespace-only changes
- Truncates on UTF-16 code point boundaries to avoid splitting
surrogate pairs and emitting malformed strings
* Add pane-key identity to worker_done/heartbeat reconciliation and server
- Records the sender's pane key on messages and dispatch contexts so
worker_done/heartbeat ownership can be verified by the remint-stable
pane leaf instead of the terminal handle, which is reissued across
restarts.
- Rejects lifecycle messages from a genuinely foreign pane while still
tolerating handle remints, tab break-outs, and older CLIs that lack
pane identity.
- Moves task-spec abbreviation server-side (orchestration.taskList
--brief) so full specs no longer cross SSH/relay transports, with a
client-side fallback for older runtimes; consolidates the shared
abbreviation helper under src/shared.
- Adds a stderr warning when a pre-peek runtime's --peek response hits
the 100-row cap, since older unread messages may be missing.
* Isolate ORCA_PANE_KEY in CLI test beforeEach to fix leaked senderPaneKey
Co-authored-by: Orca <help@stably.ai>
* Fix pane-key remint bypassing dispatch mutual-exclusion lock
- Dispatch locking only matched on assignee_handle, so a reminted
terminal handle (tab break-out) could open a second concurrent
dispatch on the same pane.
- Add leaf-UUID-based pane key comparison (parsePaneKey) as a
secondary lock, falling back to exact handle match for legacy
rows without pane keys.
* Update orchestration skill docs for lifecycle authority and CLI flag add
- Clarify that dispatch lifecycle is tied to taskId+dispatchId verified against
the dispatched pane, not the terminal handle, since handles can be reminted
after restart
- Document new `check --peek`/`--all` and `task-list --brief` flags, with
fallback guidance for older CLIs that reject them
- Note that a valid worker_done auto-completes the task/dispatch, so workers
shouldn't also call task-update manually
---------
Co-authored-by: Orca <help@stably.ai>
This commit is contained in:
@@ -1,14 +1,14 @@
|
||||
// Why: subprocess-level test for the CLI heartbeat behavior described in
|
||||
// Why: subprocess-level test for the CLI keepalive behavior described in
|
||||
// design doc §3.4. Spawns the real compiled CLI with no TTY, points it at a
|
||||
// real in-process runtime via ORCA_USER_DATA_PATH, and asserts:
|
||||
// - the first heartbeat line appears on stderr well under Claude Code's
|
||||
// - the first keepalive line appears on stderr well under Claude Code's
|
||||
// ~2 min Bash-tool silence budget (we verify with a shortened interval;
|
||||
// production uses 15 s via the same code path)
|
||||
// - ≥3 heartbeats arrive during the wait window
|
||||
// - stderr is line-flushed (we observe each heartbeat as a separate chunk
|
||||
// - ≥3 keepalives arrive during the wait window
|
||||
// - stderr is line-flushed (we observe each keepalive as a separate chunk
|
||||
// before the process exits — not in one burst at the end)
|
||||
// - stdout stays a single clean JSON payload (no heartbeats leak to stdout)
|
||||
// - a `jq "select(._heartbeat|not)"` filter on the merged stream would
|
||||
// - stdout stays a single clean JSON payload (no keepalives leak to stdout)
|
||||
// - a `jq "select(._keepalive|not)"` filter on the merged stream would
|
||||
// yield exactly the final result
|
||||
//
|
||||
// This test is skipped if the CLI hasn't been built yet (out/cli/index.js
|
||||
@@ -66,7 +66,7 @@ async function runBuiltCli(
|
||||
}
|
||||
|
||||
describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
|
||||
it('emits newline-flushed JSON heartbeats to stderr while waiting', async () => {
|
||||
it('emits newline-flushed JSON keepalives to stderr while waiting', async () => {
|
||||
const userDataPath = mkdtempSync(join(tmpdir(), 'orca-cli-sub-'))
|
||||
const runtime = new OrcaRuntimeService()
|
||||
const db = new OrchestrationDb(':memory:')
|
||||
@@ -75,12 +75,12 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
|
||||
await server.start()
|
||||
|
||||
try {
|
||||
// Why: use the ORCA_HEARTBEAT_INTERVAL_MS escape hatch to shrink the
|
||||
// Why: use the ORCA_KEEPALIVE_INTERVAL_MS escape hatch to shrink the
|
||||
// test to ~1 s wall time. Production callers never set this; the
|
||||
// production default (15 s) is exercised by §3.4's own unit tests
|
||||
// and by the fact that this same code path runs with the real
|
||||
// constant when the env var is absent.
|
||||
const heartbeatMs = 200
|
||||
const keepaliveMs = 200
|
||||
const waitTimeoutMs = 1200
|
||||
|
||||
const child = spawn(
|
||||
@@ -99,7 +99,7 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
|
||||
...process.env,
|
||||
ORCA_USER_DATA_PATH: userDataPath,
|
||||
ORCA_TERMINAL_HANDLE: 'term_nobody',
|
||||
ORCA_HEARTBEAT_INTERVAL_MS: String(heartbeatMs)
|
||||
ORCA_KEEPALIVE_INTERVAL_MS: String(keepaliveMs)
|
||||
},
|
||||
// Why: explicit pipe for all three fds so we can watch stderr
|
||||
// in real time; no TTY attached (Bash-tool parity).
|
||||
@@ -125,7 +125,7 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
|
||||
const stderr = stderrChunks.map((c) => c.data).join('')
|
||||
const stdout = stdoutChunks.map((c) => c.data).join('')
|
||||
|
||||
const heartbeatLines = stderr
|
||||
const keepaliveLines = stderr
|
||||
.split('\n')
|
||||
.filter((line) => line.trim().length > 0)
|
||||
.map((line) => {
|
||||
@@ -135,51 +135,52 @@ describeIfBuilt('orca orchestration check --wait subprocess (§3.4)', () => {
|
||||
return null
|
||||
}
|
||||
})
|
||||
.filter((p): p is Record<string, unknown> => p !== null && p._heartbeat === true)
|
||||
.filter((p): p is Record<string, unknown> => p !== null && p._keepalive === true)
|
||||
|
||||
// ≥3 heartbeats in a 1.2s window with a 200ms interval
|
||||
expect(heartbeatLines.length).toBeGreaterThanOrEqual(3)
|
||||
expect(heartbeatLines[0]).toHaveProperty('elapsedMs')
|
||||
expect(heartbeatLines[0]).toHaveProperty('deadlineMs', waitTimeoutMs)
|
||||
// ≥3 keepalives in a 1.2s window with a 200ms interval
|
||||
expect(keepaliveLines.length).toBeGreaterThanOrEqual(3)
|
||||
expect(keepaliveLines[0]).toMatchObject({ _keepalive: true, _heartbeat: true })
|
||||
expect(keepaliveLines[0]).toHaveProperty('elapsedMs')
|
||||
expect(keepaliveLines[0]).toHaveProperty('deadlineMs', waitTimeoutMs)
|
||||
|
||||
// Why: under full-suite load the child process startup may take longer
|
||||
// than one heartbeat interval. The invariant that matters is that at
|
||||
// least one heartbeat is observed before the terminal stdout payload.
|
||||
const firstHeartbeatChunk = stderrChunks.find((c) => c.data.includes('_heartbeat'))
|
||||
expect(firstHeartbeatChunk).toBeDefined()
|
||||
expect(firstHeartbeatChunk!.at).toBeLessThan(stdoutChunks[0]?.at ?? Number.POSITIVE_INFINITY)
|
||||
// than one keepalive interval. The invariant that matters is that at
|
||||
// least one keepalive is observed before the terminal stdout payload.
|
||||
const firstKeepaliveChunk = stderrChunks.find((c) => c.data.includes('_keepalive'))
|
||||
expect(firstKeepaliveChunk).toBeDefined()
|
||||
expect(firstKeepaliveChunk!.at).toBeLessThan(stdoutChunks[0]?.at ?? Number.POSITIVE_INFINITY)
|
||||
|
||||
// Why: line-flushing proof — the *first* heartbeat chunk must arrive
|
||||
// Why: line-flushing proof — the *first* keepalive chunk must arrive
|
||||
// strictly before the exit chunk; i.e. we got at least two separate
|
||||
// stderr data events (heartbeat + final). A single-chunk delivery
|
||||
// stderr data events (keepalive + final). A single-chunk delivery
|
||||
// would indicate stderr was buffered until exit.
|
||||
const lastStderrAt = stderrChunks.at(-1)?.at ?? 0
|
||||
const firstStderrAt = stderrChunks.at(0)?.at ?? 0
|
||||
expect(lastStderrAt).toBeGreaterThan(firstStderrAt)
|
||||
|
||||
// Stdout: exactly one JSON payload, the terminal result. No heartbeats
|
||||
// Stdout: exactly one JSON payload, the terminal result. No keepalives
|
||||
// leak, and the content parses as valid JSON.
|
||||
const stdoutTrimmed = stdout.trim()
|
||||
const stdoutPayload = JSON.parse(stdoutTrimmed) as Record<string, unknown>
|
||||
expect(stdoutPayload).not.toHaveProperty('_heartbeat')
|
||||
expect(stdoutTrimmed).not.toContain('_heartbeat')
|
||||
expect(stdoutPayload).not.toHaveProperty('_keepalive')
|
||||
expect(stdoutTrimmed).not.toContain('_keepalive')
|
||||
// Why: result should be an RPC success envelope with the expected
|
||||
// shape. `count: 0` and `messages: []` because the wait timed out
|
||||
// with no message for term_nobody.
|
||||
expect(stdoutPayload).toMatchObject({ ok: true })
|
||||
|
||||
// Why: the heartbeats-on-stderr design is meant to pair with shell
|
||||
// filters like `2>&1 | jq "select(._heartbeat|not)"`. jq is
|
||||
// Why: the keepalives-on-stderr design is meant to pair with shell
|
||||
// filters like `2>&1 | jq "select(._keepalive|not)"`. jq is
|
||||
// line-oriented by default, but also accepts pretty-printed JSON
|
||||
// across multiple lines. What matters here is that every
|
||||
// heartbeat line on stderr is a standalone JSON object (so jq can
|
||||
// keepalive line on stderr is a standalone JSON object (so jq can
|
||||
// match it) and doesn't span multiple lines — assert that each
|
||||
// heartbeat is a single-line JSON with no embedded newlines.
|
||||
// keepalive is a single-line JSON with no embedded newlines.
|
||||
for (const line of stderr.split('\n')) {
|
||||
if (line.trim().length === 0) {
|
||||
continue
|
||||
}
|
||||
if (line.includes('_heartbeat')) {
|
||||
if (line.includes('_keepalive')) {
|
||||
expect(() => JSON.parse(line)).not.toThrow()
|
||||
expect(line).not.toContain('\n')
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user