mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
Implement robust orchestration primitives and connected-server workers (#9925)
This commit is contained in:
OrcaWin
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,604 @@
|
||||
# Orchestration Structured Worker Output
|
||||
|
||||
Status: implemented; physical local, mixed-version, and Windows-home to Mac-worker validation complete
|
||||
Scope: orchestration `worker-read` only
|
||||
Last updated: 2026-07-24
|
||||
|
||||
## Summary
|
||||
|
||||
`orca orchestration worker-read` currently reads bounded terminal output. That is always available,
|
||||
but full-screen agent TUIs can make it noisy or incomplete.
|
||||
|
||||
Orca already knows more than the terminal logo suggests. Agent hooks associate an exact pane with:
|
||||
|
||||
- the agent type, such as Codex or Claude;
|
||||
- the provider session or conversation ID; and
|
||||
- when available, the provider-reported transcript path.
|
||||
|
||||
The sidebar, session resume, sleeping-agent recovery, and native chat already use this information.
|
||||
The missing piece is a narrow orchestration path from an exact Dispatch to that exact session on the
|
||||
server where the worker runs.
|
||||
|
||||
The proposed behavior is:
|
||||
|
||||
```text
|
||||
worker-read
|
||||
exact supported transcript is available -> structured transcript page
|
||||
otherwise -> bounded terminal page
|
||||
```
|
||||
|
||||
This does not add orchestration strategy, a dashboard, a scheduler, or a universal provider layer.
|
||||
It makes one existing observation command return the best source Orca can prove.
|
||||
|
||||
## User-facing goal
|
||||
|
||||
A coordinator should be able to inspect a worker with one predictable command:
|
||||
|
||||
```bash
|
||||
orca orchestration worker-read --dispatch <dispatch-id> --json
|
||||
```
|
||||
|
||||
The coordinator should not need to know:
|
||||
|
||||
- which Orca server owns the worker;
|
||||
- the worker's terminal or pane handle;
|
||||
- the provider session ID;
|
||||
- where a transcript lives on disk; or
|
||||
- whether structured reading is supported by that provider/server version.
|
||||
|
||||
The response must always say which source was used. It must never silently read a different agent
|
||||
session.
|
||||
|
||||
## Why the existing sidebar is relevant
|
||||
|
||||
The agent logo identifies the detected agent type. By itself, that is not enough to choose a
|
||||
transcript.
|
||||
|
||||
The richer sidebar status also carries pane-scoped provider-session metadata reported by hooks.
|
||||
That is the useful foundation:
|
||||
|
||||
```text
|
||||
Dispatch
|
||||
-> exact worker process and terminal
|
||||
-> exact tab/leaf pane
|
||||
-> hook-reported provider session
|
||||
-> provider transcript locator
|
||||
```
|
||||
|
||||
Some live status ownership is currently renderer-centric, while headless and mobile graph paths
|
||||
also retain compatible status snapshots. Implementation therefore needs one runtime-owned lookup
|
||||
that exposes the current exact pane association to `worker-read`. This is a small bridge over
|
||||
existing status data, not a second agent-status system.
|
||||
|
||||
## Design principles
|
||||
|
||||
### Exactness over convenience
|
||||
|
||||
- Never select the "latest session in this directory."
|
||||
- Never select a transcript from a terminal title or logo alone.
|
||||
- Never switch sources or sessions in the middle of a cursor chain.
|
||||
- If Orca cannot prove the association, return a labeled terminal fallback.
|
||||
|
||||
### Resource-local reads
|
||||
|
||||
The server running the worker resolves and reads its transcript. A Run home on macOS must not try
|
||||
to interpret a Windows path, and a Windows Run home must not try to interpret a macOS path.
|
||||
|
||||
Only bounded output data crosses the federation connection. Transcript paths do not.
|
||||
|
||||
### One simple agent command
|
||||
|
||||
Agents should not choose provider adapters or supply session metadata. `worker-read` defaults to
|
||||
automatic source selection. Source selection flags exist for debugging and explicit policy, not
|
||||
because they are required in the normal loop.
|
||||
|
||||
### Narrow provider support
|
||||
|
||||
Initial support should cover only providers for which Orca already has:
|
||||
|
||||
1. an exact pane-scoped session association; and
|
||||
2. an existing bounded transcript reader with test coverage.
|
||||
|
||||
Codex is the required first provider. Claude may ship in the same change only if it uses the same
|
||||
proven reader path without adding a second architecture. Other agents receive terminal fallback.
|
||||
|
||||
### Honest compatibility
|
||||
|
||||
Connected servers can run different Orca versions. A server without structured-read support must
|
||||
continue to return bounded terminal output rather than failing the whole Run.
|
||||
|
||||
## Public command contract
|
||||
|
||||
### Request
|
||||
|
||||
```bash
|
||||
orca orchestration worker-read \
|
||||
--dispatch <dispatch-id> \
|
||||
[--source auto|transcript|terminal] \
|
||||
[--cursor <opaque-cursor>] \
|
||||
[--limit <count>] \
|
||||
[--json]
|
||||
```
|
||||
|
||||
`--source` behavior:
|
||||
|
||||
| Value | Behavior |
|
||||
| ------------ | -------------------------------------------------------------------------------------- |
|
||||
| `auto` | Use an exact supported transcript; otherwise use terminal output. This is the default. |
|
||||
| `transcript` | Require an exact supported transcript. Return a typed error instead of falling back. |
|
||||
| `terminal` | Use the current bounded terminal reader. |
|
||||
|
||||
Existing numeric terminal cursors remain accepted. New responses return an opaque cursor that can
|
||||
pin either source without exposing provider paths.
|
||||
|
||||
### Response
|
||||
|
||||
The implemented structured response has this shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"dispatchId": "dispatch_123",
|
||||
"source": "transcript",
|
||||
"sourceIdentity": "opaque-source-fingerprint",
|
||||
"provider": "codex",
|
||||
"transcript": {
|
||||
"messages": [],
|
||||
"nextCursor": "opaque-next-cursor",
|
||||
"limited": false,
|
||||
"returnedMessageCount": 0
|
||||
},
|
||||
"cursor": "opaque-next-cursor",
|
||||
"status": {
|
||||
"worker": "running",
|
||||
"terminal": "running"
|
||||
},
|
||||
"fallbackReason": null,
|
||||
"warnings": []
|
||||
}
|
||||
```
|
||||
|
||||
Terminal fallback uses the same envelope and keeps the existing terminal data:
|
||||
|
||||
```json
|
||||
{
|
||||
"dispatchId": "dispatch_123",
|
||||
"source": "terminal",
|
||||
"sourceIdentity": "opaque-terminal-incarnation",
|
||||
"terminal": {
|
||||
"tail": ["..."],
|
||||
"status": "running",
|
||||
"nextCursor": "..."
|
||||
},
|
||||
"cursor": "opaque-next-cursor",
|
||||
"fallbackReason": "session_not_reported",
|
||||
"warnings": []
|
||||
}
|
||||
```
|
||||
|
||||
For backward compatibility:
|
||||
|
||||
- the terminal branch retains the existing `terminal.tail`, `terminal.status`, and
|
||||
`terminal.nextCursor` fields;
|
||||
- non-JSON output prints readable transcript entries or terminal lines without requiring an agent
|
||||
to branch on JSON manually;
|
||||
- a mixed-version federated read can return the legacy terminal shape, which the Run home wraps as
|
||||
a labeled terminal response.
|
||||
|
||||
### Fallback reasons
|
||||
|
||||
Fallback reasons are bounded typed values, not arbitrary policy:
|
||||
|
||||
- `provider_unsupported`
|
||||
- `session_not_reported`
|
||||
- `transcript_missing`
|
||||
- `transcript_unreadable`
|
||||
- `transcript_parse_failed`
|
||||
- `remote_capability_unavailable`
|
||||
|
||||
Warnings may provide safe diagnostic context, but must not contain a remote filesystem path.
|
||||
|
||||
### Typed errors
|
||||
|
||||
Errors are used when returning any source would be misleading:
|
||||
|
||||
- `dispatch_not_found`
|
||||
- `worker_identity_changed`
|
||||
- `source_changed`
|
||||
- `cursor_invalid`
|
||||
- `cursor_dispatch_mismatch`
|
||||
- `transcript_required`
|
||||
- the existing connected-server unavailable/unknown result
|
||||
|
||||
`source_changed` means the exact pane is now associated with a different provider session than the
|
||||
one pinned by the cursor. The caller starts a new read without the old cursor. Orca never silently
|
||||
jumps to the replacement session.
|
||||
|
||||
## Source identity and cursor rules
|
||||
|
||||
The cursor is opaque to clients and contains only non-sensitive routing data:
|
||||
|
||||
- cursor version;
|
||||
- Dispatch ID;
|
||||
- source kind;
|
||||
- an opaque digest of the exact source identity;
|
||||
- provider-specific or terminal paging position.
|
||||
|
||||
It must not contain a transcript path.
|
||||
|
||||
On every continued read, the worker server:
|
||||
|
||||
1. revalidates the Dispatch's exact process attachment;
|
||||
2. resolves the current pane/session association;
|
||||
3. compares its source digest with the cursor;
|
||||
4. reads only when they still match; and
|
||||
5. otherwise returns `source_changed` or `worker_identity_changed`.
|
||||
|
||||
The cursor is source-pinned even when the request uses `--source auto`. `auto` chooses only on the
|
||||
first page.
|
||||
|
||||
The implementation uses a versioned stateless token. The token is not an authority credential:
|
||||
every read revalidates the Dispatch, process, pane, source digest, and provider session before
|
||||
returning data. This lets paging survive an Orca restart without adding cursor-secret persistence.
|
||||
|
||||
## Runtime architecture
|
||||
|
||||
### 1. Resolve the Dispatch at its Run home
|
||||
|
||||
The Run home remains authoritative for Task and Dispatch state. It looks up the Dispatch and its
|
||||
pinned worker server exactly as `worker-show` and the current `worker-read` do.
|
||||
|
||||
No automatic placement or server selection is added.
|
||||
|
||||
Current implementation anchors:
|
||||
|
||||
- `src/main/runtime/rpc/methods/orchestration-worker-control.ts` owns local and federated
|
||||
`worker-read` routing.
|
||||
- `src/main/runtime/rpc/methods/orchestration-worker-observation.ts` validates the exact attached
|
||||
worker.
|
||||
- `src/cli/handlers/orchestration.ts` owns the current CLI request and terminal rendering.
|
||||
|
||||
### 2. Route the read to the worker server
|
||||
|
||||
For a local worker, the Run home and worker server are the same runtime.
|
||||
|
||||
For a federated worker, the Run home calls the existing federation read route on the server pinned
|
||||
to the Dispatch. The request contains the Dispatch ID, source preference, cursor, and limit—not a
|
||||
terminal handle, session ID, or transcript path chosen by the coordinator.
|
||||
|
||||
### 3. Revalidate the exact worker
|
||||
|
||||
The worker server uses the existing Dispatch attachment to verify:
|
||||
|
||||
- the exact managed pane;
|
||||
- the exact terminal/process incarnation; and
|
||||
- that the Dispatch has not been replaced, stopped, or detached.
|
||||
|
||||
This preserves the same no-cross-worker rule already used by worker lifecycle and terminal reads.
|
||||
|
||||
### 4. Resolve the pane's provider session
|
||||
|
||||
Add one runtime-owned resolver that returns a snapshot similar to:
|
||||
|
||||
```ts
|
||||
type ExactWorkerProviderSession = {
|
||||
paneKey: string
|
||||
agent: TuiAgent
|
||||
providerSession: AgentProviderSessionMetadata
|
||||
observedAt: number
|
||||
}
|
||||
```
|
||||
|
||||
The resolver may use the current runtime graph/headless retained status, but it must accept the
|
||||
Dispatch's exact pane identity as input. It must not search all sessions by working directory or
|
||||
agent type.
|
||||
|
||||
The source association is considered usable only when:
|
||||
|
||||
- it belongs to the Dispatch's exact current pane/process;
|
||||
- the hook metadata is fresh enough to belong to that process incarnation;
|
||||
- the provider session metadata passes existing normalization/canonicalization; and
|
||||
- a supported adapter can resolve an exact transcript.
|
||||
|
||||
If these checks fail under `auto`, the read falls back to terminal output.
|
||||
|
||||
Current session/status anchors:
|
||||
|
||||
- `src/shared/agent-session-resume.ts` defines normalized provider-session metadata.
|
||||
- `src/renderer/src/store/slices/agent-status.ts` maintains pane-scoped live agent status.
|
||||
- `src/shared/runtime-types.ts` carries compatible agent status in runtime/mobile graph snapshots.
|
||||
- `src/main/runtime/orca-runtime.ts` preserves provider-session metadata when it publishes those
|
||||
snapshots.
|
||||
|
||||
### 5. Read through a narrow adapter
|
||||
|
||||
Reuse the existing bounded native-chat transcript parser rather than adding another parser stack.
|
||||
Extract or wrap its pure reader behind a small orchestration adapter:
|
||||
|
||||
```ts
|
||||
type WorkerTranscriptReader = {
|
||||
provider: 'codex' | 'claude' | 'openclaude' | 'grok'
|
||||
readPage(input: ExactTranscriptRead): Promise<ExactTranscriptPage>
|
||||
}
|
||||
```
|
||||
|
||||
This is deliberately not a registry for every possible agent capability. Add an adapter only when
|
||||
an exact locator and a tested reader already exist.
|
||||
|
||||
Current reader anchors:
|
||||
|
||||
- `src/main/native-chat/transcript-watch.ts` provides bounded transcript reads/subscriptions.
|
||||
- `src/main/runtime/rpc/methods/native-chat.ts` exposes the existing reader over runtime RPC.
|
||||
- `src/main/ipc/native-chat.ts` exposes the same reader to the desktop renderer.
|
||||
|
||||
The transcript response should preserve the existing structured message/block representation and
|
||||
the supported fields that the proven reader already understands. Unknown or skipped input should
|
||||
produce parsing warnings rather than being silently presented as a complete transcript.
|
||||
|
||||
### 6. Return bounded data
|
||||
|
||||
Every path enforces:
|
||||
|
||||
- at most 50 transcript messages per page (40 by default);
|
||||
- a maximum serialized response size;
|
||||
- existing clipping/redaction rules for large tool input and output;
|
||||
- opaque projection of transcript-position fallback IDs and redaction of Dispatch capability tokens;
|
||||
- deterministic pagination; and
|
||||
- no transcript path leakage.
|
||||
|
||||
Transcript observation is read-only. A failure or unknown network result must never trigger worker
|
||||
restart, retry, stop, or Task mutation.
|
||||
|
||||
## Federation and cross-platform behavior
|
||||
|
||||
The core topology is:
|
||||
|
||||
```text
|
||||
Mac Run home
|
||||
-> authenticated connected-server RPC
|
||||
-> Windows worker server
|
||||
-> exact Windows pane/session
|
||||
-> Windows-local transcript reader
|
||||
-> bounded structured page back to Mac
|
||||
```
|
||||
|
||||
The reverse direction must work identically.
|
||||
|
||||
Platform rules:
|
||||
|
||||
- Use Node path operations only on the server that owns the path.
|
||||
- Do not normalize Windows paths on macOS or macOS/Linux paths on Windows.
|
||||
- SSH and WSL execution remain behind their owning Orca server.
|
||||
- If the exact transcript is accessible only on an SSH/WSL execution host, the worker server must
|
||||
use an existing host-aware read mechanism or fall back to terminal. Do not copy the path to the
|
||||
Run home.
|
||||
- Mixed-version capability negotiation applies only at the Orca server protocol boundary.
|
||||
- A server that does not advertise structured worker read receives the existing terminal-read RPC.
|
||||
|
||||
Federation adds one narrow additive RPC, `orchestration.federationReadOutput`. The Run home probes
|
||||
it by calling it. If the worker server returns `method_not_found`, the Run home calls the existing
|
||||
`orchestration.federationRead` terminal method and wraps that result as a labeled
|
||||
`remote_capability_unavailable` fallback. No generalized capability matrix is added.
|
||||
|
||||
## Lifecycle behavior
|
||||
|
||||
### Provider session appears after worker start
|
||||
|
||||
Hooks may report a session after the TUI becomes ready. An initial `auto` read may therefore return
|
||||
terminal output. A later first-page `auto` read may select transcript output.
|
||||
|
||||
Once a cursor is returned, that cursor remains pinned to its selected source.
|
||||
|
||||
### Provider session changes
|
||||
|
||||
Compaction, resume, or process replacement may produce a new provider session:
|
||||
|
||||
- a fresh read without a cursor may select the new exact session;
|
||||
- a cursor for the old session returns `source_changed`;
|
||||
- Orca does not merge the old and new transcripts implicitly.
|
||||
|
||||
### Orca restart
|
||||
|
||||
After restart, the worker server re-establishes the exact pane/process association using the same
|
||||
runtime graph and retained-hook mechanisms used by sidebar/session recovery.
|
||||
|
||||
- If exact identity and session still match, paging continues.
|
||||
- If process identity is uncertain, return `worker_identity_changed`.
|
||||
- If only transcript identity is unavailable, `auto` may start a new terminal page but must not
|
||||
reinterpret an old transcript cursor as a terminal cursor.
|
||||
|
||||
### Disconnect
|
||||
|
||||
A disconnected federated read is a read-only unknown result. Reissuing the same read is safe.
|
||||
No mutation request ledger, durable outbox, automatic failover, or worker replacement is needed.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
### Work package 1 — Correct the contract and types
|
||||
|
||||
- Update the orchestration checklist to replace the inaccurate claim that exact pane-to-session
|
||||
association does not exist.
|
||||
- Add the source preference, response envelope, fallback enums, and opaque cursor types.
|
||||
- Keep the existing terminal response fields compatible.
|
||||
- Add the CLI `--source` option and accept both legacy numeric and new opaque cursors.
|
||||
|
||||
Exit gate: contract tests cover legacy terminal JSON and the new labeled envelopes.
|
||||
|
||||
### Work package 2 — Runtime exact-session resolver
|
||||
|
||||
- Add a runtime method that resolves agent status for an exact Dispatch pane/process.
|
||||
- Reuse the existing graph/headless retained status and provider-session normalization.
|
||||
- Reject stale pane or process-incarnation associations.
|
||||
- Test multiple panes and multiple sessions in the same worktree and directory.
|
||||
|
||||
Exit gate: the resolver can never return a sibling pane's session.
|
||||
|
||||
### Work package 3 — Codex transcript adapter
|
||||
|
||||
- Reuse the current bounded native-chat Codex reader.
|
||||
- Add deterministic page conversion and parsing warnings.
|
||||
- Enforce entry and byte limits.
|
||||
- Produce a path-free source identity digest and cursor.
|
||||
- Add Claude only if it follows this same path without new infrastructure.
|
||||
|
||||
Exit gate: exact Codex transcript pages are stable, bounded, and contain no local path.
|
||||
|
||||
### Work package 4 — Local `worker-read`
|
||||
|
||||
- Resolve and validate the exact worker.
|
||||
- Implement `auto`, `transcript`, and `terminal`.
|
||||
- Pin the source across cursor pages.
|
||||
- Preserve the existing terminal fallback and non-JSON rendering.
|
||||
|
||||
Exit gate: local dogfood proves correct selection with several simultaneous same-directory Codex
|
||||
sessions.
|
||||
|
||||
### Work package 5 — Federated `worker-read`
|
||||
|
||||
- Add the narrow connected-server capability.
|
||||
- Route transcript resolution and reading to the worker server.
|
||||
- Wrap legacy remote terminal responses as labeled fallbacks.
|
||||
- Reject mismatched Dispatch/server/session cursors.
|
||||
- Ensure paths and internal server identity remain out of ordinary output.
|
||||
|
||||
Exit gate: physical Mac-to-Windows and Windows-to-Mac reads both pass.
|
||||
|
||||
### Work package 6 — Restart, fallback, and documentation
|
||||
|
||||
- Cover runtime restart, renderer restart, disconnect, stale status, missing hooks, unreadable
|
||||
transcript, unsupported providers, and mixed server versions.
|
||||
- Update CLI help, the orchestration skill, and the implementation checklist.
|
||||
- Dogfood the common coordinator loop using only the documented commands.
|
||||
|
||||
Exit gate: every fallback is truthful and no fallback changes worker lifecycle state.
|
||||
|
||||
## Validation plan
|
||||
|
||||
### Unit and contract tests
|
||||
|
||||
| Area | Required proof |
|
||||
| --------------------- | ------------------------------------------------------------------------------------ |
|
||||
| Exact resolution | A Dispatch resolves only its attached pane and process incarnation. |
|
||||
| No directory guessing | Two Codex sessions in the same worktree cannot cross-read. |
|
||||
| Source choice | `auto` prefers an exact supported transcript and otherwise labels terminal fallback. |
|
||||
| Explicit source | `transcript` fails truthfully when unavailable; `terminal` never probes transcript. |
|
||||
| Cursor pinning | Continued pages stay on the same source and provider session. |
|
||||
| Session replacement | An old cursor returns `source_changed`. |
|
||||
| Cursor custody | A cursor for another Dispatch is rejected. |
|
||||
| Parsing | Malformed/skipped transcript records produce bounded warnings. |
|
||||
| Limits | Entry count, block size, and total serialized response are bounded. |
|
||||
| Privacy | Responses and cursors contain no transcript path. |
|
||||
| Compatibility | Existing terminal fields and numeric cursors continue to work. |
|
||||
|
||||
Likely focused test locations:
|
||||
|
||||
- provider-session normalization and pane association tests;
|
||||
- native-chat transcript reader tests;
|
||||
- orchestration worker-control RPC tests;
|
||||
- orchestration worker CLI tests; and
|
||||
- federation protocol and physical harness tests.
|
||||
|
||||
### Local integration scenarios
|
||||
|
||||
1. Start a Codex worker and confirm the sidebar reports its provider session.
|
||||
2. Read the Dispatch and verify `source=transcript`.
|
||||
3. Start two Codex workers in the same worktree.
|
||||
4. Give them distinct prompts and verify neither read contains the other's content.
|
||||
5. Page both transcripts and verify stable source identities.
|
||||
6. replace or resume one session and verify its old cursor returns `source_changed`.
|
||||
7. Disable hooks and verify a labeled terminal fallback.
|
||||
8. Remove or make the transcript unreadable and verify a safe fallback or
|
||||
`transcript_required`, depending on the requested source.
|
||||
|
||||
### Physical federation matrix
|
||||
|
||||
| Run home | Worker server | Worker location | Required outcome |
|
||||
| -------- | ------------- | --------------- | --------------------------------------------------------------- |
|
||||
| macOS | Windows | native Windows | Exact structured page or labeled supported fallback |
|
||||
| Windows | macOS | native macOS | Exact structured page or labeled supported fallback |
|
||||
| macOS | macOS/Linux | SSH host | Exact host-aware page or terminal fallback without path leakage |
|
||||
| Windows | Windows | WSL | Exact host-aware page or terminal fallback without path leakage |
|
||||
|
||||
For both Mac/Windows directions:
|
||||
|
||||
- run multiple workers at once;
|
||||
- page beyond the first response;
|
||||
- restart the Run home;
|
||||
- restart the worker server;
|
||||
- disconnect and reconnect the server;
|
||||
- verify mixed-version fallback with one server lacking the new capability; and
|
||||
- compare the selected provider session with the sidebar/native-chat session for the same pane.
|
||||
|
||||
### Dogfood procedure
|
||||
|
||||
The dogfood is successful only if a coordinator can follow this loop without internal IDs:
|
||||
|
||||
1. Create/use a Run.
|
||||
2. Start one local worker and one connected-server worker.
|
||||
3. Wait for both starts to settle.
|
||||
4. Call `worker-read --source auto` for each Dispatch.
|
||||
5. Continue each cursor through at least two pages.
|
||||
6. Confirm output belongs to the correct prompt and machine.
|
||||
7. Trigger one fallback case.
|
||||
8. Complete both workers and confirm reads never altered lifecycle state.
|
||||
|
||||
Record:
|
||||
|
||||
- command and response;
|
||||
- chosen source and fallback reason;
|
||||
- worker server/platform;
|
||||
- provider and session match;
|
||||
- cursor behavior;
|
||||
- path-leak check;
|
||||
- restart/disconnect outcome; and
|
||||
- any agent confusion using only CLI help and the orchestration skill.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
Implementation is complete only when:
|
||||
|
||||
- `worker-read` selects an exact transcript or returns a clearly labeled terminal fallback.
|
||||
- No test or dogfood scenario reads a sibling or previous provider session.
|
||||
- A cursor never switches source or provider session silently.
|
||||
- Mac-to-Windows and Windows-to-Mac physical reads pass.
|
||||
- Runtime restart and disconnect behavior are safe and understandable.
|
||||
- Unsupported agents and mixed versions retain useful terminal output.
|
||||
- Transcript paths never leave the server that owns them.
|
||||
- Existing terminal-read clients remain compatible.
|
||||
- The common agent path remains one command with no server/session/path inputs.
|
||||
- No UI, scheduling, retry, integration tracking, or generalized provider framework is added.
|
||||
|
||||
## Explicit non-goals
|
||||
|
||||
- No dashboard or sidebar changes.
|
||||
- No coordinator chat changes.
|
||||
- No automatic worker placement, retry, replacement, or recovery.
|
||||
- No commit, test, branch, merge, or integration tracking.
|
||||
- No provider-session locking or resume orchestration.
|
||||
- No live transcript subscription in the orchestration API.
|
||||
- No universal transcript/event ontology.
|
||||
- No cross-server filesystem access from the Run home.
|
||||
- No replicated Run database or automatic Run-home failover.
|
||||
- No generalized access-control or capability framework.
|
||||
|
||||
## Main risks and mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------- |
|
||||
| Live session status is stale or renderer-owned | Resolve through a runtime-owned exact-pane snapshot and bind it to process incarnation. |
|
||||
| Two sessions share a directory | Never use directory/latest-session lookup; require exact pane/session metadata. |
|
||||
| A session changes between pages | Pin source identity in the cursor and return `source_changed`. |
|
||||
| Remote path is meaningless or sensitive | Read only on the worker server and never serialize the path. |
|
||||
| Transcript parser drops data | Preserve supported structured blocks and return parsing warnings. |
|
||||
| Transcript metadata exposes a path/credential | Make file-position IDs opaque and redact Dispatch capabilities from all structured text/payloads. |
|
||||
| Mixed server versions | Negotiate one narrow capability and fall back to existing terminal read. |
|
||||
| Full-screen terminal output remains noisy | Prefer structured output only when exact; retain terminal as the universal safety path. |
|
||||
| Scope expands into a provider platform | Ship Codex first and require proven exact association plus an existing reader for every addition. |
|
||||
|
||||
## Decision
|
||||
|
||||
Implement structured worker output as a narrow extension of `worker-read`.
|
||||
|
||||
The prerequisite is not a new sidebar or status system: Orca already tracks exact pane-scoped
|
||||
provider sessions. The work is to make that existing association available to the worker-owning
|
||||
runtime, read the transcript locally through proven readers, pin pagination to that source, and
|
||||
federate only the bounded result.
|
||||
@@ -0,0 +1,40 @@
|
||||
import { chmodSync, mkdirSync, writeFileSync } from 'node:fs'
|
||||
import path from 'node:path'
|
||||
|
||||
function escapeWindowsBatchValue(value) {
|
||||
// Why: cmd.exe expands %NAME% even inside quotes, so literal path percent signs must be doubled.
|
||||
return value.replaceAll('%', '%%')
|
||||
}
|
||||
|
||||
export function prepareDevCliTerminalWrappers({
|
||||
repoRoot,
|
||||
userDataPath,
|
||||
electronExecutable,
|
||||
platform = process.platform
|
||||
}) {
|
||||
const binDir = path.join(repoRoot, 'out', 'bin')
|
||||
const userDataBinDir = path.join(userDataPath, 'cli', 'bin')
|
||||
const cliPath = path.join(repoRoot, 'out', 'cli', 'index.js')
|
||||
mkdirSync(binDir, { recursive: true })
|
||||
mkdirSync(userDataBinDir, { recursive: true })
|
||||
|
||||
if (platform === 'win32') {
|
||||
const wrapperContent = `@echo off\r\nset "ORCA_USER_DATA_PATH=${escapeWindowsBatchValue(userDataPath)}"\r\nset "ORCA_DEV_CLI_INVOCATION=1"\r\nset "ORCA_APP_EXECUTABLE=${escapeWindowsBatchValue(electronExecutable)}"\r\nset "ORCA_APP_EXECUTABLE_NEEDS_APP_ROOT=1"\r\nnode "${escapeWindowsBatchValue(cliPath)}" %*\r\n`
|
||||
for (const targetDir of [binDir, userDataBinDir]) {
|
||||
for (const commandName of ['orca-dev.cmd', 'orca.cmd']) {
|
||||
writeFileSync(path.join(targetDir, commandName), wrapperContent, 'utf8')
|
||||
}
|
||||
}
|
||||
} else {
|
||||
const wrapperContent = `#!/usr/bin/env bash\nexport ORCA_USER_DATA_PATH=${JSON.stringify(userDataPath)}\nexport ORCA_DEV_CLI_INVOCATION=1\nexport ORCA_APP_EXECUTABLE=${JSON.stringify(electronExecutable)}\nexport ORCA_APP_EXECUTABLE_NEEDS_APP_ROOT=1\nexec node ${JSON.stringify(cliPath)} "$@"\n`
|
||||
for (const targetDir of [binDir, userDataBinDir]) {
|
||||
for (const commandName of ['orca-dev', 'orca']) {
|
||||
const wrapperPath = path.join(targetDir, commandName)
|
||||
writeFileSync(wrapperPath, wrapperContent, 'utf8')
|
||||
chmodSync(wrapperPath, 0o755)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return { binDir, userDataBinDir }
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
import { mkdtempSync, readFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import path from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { prepareDevCliTerminalWrappers } from './dev-cli-terminal-wrapper.mjs'
|
||||
|
||||
describe('dev CLI terminal wrappers', () => {
|
||||
it('writes profile-scoped Windows wrappers for worker terminals', () => {
|
||||
const root = mkdtempSync(path.join(tmpdir(), 'orca-dev-terminal-wrapper-'))
|
||||
const userDataPath = path.join(root, 'profile')
|
||||
prepareDevCliTerminalWrappers({
|
||||
repoRoot: root,
|
||||
userDataPath,
|
||||
electronExecutable: path.join(root, 'electron.exe'),
|
||||
platform: 'win32'
|
||||
})
|
||||
|
||||
const wrapper = readFileSync(path.join(userDataPath, 'cli', 'bin', 'orca-dev.cmd'), 'utf8')
|
||||
expect(wrapper).toContain(`set "ORCA_USER_DATA_PATH=${userDataPath}"`)
|
||||
expect(wrapper).toContain('set "ORCA_DEV_CLI_INVOCATION=1"')
|
||||
expect(wrapper).toContain(`node "${path.join(root, 'out', 'cli', 'index.js')}" %*`)
|
||||
expect(readFileSync(path.join(userDataPath, 'cli', 'bin', 'orca.cmd'), 'utf8')).toBe(wrapper)
|
||||
expect(readFileSync(path.join(root, 'out', 'bin', 'orca-dev.cmd'), 'utf8')).toBe(wrapper)
|
||||
expect(readFileSync(path.join(root, 'out', 'bin', 'orca.cmd'), 'utf8')).toBe(wrapper)
|
||||
})
|
||||
|
||||
it('escapes literal percent signs in every Windows batch path', () => {
|
||||
const root = path.join(mkdtempSync(path.join(tmpdir(), 'orca-dev-terminal-wrapper-')), '%repo%')
|
||||
const userDataPath = path.join(root, '%profile%')
|
||||
const electronExecutable = path.join(root, '%electron%', 'electron.exe')
|
||||
prepareDevCliTerminalWrappers({
|
||||
repoRoot: root,
|
||||
userDataPath,
|
||||
electronExecutable,
|
||||
platform: 'win32'
|
||||
})
|
||||
|
||||
const wrapper = readFileSync(path.join(userDataPath, 'cli', 'bin', 'orca-dev.cmd'), 'utf8')
|
||||
expect(wrapper).toContain(`set "ORCA_USER_DATA_PATH=${userDataPath.replaceAll('%', '%%')}"`)
|
||||
expect(wrapper).toContain(
|
||||
`set "ORCA_APP_EXECUTABLE=${electronExecutable.replaceAll('%', '%%')}"`
|
||||
)
|
||||
expect(wrapper).toContain(
|
||||
`node "${path.join(root, 'out', 'cli', 'index.js').replaceAll('%', '%%')}" %*`
|
||||
)
|
||||
expect(readFileSync(path.join(root, 'out', 'bin', 'orca-dev.cmd'), 'utf8')).toBe(wrapper)
|
||||
expect(readFileSync(path.join(root, 'out', 'bin', 'orca.cmd'), 'utf8')).toBe(wrapper)
|
||||
})
|
||||
|
||||
it('writes executable-style POSIX wrappers with the same profile identity', () => {
|
||||
const root = mkdtempSync(path.join(tmpdir(), 'orca-dev-terminal-wrapper-'))
|
||||
const userDataPath = path.join(root, 'profile')
|
||||
prepareDevCliTerminalWrappers({
|
||||
repoRoot: root,
|
||||
userDataPath,
|
||||
electronExecutable: path.join(root, 'electron'),
|
||||
platform: 'linux'
|
||||
})
|
||||
|
||||
const wrapper = readFileSync(path.join(userDataPath, 'cli', 'bin', 'orca-dev'), 'utf8')
|
||||
expect(wrapper).toContain(`export ORCA_USER_DATA_PATH=${JSON.stringify(userDataPath)}`)
|
||||
expect(wrapper).toContain('export ORCA_DEV_CLI_INVOCATION=1')
|
||||
expect(wrapper).toContain(
|
||||
`exec node ${JSON.stringify(path.join(root, 'out', 'cli', 'index.js'))}`
|
||||
)
|
||||
expect(readFileSync(path.join(userDataPath, 'cli', 'bin', 'orca'), 'utf8')).toBe(wrapper)
|
||||
expect(readFileSync(path.join(root, 'out', 'bin', 'orca-dev'), 'utf8')).toBe(wrapper)
|
||||
expect(readFileSync(path.join(root, 'out', 'bin', 'orca'), 'utf8')).toBe(wrapper)
|
||||
})
|
||||
})
|
||||
@@ -25,6 +25,7 @@ describe('orca-dev package bin', () => {
|
||||
`fs.writeFileSync(${JSON.stringify(outputPath)}, JSON.stringify({`,
|
||||
' argv: process.argv.slice(2),',
|
||||
' userDataPath: process.env.ORCA_USER_DATA_PATH,',
|
||||
' devCliInvocation: process.env.ORCA_DEV_CLI_INVOCATION,',
|
||||
' appExecutable: process.env.ORCA_APP_EXECUTABLE',
|
||||
'}));'
|
||||
].join('\n'),
|
||||
@@ -47,6 +48,7 @@ describe('orca-dev package bin', () => {
|
||||
expect(JSON.parse(readFileSync(outputPath, 'utf8'))).toEqual({
|
||||
argv: ['--help'],
|
||||
userDataPath: path.join(root, 'user-data'),
|
||||
devCliInvocation: '1',
|
||||
appExecutable: path.join(root, 'Electron')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
import { spawnSync } from 'node:child_process'
|
||||
import { accessSync, constants, existsSync, realpathSync, statSync } from 'node:fs'
|
||||
import path from 'node:path'
|
||||
import { prepareDevCliTerminalWrappers } from './dev-cli-terminal-wrapper.mjs'
|
||||
|
||||
const scriptPath = realpathSync(import.meta.filename)
|
||||
const scriptDir = path.dirname(scriptPath)
|
||||
@@ -16,6 +17,8 @@ if (!existsSync(cliEntry)) {
|
||||
}
|
||||
|
||||
process.env.ORCA_USER_DATA_PATH = process.env.ORCA_DEV_USER_DATA_PATH ?? getDefaultDevUserDataPath()
|
||||
// Why: custom dev profiles do not necessarily contain "orca-dev" in their path; carry explicit provenance into the CLI.
|
||||
process.env.ORCA_DEV_CLI_INVOCATION = '1'
|
||||
|
||||
const electronExecutable = getElectronExecutable()
|
||||
if (!process.env.ORCA_APP_EXECUTABLE && isRunnableFile(electronExecutable)) {
|
||||
@@ -23,6 +26,13 @@ if (!process.env.ORCA_APP_EXECUTABLE && isRunnableFile(electronExecutable)) {
|
||||
process.env.ORCA_APP_EXECUTABLE_NEEDS_APP_ROOT = '1'
|
||||
}
|
||||
|
||||
// Why: headless `orca-dev serve` skips the Electron dev runner that normally installs terminal CLI shims.
|
||||
prepareDevCliTerminalWrappers({
|
||||
repoRoot,
|
||||
userDataPath: process.env.ORCA_USER_DATA_PATH,
|
||||
electronExecutable: process.env.ORCA_APP_EXECUTABLE ?? electronExecutable
|
||||
})
|
||||
|
||||
const result = spawnSync(process.execPath, [cliEntry, ...process.argv.slice(2)], {
|
||||
stdio: 'inherit',
|
||||
env: process.env
|
||||
|
||||
@@ -29,10 +29,14 @@ describe('orchestration skill guidance', () => {
|
||||
const skill = readSkill()
|
||||
const toolBoundary = getSection(skill, 'Tool Boundary')
|
||||
|
||||
expect(toolBoundary).toContain(
|
||||
'must create Orca runtime state with `orca orchestration task-create` and `orca orchestration dispatch --inject`'
|
||||
expect(toolBoundary).toContain('must create or bind a Run')
|
||||
expect(toolBoundary).toContain('create the Task with `orca orchestration task-create`')
|
||||
expect(toolBoundary).toContain('preferred `orca orchestration worker-start` composition')
|
||||
expect(toolBoundary).toContain('low-level `orca orchestration dispatch --inject` path')
|
||||
expect(toolBoundary).not.toContain('or `orca orchestration run`')
|
||||
expect(skill).toContain(
|
||||
'`coordinator-start`, `coordinator-stop`, `run`, and `run-stop` are retired scheduler commands'
|
||||
)
|
||||
expect(toolBoundary).toContain('or `orca orchestration run`')
|
||||
expect(toolBoundary).toContain(
|
||||
'Do not substitute non-Orca subagent tools, generic agent-spawn APIs, or chat-only parallel worker features'
|
||||
)
|
||||
@@ -47,6 +51,20 @@ describe('orchestration skill guidance', () => {
|
||||
)
|
||||
})
|
||||
|
||||
it('teaches the hard cutover without reviving a legacy executor', () => {
|
||||
const skill = readSkill()
|
||||
const migration = getSection(skill, 'Contract Migration')
|
||||
|
||||
expect(migration).toContain('hard cutover')
|
||||
expect(migration).toContain('effectsApplied')
|
||||
expect(migration).toContain('skills get orchestration --full')
|
||||
expect(migration).toContain('Do not retry the rejected command unchanged')
|
||||
expect(migration).toContain('no longer supervised')
|
||||
expect(migration).toContain('task-list --run run_legacy_local')
|
||||
expect(migration).toContain('Read-only inspection never consumes legacy mail')
|
||||
expect(migration).toContain('does not run a legacy scheduler, translate old writes, or drain')
|
||||
})
|
||||
|
||||
it('treats long-running worker waits as liveness checkpoints, not failures', () => {
|
||||
const skill = readSkill()
|
||||
|
||||
@@ -213,13 +231,13 @@ describe('orchestration skill guidance', () => {
|
||||
const messaging = getSection(skill, 'Messaging')
|
||||
const workerTerminals = getSection(skill, 'Worker Terminals')
|
||||
const agentFirstExample = workerTerminals.match(
|
||||
/```bash\norca worktree create --name <task-name> --agent codex --json\n[\s\S]*?```/
|
||||
/```bash\norca worktree create --name <task-name> --agent codex --setup run --json\n[\s\S]*?```/
|
||||
)?.[0]
|
||||
|
||||
expect(workerTerminals).toContain('For an allowed new worktree, use agent-first:')
|
||||
expect(workerTerminals).toContain('fallback shell + agent pair')
|
||||
expect(workerTerminals).toContain(
|
||||
'Repo setup or default-terminal settings may still add tabs or splits'
|
||||
'repo setup and default-terminal settings may add intentional tabs or splits'
|
||||
)
|
||||
expect(workerTerminals).toContain('without configured default tabs')
|
||||
expect(workerTerminals).toContain(
|
||||
@@ -229,12 +247,11 @@ describe('orchestration skill guidance', () => {
|
||||
expect(workerTerminals).not.toContain('ends with **one** agent tab')
|
||||
expect(agentFirstExample).toBeDefined()
|
||||
expect(agentFirstExample).not.toContain('orca terminal list')
|
||||
expect(agentFirstExample).toContain('agentTerminalHandle')
|
||||
expect(agentFirstExample).toContain('startupTerminal.handle')
|
||||
expect(messaging).toContain(
|
||||
'Use `startupTerminal.handle` from the create response when present'
|
||||
)
|
||||
expect(messaging).toContain('continue with the replacement only')
|
||||
expect(messaging).toContain('it does not remotely wake another terminal')
|
||||
expect(messaging).toContain('Prefer `agentTerminalHandle` from the create response')
|
||||
expect(messaging).toContain('Continue with the replacement handle only')
|
||||
expect(messaging).toContain('never writes to terminal input or remotely wakes another terminal')
|
||||
expect(messaging).toContain('Use `orchestration dispatch --inject` to deliver a tracked task')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
import { execFileSync, spawn } from 'node:child_process'
|
||||
import { createHash } from 'node:crypto'
|
||||
import {
|
||||
chmodSync,
|
||||
cpSync,
|
||||
existsSync,
|
||||
lstatSync,
|
||||
@@ -17,6 +16,7 @@ import {
|
||||
import net from 'node:net'
|
||||
import { createRequire } from 'node:module'
|
||||
import path from 'node:path'
|
||||
import { prepareDevCliTerminalWrappers } from './dev-cli-terminal-wrapper.mjs'
|
||||
|
||||
// Why: Electron-based hosts (e.g. Claude Code, VS Code) set
|
||||
// ELECTRON_RUN_AS_NODE=1 in their terminal environment. If this leaks into
|
||||
@@ -314,35 +314,12 @@ function getDevUserDataPath() {
|
||||
}
|
||||
|
||||
function prepareDevCliWrapper() {
|
||||
const binDir = path.join(repoRoot, 'out', 'bin')
|
||||
mkdirSync(binDir, { recursive: true })
|
||||
const userDataPath = getDevUserDataPath()
|
||||
const userDataBinDir = path.join(userDataPath, 'cli', 'bin')
|
||||
const cliPath = path.join(repoRoot, 'out', 'cli', 'index.js')
|
||||
const electronBin = getElectronExecutable()
|
||||
|
||||
if (process.platform === 'win32') {
|
||||
writeFileSync(
|
||||
path.join(binDir, 'orca-dev.cmd'),
|
||||
`@echo off\r\nset "ORCA_USER_DATA_PATH=${userDataPath}"\r\nset "ORCA_APP_EXECUTABLE=${electronBin}"\r\nset "ORCA_APP_EXECUTABLE_NEEDS_APP_ROOT=1"\r\nnode "${cliPath}" %*\r\n`,
|
||||
'utf8'
|
||||
)
|
||||
} else {
|
||||
const wrapperContent = `#!/usr/bin/env bash\nexport ORCA_USER_DATA_PATH=${JSON.stringify(userDataPath)}\nexport ORCA_APP_EXECUTABLE=${JSON.stringify(electronBin)}\nexport ORCA_APP_EXECUTABLE_NEEDS_APP_ROOT=1\nexec node ${JSON.stringify(cliPath)} "$@"\n`
|
||||
const wrapperPath = path.join(binDir, 'orca-dev')
|
||||
writeFileSync(wrapperPath, wrapperContent, 'utf8')
|
||||
chmodSync(wrapperPath, 0o755)
|
||||
|
||||
mkdirSync(userDataBinDir, { recursive: true })
|
||||
for (const commandName of ['orca-dev', 'orca']) {
|
||||
const userDataWrapperPath = path.join(userDataBinDir, commandName)
|
||||
// Why: dev Orca terminals prepend this directory to PATH; refreshing the
|
||||
// `orca` alias prevents stale global/userData wrappers from hijacking
|
||||
// Orca-owned commands such as `orca claude-teams`.
|
||||
writeFileSync(userDataWrapperPath, wrapperContent, 'utf8')
|
||||
chmodSync(userDataWrapperPath, 0o755)
|
||||
}
|
||||
}
|
||||
const { binDir } = prepareDevCliTerminalWrappers({
|
||||
repoRoot,
|
||||
userDataPath,
|
||||
electronExecutable: getElectronExecutable()
|
||||
})
|
||||
|
||||
process.env.PATH = `${binDir}${path.delimiter}${process.env.PATH ?? ''}`
|
||||
console.log(`[orca-dev] Prepared wrapper in ${binDir}`)
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
+143
-31
@@ -22,7 +22,7 @@ Use this skill when coordination state matters. For lightweight terminal prompts
|
||||
|
||||
## Tool Boundary
|
||||
|
||||
If a task says to use Orca orchestration, the coordinator must create Orca runtime state with `orca orchestration task-create` and `orca orchestration dispatch --inject` or `orca orchestration run`.
|
||||
If a task says to use Orca orchestration, the coordinator must create or bind a Run, create the Task with `orca orchestration task-create`, then attach the worker with either the preferred `orca orchestration worker-start` composition or the low-level `orca orchestration dispatch --inject` path.
|
||||
|
||||
Do not substitute non-Orca subagent tools, generic agent-spawn APIs, or chat-only parallel worker features. Those may create useful workers, but they do not create Orca task/dispatch provenance, injected lifecycle preambles, `worker_done` authority, or decision gates.
|
||||
|
||||
@@ -51,9 +51,35 @@ Do not use orchestration merely because the user says "hand off", "handoff", "ha
|
||||
- The orchestration experimental feature must be enabled in Settings > Experimental.
|
||||
- `orca orchestration` commands are RPC calls to the running Orca runtime.
|
||||
|
||||
## Contract Migration
|
||||
|
||||
Orca uses a hard cutover for orchestration mutations. It does not run a legacy scheduler, translate old writes, or drain pre-upgrade orchestration work.
|
||||
|
||||
If a command returns `orchestration_migration_required`, `run_required`, or a lifecycle validation error with `nextCommandArgs`:
|
||||
|
||||
1. Confirm `effectsApplied` is `false`.
|
||||
2. Using the same CLI executable that returned the error, run the returned arguments: `skills get orchestration --full`.
|
||||
3. Read the guide completely. Do not retry the rejected command unchanged.
|
||||
4. Create or bind a lightweight Run, then restart the work using Run -> Task -> `worker-start`.
|
||||
5. Inspect any pre-upgrade terminal before creating replacement work.
|
||||
|
||||
The arguments intentionally omit an executable name so this works with `orca`, `orca-ide`, `orca-dev`, or another configured Orca CLI command.
|
||||
|
||||
Pre-upgrade terminals and agents are not killed during upgrade, but they are no longer supervised: old heartbeat, question, completion, scheduler, and mutation calls are rejected before effects. Legacy database rows remain available only for explicit inspection:
|
||||
|
||||
```bash
|
||||
orca orchestration run-list --json
|
||||
orca orchestration run-show --id run_legacy_local --json
|
||||
orca orchestration task-list --run run_legacy_local --json
|
||||
orca orchestration inbox --full --json
|
||||
orca orchestration check --terminal <legacy_handle> --peek --json
|
||||
```
|
||||
|
||||
Read-only inspection never consumes legacy mail. Do not use actionable `check`, acknowledgment, send, retry, or task updates against the legacy Run.
|
||||
|
||||
## Ownership
|
||||
|
||||
Orchestration messages and tasks are runtime-global. Lifecycle authority comes from the payload `taskId` + `dispatchId` of the active dispatch, verified against the dispatched pane. Terminal handles are routing metadata — a pane can receive a new handle after restart — so never accept or reject lifecycle provenance by comparing handles. Send `worker_done` and `heartbeat` from the worker's own terminal; the runtime ignores them when sent from a different pane.
|
||||
New orchestration messages and tasks belong to one explicitly bound Run. A Run is only a durable namespace and coordinator inbox; it never schedules or places workers. Lifecycle authority comes from the active Dispatch, and terminal handles remain routing metadata rather than durable identity. Send `worker_done` and `heartbeat` from the worker's own terminal; Orca routes them to that Dispatch's Run.
|
||||
|
||||
Classify inherited context before sending lifecycle messages:
|
||||
|
||||
@@ -78,36 +104,39 @@ orca orchestration dispatch-show --task <task_id> --json
|
||||
## Messaging
|
||||
|
||||
```bash
|
||||
orca orchestration send --to <handle|@group> --subject <text> [--from <handle>] [--body <text>] [--type <type>] [--priority <level>] [--thread-id <id>] [--payload <json>] [--json]
|
||||
orca orchestration check [--terminal <handle>] [--unread|--peek|--all] [--types <type,...>] [--inject] [--wait] [--timeout-ms <n>] [--json]
|
||||
orca orchestration send --subject <text> [--to <run:id|dispatch:id|legacy_handle>] [--from <handle>] [--body <text>] [--type <type>] [--priority <level>] [--thread-id <id>] [--payload <json>] [--json]
|
||||
orca orchestration check [--terminal <handle>] [--ack <delivery_id>] [--peek|--all] [--types <type,...>] [--format] [--wait] [--timeout-ms <n>] [--json]
|
||||
orca orchestration reply --id <msg_id> --body <text> [--from <handle>] [--json]
|
||||
orca orchestration ask --to <handle> --question <text> [--options <csv>] [--timeout-ms <n>] [--from <handle>] [--json]
|
||||
orca orchestration ask (--question <text>|--resume <msg_id>) [--options <csv>] [--timeout-ms <n>] [--from <handle>] [--json]
|
||||
orca orchestration inbox [--limit <n>] [--json]
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- Omit `--from` unless impersonating another terminal; Orca auto-resolves it from the current terminal.
|
||||
- `check` and `check --unread` return unread matches and mark them read. Use `--peek` for unread matches without consuming them; use `--all` for read and unread history without consuming anything. If an older CLI rejects `--peek` as an unknown flag, use `--all` and filter unread rows yourself.
|
||||
- Message **one** live agent handle per worker. Use `startupTerminal.handle` from the create response when present; if it is missing or later returns `terminal_handle_stale`, re-resolve with `orca terminal list --worktree ... --json` and continue with the replacement only.
|
||||
- `orca orchestration check --unread --inject --json` renders unread mail for the agent terminal that runs it; it does not remotely wake another terminal. Use `orchestration dispatch --inject` to deliver a tracked task, or `terminal send` when an existing agent needs a free-form prompt.
|
||||
- While supervising workers manually, use `check --wait --types worker_done,escalation,decision_gate --timeout-ms <n>` instead of sleep/poll loops. Reply to `decision_gate` messages with `orca orchestration reply --id <msg_id> --body <answer> --json`, then keep waiting.
|
||||
- A coordinator `check` returns the bound Run's oldest FIFO Delivery (up to 50 messages) and replays that exact batch until `--ack <delivery_id>`. Process every message before acknowledging; `check --ack <id> --wait` acknowledges, checks, and waits in one operation.
|
||||
- Use `--peek` and `--all` only for read-only history/debugging. Type filters decide when a waiter wakes; the returned actionable Delivery is still the oldest full batch.
|
||||
- Use `dispatch:<id>` for coordinator guidance to one supervised worker. Orca routes that stable address locally or through the connected-server relay; do not substitute a remote terminal handle.
|
||||
- Terminal handles remain appropriate for low-level pre-Dispatch messaging. Prefer `agentTerminalHandle` from the create response, fall back to `startupTerminal.handle` for older runtimes, then re-resolve with `orca terminal list --worktree ... --json` if missing or stale. Continue with the replacement handle only; never dual-send to old and new handles.
|
||||
- `orca orchestration check --peek --format --json` returns locally formatted unread mail without consuming it; it never writes to terminal input or remotely wakes another terminal. Use `orchestration dispatch --inject` to deliver a tracked task, or `terminal send` when an existing agent needs a free-form prompt.
|
||||
- While supervising workers manually, use `check --wait --types worker_done,escalation,question --timeout-ms <n>` instead of sleep/poll loops. Process the whole Delivery, reply to `question` messages with `orca orchestration reply --id <msg_id> --body <answer> --json`, then acknowledge and keep waiting.
|
||||
- Treat a `check --wait` timeout or `{count:0}` as a checkpoint, not a worker failure. Long coding tasks routinely run 15-60 minutes; keep using rolling waits unless you receive `worker_done`/`escalation`, the terminal exits or disappears, or the user explicitly asks you to stop.
|
||||
- Heartbeats and visible terminal activity mean the worker is alive, not done. Do not stop, close, kill, or restart a worker just because it has not produced a completion message yet.
|
||||
- Use `ask` when a worker needs a blocking answer from the coordinator; it waits for the reply and returns the answer directly.
|
||||
- `check --wait` returns one message at a time. If N workers may finish together, loop N times and dispatch newly ready tasks after each completion.
|
||||
- Use `ask` when a worker needs a blocking answer from the coordinator; it defaults to the active Dispatch's Run. Timeout or disconnect leaves the question pending, so resume by its original message ID instead of asking again.
|
||||
- `check --wait` returns one bounded Delivery, not every future completion. Process every message, acknowledge it, then keep waiting until every expected Dispatch settles.
|
||||
- Group addresses include `@all`, `@idle`, `@claude`, `@codex`, `@opencode`, `@gemini`, `@droid`, `@grok`, `@cursor`, and `@worktree:<id>`.
|
||||
- Message types include `status`, `dispatch`, `worker_done`, `merge_ready`, `escalation`, `handoff`, `decision_gate`, and `heartbeat`.
|
||||
- Message types include `status`, `dispatch`, `worker_done`, `merge_ready`, `escalation`, `handoff`, `question`, `decision_gate` (legacy/gates), and `heartbeat`.
|
||||
- Use group addresses only for messages that are genuinely useful to many terminals, such as `status` broadcasts or intentional fan-out questions. Do not send dispatch lifecycle messages to groups.
|
||||
- `worker_done` must target the concrete coordinator handle from the live preamble. It is completion authority for one dispatch; group fanout would create false lifecycle mail in unrelated terminals.
|
||||
- `worker_done` belongs to the active Dispatch and defaults to its Run mailbox; never target a group.
|
||||
- A valid `worker_done` for the active `taskId` + `dispatchId` marks the task and dispatch completed automatically. Do not follow it with `task-update --status completed`; reserve manual updates for explicit recovery or overrides.
|
||||
- `heartbeat` is also dispatch-scoped. Send it only to the concrete coordinator handle with both `taskId` and `dispatchId`; use `status` for broad progress updates.
|
||||
- `heartbeat` is also Dispatch-scoped. Include both IDs and omit `--to` so Orca uses the owning Run; use `status` for broad progress updates.
|
||||
|
||||
## Tasks And Dispatch
|
||||
|
||||
A task is the work item, a dispatch assigns it to a terminal, and a gate blocks progress until a coordinator or user decision is recorded.
|
||||
A Run is the namespace/inbox, a Task is the work item, and a Dispatch assigns one Task attempt to a terminal. Create or bind a Run once before the common loop.
|
||||
|
||||
```bash
|
||||
orca orchestration run-create --objective <text> --json
|
||||
orca orchestration task-create --spec <text> [--deps <json_array>] [--parent <task_id>] [--json]
|
||||
orca orchestration task-list [--status <status>] [--ready] [--brief] [--json]
|
||||
orca orchestration task-update --id <task_id> --status <status> [--result <json>] [--json]
|
||||
@@ -124,19 +153,97 @@ Dispatch rules:
|
||||
- After 3 consecutive failures on one task, the dispatch context circuit-breaks and the task is marked failed.
|
||||
- Use `task-list --brief --json` for coordinator sweeps; it collapses whitespace and caps each echoed spec at 160 characters (`spec_truncated` marks shortened rows). Omit `--brief` when the full spec is required, or when an older CLI rejects it as an unknown flag.
|
||||
|
||||
## Gates And Coordinator
|
||||
## Preferred Supervised Worker Loop
|
||||
|
||||
Use `worker-start` for the normal supervised path. It composes the existing worktree, terminal, readiness, and dispatch primitives while returning exact created/reused effects. Agents still choose placement and concurrency; Orca does not schedule workers or infer conflicts.
|
||||
|
||||
Create the Run and every independent Task first, then start all independent workers before waiting:
|
||||
|
||||
```bash
|
||||
orca orchestration run-create --objective "<objective>" --json
|
||||
orca orchestration task-create --spec "<worker A task>" --json
|
||||
orca orchestration task-create --spec "<worker B task>" --json
|
||||
orca orchestration worker-start --task <task_a> --worktree current --agent codex --json
|
||||
orca orchestration worker-start --task <task_b> --worktree current --agent claude --json
|
||||
```
|
||||
|
||||
`current` and exact existing worktrees create a fresh agent terminal and do not rerun setup. Reuse an existing agent only with `--terminal <handle>`.
|
||||
|
||||
For a new worktree, setup runs by default and agent-first creation reuses the returned startup agent terminal:
|
||||
|
||||
```bash
|
||||
orca orchestration worker-start --task <task_id> --worktree new-child --name <name> --agent codex --setup run --json
|
||||
# Independent/top-level:
|
||||
orca orchestration worker-start --task <task_id> --worktree new-top-level --name <name> --agent codex --setup run --json
|
||||
```
|
||||
|
||||
Setup normally starts alongside the agent. Only a repository explicitly configured with `wait-for-setup` delays agent launch until setup succeeds. Use `--setup skip` or `--setup inherit` only for a concrete reason.
|
||||
|
||||
Read the returned receipt before continuing: `ready` plus setup `running` is normal for start-immediately, while wait-for-setup returns setup `succeeded` before accepting task input. A failed or unknown start exits nonzero; inspect its `stage`, `effects`, and `residualResources` instead of guessing or automatically retrying. A wait-for-setup timeout can honestly leave setup `running`, which is not proof of failure.
|
||||
|
||||
To run the worker on another connected Orca server, add `--on <saved-environment>`. The Run and Tasks remain authoritative on the current server; later commands route by Dispatch ID, so never repeat `--on`:
|
||||
|
||||
```bash
|
||||
# Mac Run home -> Windows worker (the reverse is identical from a Windows Run home)
|
||||
orca orchestration worker-start --task <task_id> --on windows --worktree new-top-level --repo <exact_remote_repo_selector> --name <name> --agent codex --setup run --json
|
||||
orca orchestration worker-show --dispatch <dispatch_id> --json
|
||||
orca orchestration worker-read --dispatch <dispatch_id> --limit 50 --json
|
||||
orca orchestration send --to dispatch:<dispatch_id> --subject "Follow-up" --body "<attempt-specific guidance>" --json
|
||||
```
|
||||
|
||||
Remote `current` and `new-child` are intentionally invalid because those words are ambiguous across servers. Use an exact discovered remote worktree selector or `new-top-level` with an explicit remote repo selector.
|
||||
|
||||
The follow-up is structured inbox mail, not prompt injection. The worker's next
|
||||
`orchestration check` receives it even when the Dispatch is on another connected Orca server.
|
||||
|
||||
`worker-read` defaults to `--source auto`: Orca returns the exact hook-reported Codex, Claude, OpenClaude, or Grok transcript when it can prove the worker session, otherwise it returns bounded terminal output with `source: "terminal"` and a typed `fallbackReason`. Continue with the returned top-level `cursor`; it stays pinned to that exact source. If Orca reports `source_changed`, start a fresh read without the old cursor. Never supply or guess a provider session ID or transcript path.
|
||||
|
||||
Wait until every expected Dispatch settles, not for a fixed number of batches:
|
||||
|
||||
```bash
|
||||
orca orchestration check --wait --types worker_done,escalation,question --timeout-ms 900000 --json
|
||||
# Process every message in the returned Delivery, then atomically ack and continue:
|
||||
orca orchestration check --ack <delivery_id> --wait --types worker_done,escalation,question --timeout-ms 900000 --json
|
||||
```
|
||||
|
||||
Workers report exactly once using the IDs and capability injected by Orca; they do not supply Run/server/terminal identity:
|
||||
|
||||
```bash
|
||||
orca orchestration send --type worker_done --subject "<status>" --body "<what changed, findings, and what remains>" --task-id <task_id> --dispatch-id <dispatch_id> --outcome succeeded --files-modified "path/a,path/b" --json
|
||||
# On failure, use --outcome failed; never encode failure only in prose.
|
||||
```
|
||||
|
||||
A worker question defaults to its owning Run. Timeout leaves it pending:
|
||||
|
||||
```bash
|
||||
orca orchestration ask --question "<question>" --options "yes,no" --timeout-ms 600000 --json
|
||||
orca orchestration ask --resume <message_id> --timeout-ms 600000 --json
|
||||
# Coordinator:
|
||||
orca orchestration reply --id <message_id> --body "<answer>" --json
|
||||
```
|
||||
|
||||
Recovery is conditional, never a fixed destructive sequence:
|
||||
|
||||
- `worker-show --dispatch <id>` says `ready`: keep waiting or read bounded output.
|
||||
- It proves `failed` or `stopped`: start a replacement with `worker-start --task <task> --retry-of <id>` plus an explicit `--on`/`--worktree` and `--agent`/`--terminal` choice. Retry does not silently inherit placement.
|
||||
- It remains `outcome_unknown`: either `worker-stop --dispatch <id>` and inspect again, or explicitly `worker-abandon --dispatch <id>` while accepting that resources may still be live. Abandon performs no remote, process, or filesystem action.
|
||||
- `worker-stop` closes only the exact supervised agent terminal. It never deletes the worktree, setup terminal, configured tabs, or unrelated processes.
|
||||
|
||||
Low-level `worktree create`, `terminal create`, and `dispatch --inject` remain valid recipes for custom argv or topology that `worker-start` does not express.
|
||||
|
||||
## Gates And Legacy Inspection
|
||||
|
||||
```bash
|
||||
orca orchestration gate-create --task <task_id> --question <text> [--options <json_array>] [--json]
|
||||
orca orchestration gate-resolve --id <gate_id> --resolution <text> [--json]
|
||||
orca orchestration gate-list [--task <task_id>] [--status <status>] [--json]
|
||||
orca orchestration run --spec <text> [--from <handle>] [--poll-interval-ms <n>] [--max-concurrent <n>] [--worktree <selector>] [--json]
|
||||
orca orchestration run-stop [--json]
|
||||
```
|
||||
|
||||
`run` returns immediately with a run ID. Query progress with `task-list`. Use `ask` for worker-to-coordinator questions; it creates a `decision_gate` message that the coordinator answers with `reply`. Use `gate-create` only for coordinator-managed task DAG decisions, not for answering a worker's `ask`.
|
||||
Use `ask` for worker-to-coordinator questions; it creates a `question` message that the coordinator answers with `reply`. Use `gate-create` only for coordinator-managed task DAG decisions, not for answering a worker's `ask`.
|
||||
|
||||
Recovery only: `orca orchestration reset --tasks|--messages|--all --json` clears runtime-global orchestration state. Do not run it during active coordination unless explicitly abandoning that state.
|
||||
`coordinator-start`, `coordinator-stop`, `run`, and `run-stop` are retired scheduler commands. They perform no effects and return the current-skill recovery action. They are not aliases for lightweight Run creation or binding.
|
||||
|
||||
Recovery only: `orca orchestration reset --tasks|--messages|--all --json` clears the selected local orchestration database state. Do not run it during active coordination unless explicitly abandoning that state.
|
||||
|
||||
## Full Handoffs
|
||||
|
||||
@@ -151,7 +258,7 @@ Do not run `orca orchestration task-create`, `orca orchestration dispatch --inje
|
||||
New top-level worktree handoff:
|
||||
|
||||
```bash
|
||||
orca worktree create --name <task-name> --no-parent --agent codex --prompt "<task brief>" --json
|
||||
orca worktree create --name <task-name> --no-parent --agent codex --prompt "<task brief>" --setup run --json
|
||||
```
|
||||
|
||||
Before creating a new worktree from an active feature branch, decide and state whether the desired Orca lineage is child or top-level. Use child worktree lineage only when the new work is conceptually stacked under or dependent on the active worktree. For independent repo-wide fixes, standalone feature work, or unrelated follow-up tasks, create a top-level worktree with `--no-parent`.
|
||||
@@ -166,12 +273,14 @@ Custom Codex model/effort handoff:
|
||||
|
||||
`orca worktree create --agent codex --prompt ...` launches the known Codex agent but does not accept Codex-specific `--model` or `-c model_reasoning_effort=...` arguments. When the user asks for a specific Codex model or effort, create the independent worktree first, launch Codex with the requested command in that worktree, wait only for TUI readiness if prompt delivery would otherwise race startup, send the prompt, and stop.
|
||||
|
||||
The two-step custom-argv path cannot enforce a repository's explicit `wait-for-setup` startup policy because the later `terminal create` is not the startup owned by `worktree create`. Use it only when the repository starts agents immediately. If the repository requires `wait-for-setup`, use an agent-first configured launcher that can preserve sequencing, or stop and ask rather than silently bypassing the policy.
|
||||
|
||||
Note: when no repo default-terminal configuration supplies a primary terminal, bare create opens a fallback shell before `terminal create` adds the agent. Configured default tabs are materialized instead and may run real commands. Prefer `--agent` whenever custom argv is not required. With the two-step path, target only the agent handle; close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell.
|
||||
|
||||
Use the exact full `<repo-id>::<path>` worktree id returned by `orca worktree create --json`; a bare repo id cannot target the new worktree.
|
||||
|
||||
```bash
|
||||
orca worktree create --name <task-name> --no-parent --json
|
||||
orca worktree create --name <task-name> --no-parent --setup run --json
|
||||
orca terminal create --worktree id:<newFullWorktreeId> --title <task-name> --command 'codex --model gpt-5.5 -c model_reasoning_effort="xhigh"' --json
|
||||
orca terminal wait --terminal <handle> --for tui-idle --timeout-ms 60000 --json
|
||||
orca terminal send --terminal <handle> --text "<task brief>" --enter --json
|
||||
@@ -195,17 +304,19 @@ Reuse an idle agent in the required worktree only if the prompt allows reuse; ot
|
||||
|
||||
When a new worktree is allowed, use child lineage for isolated work that is stacked under or dependent on the active worktree, and use `--no-parent` when it is not stacked. Decide the Git base separately: `--no-parent` makes the worktree top-level in Orca, while omitted `--base-branch` uses the repo default base.
|
||||
|
||||
For every new worktree, pass `--setup run` so any configured repository setup hook runs. This does not mean waiting for setup before agent launch: preserve the repository's startup policy, whose default starts setup and the agent side by side. Use `--setup skip` or `--setup inherit` only when there is a concrete task-specific reason, and state that reason before creating the worktree. This rule does not rerun setup for current or existing worktrees.
|
||||
|
||||
```bash
|
||||
orca worktree create --name <task-name> --agent codex --json
|
||||
orca worktree create --name <task-name> --agent codex --setup run --json
|
||||
# or: --agent claude | omp | pi | grok | ...
|
||||
# Read <handle> from startupTerminal.handle in the create response.
|
||||
# Read <handle> from agentTerminalHandle, falling back to startupTerminal.handle.
|
||||
orca terminal wait --terminal <handle> --for tui-idle --timeout-ms 60000 --json
|
||||
orca orchestration dispatch --task <task_id> --to <handle> --inject --json
|
||||
```
|
||||
|
||||
For new-worktree workers, read the id and `startupTerminal.handle` from `worktree create`. Use that as the sole worker handle when present; otherwise use `terminal list` to resolve the agent handle. Omit `--repo` only inside an Orca-managed worktree; otherwise pass `--repo <selector>`.
|
||||
For new-worktree workers, read the id and `agentTerminalHandle` from `worktree create`, falling back to `startupTerminal.handle` for older runtimes. Use that as the sole worker handle when present; otherwise use `terminal list` to resolve the agent handle. Omit `--repo` only inside an Orca-managed worktree; otherwise pass `--repo <selector>`.
|
||||
|
||||
**For an allowed new worktree, use agent-first:** `--agent` reveals the new worktree and launches the selected agent **in its first terminal**, without adding a separate fallback shell for that worker. Repo setup or default-terminal settings may still add tabs or splits. Do **not** run bare `worktree create` and then `terminal create --command <agent>` for the same worker when agent-first create is available: without configured default tabs, that two-step path leaves a fallback shell + agent pair. Only use it when custom agent argv is required (for example Codex model/effort flags) or when an older CLI rejects `--agent`; if you must, message only the agent handle. Configured default tabs are intentional surfaces, so close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell. Do not run `worktree create` when the task must stay in the current worktree.
|
||||
**For an allowed new worktree, use agent-first:** `--agent` reveals the new worktree and launches the selected agent **in its first terminal**, without adding a separate fallback shell for that worker. Pass `--setup run`; repo setup and default-terminal settings may add intentional tabs or splits. Do **not** run bare `worktree create` and then `terminal create --command <agent>` for the same worker when agent-first create is available: without configured default tabs, that two-step path leaves a fallback shell + agent pair. Only use it when custom agent argv is required (for example Codex model/effort flags) or when an older CLI rejects `--agent`; if you must, message only the agent handle. Configured default tabs are intentional surfaces, so close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell. Do not run `worktree create` when the task must stay in the current worktree.
|
||||
|
||||
Use `orca worktree create --prompt ...` or `orca terminal send ...` for full handoffs or untracked/lightweight prompts. Those paths do not attach `taskId`/`dispatchId`; the worker should not send lifecycle messages unless the prompt supplies a live orchestration preamble.
|
||||
|
||||
@@ -228,11 +339,12 @@ Wait for `tui-idle` before dispatching. Always pass `--timeout-ms`; real coding
|
||||
|
||||
## Agent Guidance
|
||||
|
||||
- Workers with a valid live preamble must send `worker_done` exactly once from their own terminal, even on failure:
|
||||
`orca orchestration send --to <coordinator_handle> --type worker_done --subject "<short status>" --body "<3-sentence summary: what you did, what you found, what's left>" --payload '{"taskId":"<task_id>","dispatchId":"<dispatch_id>","filesModified":["path/a"],"reportPath":"<optional>"}' --json`
|
||||
- Workers with a valid live preamble must send `worker_done` exactly once from their own terminal with an explicit `--outcome succeeded` or `--outcome failed`:
|
||||
`orca orchestration send --type worker_done --subject "<short status>" --body "<3-sentence summary: what you did, what you found, what's left>" --task-id <task_id> --dispatch-id <dispatch_id> --outcome succeeded --files-modified "path/a" --report-path "<optional>" --json`
|
||||
- A failed outcome is still a terminal report, but Orca records both the Dispatch and Task as failed. Never encode failure only in the subject/body.
|
||||
- After sending `worker_done`, end your turn and idle at the agent prompt. Do not poll or keep calling `orca orchestration check`; the coordinator re-engages you with a fresh preamble + TASK block delivered as new terminal input.
|
||||
- For long tasks, send heartbeat/status only when the preamble asks for it, including both IDs:
|
||||
`orca orchestration send --to <coordinator_handle> --type heartbeat --subject "alive" --payload '{"taskId":"<task_id>","dispatchId":"<dispatch_id>","phase":"implementing"}' --json`
|
||||
`orca orchestration send --type heartbeat --subject "alive" --payload '{"taskId":"<task_id>","dispatchId":"<dispatch_id>","phase":"implementing"}' --json`
|
||||
- If blocked before completion, use `ask`; use `escalation` only when ownership is valid and the coordinator must intervene.
|
||||
- Treat preambles inherited through terminal history or full handoffs as stale unless the current prompt explicitly keeps that coordinator in the loop.
|
||||
- Coordinators should use `task-list --ready` as external memory, dispatch parallel waves, and avoid dependency chains deeper than 3-4 steps.
|
||||
@@ -244,11 +356,11 @@ orca terminal create --worktree active --title login-css-worker --command "claud
|
||||
orca terminal wait --terminal <handle> --for tui-idle --timeout-ms 60000 --json
|
||||
orca orchestration task-create --spec "Fix the login button CSS" --json
|
||||
orca orchestration dispatch --task <task_id> --to <handle> --inject --json
|
||||
orca orchestration check --wait --types worker_done,escalation,decision_gate --timeout-ms 900000 --json
|
||||
orca orchestration check --wait --types worker_done,escalation,question --timeout-ms 900000 --json
|
||||
```
|
||||
|
||||
## Next Action
|
||||
|
||||
Coordinator: confirm `orca status --json`, inspect `task-list`/`dispatch-show` if inheriting state, then choose either a manual loop (`task-create` -> worker -> `dispatch --inject` -> `check --wait`) or `orchestration run`.
|
||||
Coordinator: confirm `orca status --json`, create or bind a Run, inspect `task-list`/`dispatch-show` if inheriting state, then use the explicit supervised loop (`task-create` -> `worker-start` -> `check --wait`). Use low-level terminal creation plus `dispatch --inject` only when the composed start does not express the needed topology.
|
||||
|
||||
Worker: if the current prompt contains a live dispatch preamble, do the task, use `ask` for blocking questions, and send `worker_done` once with the required payload. If the preamble is stale or absent, do not send lifecycle messages; inspect state or treat the prompt as an ordinary handoff.
|
||||
|
||||
File diff suppressed because one or more lines are too long
+34
-1
@@ -12,7 +12,8 @@ import {
|
||||
formatTerminalList,
|
||||
formatTerminalRead,
|
||||
formatWorktreeList,
|
||||
printResult
|
||||
printResult,
|
||||
reportCliError
|
||||
} from './format'
|
||||
import type { ComputerActionResult, RuntimeWorktreeRecord } from '../shared/runtime-types'
|
||||
import type { Automation } from '../shared/automations-types'
|
||||
@@ -116,6 +117,38 @@ describe('formatCliError', () => {
|
||||
].join('\n')
|
||||
)
|
||||
})
|
||||
|
||||
it('preserves orchestration migration recovery in human and JSON errors', () => {
|
||||
const error = new RuntimeRpcFailureError({
|
||||
id: 'req_migration',
|
||||
ok: false,
|
||||
error: {
|
||||
code: 'orchestration_migration_required',
|
||||
message: 'No effects were applied.',
|
||||
data: {
|
||||
effectsApplied: false,
|
||||
nextCommandArgs: ['skills', 'get', 'orchestration', '--full'],
|
||||
nextSteps: ['Using this same Orca CLI executable, run: skills get orchestration --full']
|
||||
}
|
||||
},
|
||||
_meta: { runtimeId: 'runtime-1' }
|
||||
})
|
||||
|
||||
expect(formatCliError(error)).toContain(
|
||||
'Next step: Using this same Orca CLI executable, run: skills get orchestration --full'
|
||||
)
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
reportCliError(error, true)
|
||||
expect(JSON.parse(String(log.mock.calls[0]?.[0]))).toMatchObject({
|
||||
error: {
|
||||
code: 'orchestration_migration_required',
|
||||
data: {
|
||||
effectsApplied: false,
|
||||
nextCommandArgs: ['skills', 'get', 'orchestration', '--full']
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('formatWorktreeList', () => {
|
||||
|
||||
@@ -87,6 +87,11 @@ export const HANDLER_GROUPS: readonly HandlerGroup[] = [
|
||||
{
|
||||
name: 'orchestration',
|
||||
keys: [
|
||||
'orchestration run-create',
|
||||
'orchestration run-use',
|
||||
'orchestration run-current',
|
||||
'orchestration run-list',
|
||||
'orchestration run-show',
|
||||
'orchestration send',
|
||||
'orchestration check',
|
||||
'orchestration reply',
|
||||
@@ -94,11 +99,16 @@ export const HANDLER_GROUPS: readonly HandlerGroup[] = [
|
||||
'orchestration task-create',
|
||||
'orchestration task-list',
|
||||
'orchestration task-update',
|
||||
'orchestration worker-start',
|
||||
'orchestration worker-show',
|
||||
'orchestration worker-read',
|
||||
'orchestration worker-stop',
|
||||
'orchestration worker-abandon',
|
||||
'orchestration dispatch',
|
||||
'orchestration ask',
|
||||
'orchestration dispatch-show',
|
||||
'orchestration run',
|
||||
'orchestration run-stop',
|
||||
'orchestration coordinator-start',
|
||||
'orchestration coordinator-stop',
|
||||
'orchestration gate-create',
|
||||
'orchestration gate-resolve',
|
||||
'orchestration gate-list',
|
||||
|
||||
@@ -30,7 +30,8 @@ it('prints a lifecycle rejection and exits unsuccessfully', async () => {
|
||||
['from', 'term_foreign'],
|
||||
['to', 'term_coord'],
|
||||
['subject', 'done'],
|
||||
['type', 'worker_done']
|
||||
['type', 'worker_done'],
|
||||
['outcome', 'succeeded']
|
||||
]),
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { ORCHESTRATION_HANDLERS } from './orchestration'
|
||||
|
||||
describe('orchestration CLI migration recovery', () => {
|
||||
it('redirects worker_done without an outcome before resolving or calling the runtime', async () => {
|
||||
const call = vi.fn()
|
||||
|
||||
await expect(
|
||||
ORCHESTRATION_HANDLERS['orchestration send']({
|
||||
flags: new Map<string, string | boolean>([
|
||||
['from', 'term_worker'],
|
||||
['subject', 'Done'],
|
||||
['type', 'worker_done']
|
||||
]),
|
||||
client: { call },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
).rejects.toMatchObject({
|
||||
code: 'invalid_argument',
|
||||
data: {
|
||||
effectsApplied: false,
|
||||
nextCommandArgs: ['skills', 'get', 'orchestration', '--full']
|
||||
}
|
||||
})
|
||||
expect(call).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,160 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const callMock = vi.fn()
|
||||
const getTerminalHandleMock = vi.hoisted(() => vi.fn())
|
||||
|
||||
vi.mock('../format', () => ({ printResult: vi.fn() }))
|
||||
vi.mock('../selectors', () => ({ getTerminalHandle: getTerminalHandleMock }))
|
||||
|
||||
import { printResult } from '../format'
|
||||
import { ORCHESTRATION_HANDLERS } from './orchestration'
|
||||
|
||||
describe('lightweight Run CLI handlers', () => {
|
||||
beforeEach(() => {
|
||||
callMock.mockReset()
|
||||
getTerminalHandleMock.mockReset()
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_coord'
|
||||
})
|
||||
|
||||
it('creates a Run with the resolved coordinator terminal', async () => {
|
||||
callMock.mockResolvedValue({
|
||||
result: { run: { id: 'run_1', objective: 'Coordinate work', consumer_generation: 1 } }
|
||||
})
|
||||
await ORCHESTRATION_HANDLERS['orchestration run-create']({
|
||||
flags: new Map<string, string | boolean>([
|
||||
['objective', 'Coordinate work'],
|
||||
['json', true]
|
||||
]),
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.runCreate', {
|
||||
objective: 'Coordinate work',
|
||||
from: 'term_coord'
|
||||
})
|
||||
})
|
||||
|
||||
it('reuses the same explicit binding path for run-use and run-current', async () => {
|
||||
callMock
|
||||
.mockResolvedValueOnce({ result: { run: { id: 'run_1', objective: 'Work' } } })
|
||||
.mockResolvedValueOnce({ result: { run: { id: 'run_1', objective: 'Work' } } })
|
||||
await ORCHESTRATION_HANDLERS['orchestration run-use']({
|
||||
flags: new Map([
|
||||
['id', 'run_1'],
|
||||
['from', 'term_coord']
|
||||
]),
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
await ORCHESTRATION_HANDLERS['orchestration run-current']({
|
||||
flags: new Map([['from', 'term_coord']]),
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
expect(callMock).toHaveBeenNthCalledWith(1, 'orchestration.runUse', {
|
||||
id: 'run_1',
|
||||
from: 'term_coord'
|
||||
})
|
||||
expect(callMock).toHaveBeenNthCalledWith(2, 'orchestration.runCurrent', {
|
||||
from: 'term_coord'
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('orchestration reset CLI handler', () => {
|
||||
beforeEach(() => {
|
||||
callMock.mockReset().mockResolvedValue({ result: { reset: 'all' } })
|
||||
})
|
||||
const invoke = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration reset']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
it('rejects a bare reset before calling the runtime', async () => {
|
||||
await expect(invoke(new Map())).rejects.toMatchObject({
|
||||
code: 'invalid_argument',
|
||||
message: 'Choose exactly one reset scope: --all, --tasks, or --messages.'
|
||||
})
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('sends only the tasks scope for --tasks', async () => {
|
||||
await invoke(new Map([['tasks', true]]))
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', {
|
||||
all: undefined,
|
||||
tasks: true,
|
||||
messages: undefined
|
||||
})
|
||||
})
|
||||
|
||||
it('sends only the all scope for --all', async () => {
|
||||
await invoke(new Map([['all', true]]))
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', {
|
||||
all: true,
|
||||
tasks: undefined,
|
||||
messages: undefined
|
||||
})
|
||||
})
|
||||
|
||||
it.each([
|
||||
new Map<string, string | boolean>([
|
||||
['tasks', true],
|
||||
['messages', true]
|
||||
]),
|
||||
new Map<string, string | boolean>([
|
||||
['all', true],
|
||||
['tasks', true]
|
||||
])
|
||||
])('rejects multiple reset scopes before calling the runtime', async (flags) => {
|
||||
await expect(invoke(flags)).rejects.toMatchObject({ code: 'invalid_argument' })
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('orchestration task-list brief output', () => {
|
||||
it('requests server-side brief and falls back client-side for older runtimes', async () => {
|
||||
callMock.mockReset().mockResolvedValue({
|
||||
result: {
|
||||
tasks: [{ id: 'task_1', spec: `First line\n${'detail '.repeat(40)}`, status: 'ready' }],
|
||||
count: 1
|
||||
}
|
||||
})
|
||||
vi.mocked(printResult).mockClear()
|
||||
await ORCHESTRATION_HANDLERS['orchestration task-list']({
|
||||
flags: new Map([['brief', true]]),
|
||||
client: { call: callMock },
|
||||
json: true
|
||||
} as never)
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.taskList',
|
||||
expect.objectContaining({ brief: true })
|
||||
)
|
||||
const response = vi.mocked(printResult).mock.calls[0]?.[0] as {
|
||||
result: { tasks: { spec: string; spec_truncated: boolean }[] }
|
||||
}
|
||||
expect(response.result.tasks[0].spec).toHaveLength(160)
|
||||
expect(response.result.tasks[0].spec_truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('passes server-abbreviated rows through untouched', async () => {
|
||||
const serverTasks = [
|
||||
{ id: 'task_1', spec: 'already brief…', status: 'ready', spec_truncated: true }
|
||||
]
|
||||
callMock.mockReset().mockResolvedValue({ result: { tasks: serverTasks, count: 1 } })
|
||||
vi.mocked(printResult).mockClear()
|
||||
await ORCHESTRATION_HANDLERS['orchestration task-list']({
|
||||
flags: new Map([['brief', true]]),
|
||||
client: { call: callMock },
|
||||
json: true
|
||||
} as never)
|
||||
const response = vi.mocked(printResult).mock.calls[0]?.[0] as {
|
||||
result: { tasks: { spec: string; spec_truncated: boolean }[] }
|
||||
}
|
||||
expect(response.result.tasks).toBe(serverTasks)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,230 @@
|
||||
import { beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const callMock = vi.fn()
|
||||
|
||||
vi.mock('../format', () => ({ printResult: vi.fn() }))
|
||||
vi.mock('../selectors', () => ({ getTerminalHandle: vi.fn() }))
|
||||
|
||||
import { printResult } from '../format'
|
||||
import { ORCHESTRATION_HANDLERS } from './orchestration'
|
||||
|
||||
describe('orchestration timeout flag validation', () => {
|
||||
const invalidTimeoutValues: [string, string | boolean][] = [
|
||||
['missing', true],
|
||||
['empty', ''],
|
||||
['non-numeric', 'not-a-number'],
|
||||
['zero', '0'],
|
||||
['negative', '-1']
|
||||
]
|
||||
|
||||
beforeEach(() => {
|
||||
callMock.mockReset()
|
||||
delete process.env.ORCA_TERMINAL_HANDLE
|
||||
delete process.env.ORCA_PANE_KEY
|
||||
})
|
||||
|
||||
const invokeCheck = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration check']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
const invokeAsk = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration ask']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
it.each(invalidTimeoutValues)('rejects invalid check --timeout-ms: %s', async (_label, value) => {
|
||||
await expect(
|
||||
invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['wait', true],
|
||||
['timeout-ms', value]
|
||||
])
|
||||
)
|
||||
).rejects.toThrow(/--timeout-ms/)
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('passes a parsed check timeout and peek mode into the RPC payload', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({ result: { messages: [], count: 0 } })
|
||||
await invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['wait', true],
|
||||
['peek', true],
|
||||
['timeout-ms', '250']
|
||||
])
|
||||
)
|
||||
// Why: unread:false makes pre-peek runtimes fall back to non-consuming all mode.
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.check', {
|
||||
terminal: 'term_worker',
|
||||
unread: false,
|
||||
peek: true,
|
||||
all: undefined,
|
||||
types: undefined,
|
||||
format: undefined,
|
||||
run: undefined,
|
||||
ack: undefined,
|
||||
wait: true,
|
||||
timeoutMs: 250
|
||||
})
|
||||
})
|
||||
|
||||
it('filters already-read rows from a peek response for pre-peek runtimes', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
messages: [
|
||||
{ id: 'msg_old', from_handle: 'a', subject: 'seen', read: 1 },
|
||||
{ id: 'msg_new', from_handle: 'a', subject: 'fresh', read: 0 }
|
||||
],
|
||||
count: 2,
|
||||
formatted: 'banners built from all rows'
|
||||
}
|
||||
})
|
||||
vi.mocked(printResult).mockClear()
|
||||
await invokeCheck(new Map<string, string | boolean>([['peek', true]]))
|
||||
const response = vi.mocked(printResult).mock.calls[0]?.[0] as {
|
||||
result: { messages: { id: string }[]; count: number; formatted?: string }
|
||||
}
|
||||
expect(response.result.messages.map((message) => message.id)).toEqual(['msg_new'])
|
||||
expect(response.result.count).toBe(1)
|
||||
expect(response.result.formatted).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects combined read modes before calling the runtime', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
await expect(
|
||||
invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['unread', true],
|
||||
['peek', true]
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({
|
||||
code: 'invalid_argument',
|
||||
message: expect.stringContaining('read mode')
|
||||
})
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('warns when a pre-peek runtime returned a full 100-row page', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
const rows = Array.from({ length: 100 }, (_, index) => ({
|
||||
id: `msg_${index}`,
|
||||
from_handle: 'a',
|
||||
subject: `s${index}`,
|
||||
read: index === 0 ? 0 : 1
|
||||
}))
|
||||
callMock.mockResolvedValue({ result: { messages: rows, count: 100 } })
|
||||
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {})
|
||||
await invokeCheck(new Map<string, string | boolean>([['peek', true]]))
|
||||
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('newest 100 messages'))
|
||||
errorSpy.mockRestore()
|
||||
})
|
||||
|
||||
it('fails --peek --wait against a runtime that returned only read rows', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
messages: [{ id: 'msg_old', from_handle: 'a', subject: 'seen', read: 1 }],
|
||||
count: 1
|
||||
}
|
||||
})
|
||||
await expect(
|
||||
invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['peek', true],
|
||||
['wait', true]
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({ code: 'peek_wait_unsupported' })
|
||||
})
|
||||
|
||||
it.each(invalidTimeoutValues)('rejects invalid ask --timeout-ms: %s', async (_label, value) => {
|
||||
await expect(
|
||||
invokeAsk(
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['question', 'Proceed?'],
|
||||
['timeout-ms', value]
|
||||
])
|
||||
)
|
||||
).rejects.toThrow(/--timeout-ms/)
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('uses the parsed ask timeout for both runtime wait and client timeout', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: { answer: 'yes', messageId: 'msg_1', threadId: 'thread_1', timedOut: false }
|
||||
})
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
await invokeAsk(
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['question', 'Proceed?'],
|
||||
['timeout-ms', '123']
|
||||
])
|
||||
)
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.ask',
|
||||
{
|
||||
to: 'term_coord',
|
||||
run: undefined,
|
||||
question: 'Proceed?',
|
||||
resume: undefined,
|
||||
options: undefined,
|
||||
timeoutMs: 123,
|
||||
from: 'term_worker'
|
||||
},
|
||||
{ timeoutMs: 5_123, orchestrationCapability: undefined }
|
||||
)
|
||||
})
|
||||
|
||||
it('passes an ask resume without creating a new question payload', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
answer: 'yes',
|
||||
messageId: 'msg_question',
|
||||
threadId: 'msg_question',
|
||||
timedOut: false
|
||||
}
|
||||
})
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
await invokeAsk(new Map<string, string | boolean>([['resume', 'msg_question']]))
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.ask',
|
||||
{
|
||||
to: undefined,
|
||||
run: undefined,
|
||||
question: undefined,
|
||||
resume: 'msg_question',
|
||||
options: undefined,
|
||||
timeoutMs: undefined,
|
||||
from: 'term_worker'
|
||||
},
|
||||
{ timeoutMs: 605_000, orchestrationCapability: undefined }
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects ambiguous ask create/resume input before RPC', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
await expect(
|
||||
invokeAsk(
|
||||
new Map<string, string | boolean>([
|
||||
['question', 'new'],
|
||||
['resume', 'msg_old']
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({ code: 'invalid_argument' })
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,161 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const callMock = vi.fn()
|
||||
const originalExitCode = process.exitCode
|
||||
|
||||
vi.mock('../format', () => ({ printResult: vi.fn() }))
|
||||
vi.mock('../selectors', () => ({ getTerminalHandle: vi.fn() }))
|
||||
|
||||
import { ORCHESTRATION_HANDLERS } from './orchestration'
|
||||
|
||||
describe('orchestration worker-start CLI contract', () => {
|
||||
beforeEach(() => {
|
||||
callMock.mockReset()
|
||||
process.exitCode = undefined
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
process.exitCode = originalExitCode
|
||||
})
|
||||
|
||||
const invokeWorkerStart = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration worker-start']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
it('passes the complete supported creation contract and retry receipt', async () => {
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
runId: 'run_1',
|
||||
taskId: 'task_1',
|
||||
dispatchId: 'ctx_1',
|
||||
state: 'ready',
|
||||
effects: [],
|
||||
residualResources: []
|
||||
}
|
||||
})
|
||||
|
||||
await invokeWorkerStart(
|
||||
new Map<string, string | boolean>([
|
||||
['task', 'task_1'],
|
||||
['on', 'windows'],
|
||||
['worktree', 'new-top-level'],
|
||||
['name', 'release-audit'],
|
||||
['repo', 'id:windows-repo'],
|
||||
['base-branch', 'origin/release'],
|
||||
['display-name', 'Release audit'],
|
||||
['comment', 'Supervised from the Mac Run home'],
|
||||
['setup', 'run'],
|
||||
['agent', 'codex'],
|
||||
['timeout-ms', '90000'],
|
||||
['run', 'run_1'],
|
||||
['from', 'term_coord'],
|
||||
['retry-request', 'request_1']
|
||||
])
|
||||
)
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.workerStart',
|
||||
{
|
||||
task: 'task_1',
|
||||
on: 'windows',
|
||||
worktree: 'new-top-level',
|
||||
name: 'release-audit',
|
||||
repo: 'id:windows-repo',
|
||||
baseBranch: 'origin/release',
|
||||
displayName: 'Release audit',
|
||||
comment: 'Supervised from the Mac Run home',
|
||||
setup: 'run',
|
||||
agent: 'codex',
|
||||
terminal: undefined,
|
||||
retryOf: undefined,
|
||||
timeoutMs: 90_000,
|
||||
run: 'run_1',
|
||||
from: 'term_coord',
|
||||
devMode: false
|
||||
},
|
||||
{ orchestrationRequestId: 'request_1' }
|
||||
)
|
||||
expect(process.exitCode).toBeUndefined()
|
||||
})
|
||||
|
||||
it('sets an unsuccessful exit code for failed and unknown receipts', async () => {
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
taskId: 'task_1',
|
||||
dispatchId: 'ctx_1',
|
||||
state: 'outcome_unknown',
|
||||
effects: [],
|
||||
residualResources: []
|
||||
}
|
||||
})
|
||||
|
||||
await invokeWorkerStart(
|
||||
new Map<string, string | boolean>([
|
||||
['task', 'task_1'],
|
||||
['agent', 'codex'],
|
||||
['from', 'term_coord']
|
||||
])
|
||||
)
|
||||
|
||||
expect(process.exitCode).toBe(1)
|
||||
})
|
||||
|
||||
it('allows the initial zero cursor when paging worker output', async () => {
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
dispatchId: 'ctx_1',
|
||||
terminal: { tail: [], status: 'running', nextCursor: '0' }
|
||||
}
|
||||
})
|
||||
|
||||
await ORCHESTRATION_HANDLERS['orchestration worker-read']({
|
||||
flags: new Map<string, string | boolean>([
|
||||
['dispatch', 'ctx_1'],
|
||||
['cursor', '0'],
|
||||
['limit', '100']
|
||||
]),
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.workerRead', {
|
||||
dispatch: 'ctx_1',
|
||||
cursor: 0,
|
||||
limit: 100,
|
||||
source: undefined
|
||||
})
|
||||
})
|
||||
|
||||
it('passes opaque source-pinned cursors and explicit source selection', async () => {
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
dispatchId: 'ctx_1',
|
||||
source: 'transcript',
|
||||
transcript: { messages: [], nextCursor: 'owr1_next' }
|
||||
}
|
||||
})
|
||||
|
||||
await ORCHESTRATION_HANDLERS['orchestration worker-read']({
|
||||
flags: new Map<string, string | boolean>([
|
||||
['dispatch', 'ctx_1'],
|
||||
['cursor', 'owr1_previous'],
|
||||
['source', 'transcript']
|
||||
]),
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.workerRead', {
|
||||
dispatch: 'ctx_1',
|
||||
cursor: 'owr1_previous',
|
||||
limit: undefined,
|
||||
source: 'transcript'
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -5,7 +5,7 @@ const getTerminalHandleMock = vi.hoisted(() => vi.fn())
|
||||
const originalTerminalHandle = process.env.ORCA_TERMINAL_HANDLE
|
||||
const originalPaneKey = process.env.ORCA_PANE_KEY
|
||||
function lifecycleGroupRecipientError(type: 'worker_done' | 'heartbeat'): string {
|
||||
return `${type} messages must be sent to a concrete coordinator terminal handle, not a group address.`
|
||||
return `${type} messages belong to one exact Dispatch and cannot target a group address.`
|
||||
}
|
||||
|
||||
// Why: isolate the handler's flag-to-param mapping; printResult only writes output.
|
||||
@@ -48,46 +48,6 @@ afterEach(() => {
|
||||
}
|
||||
})
|
||||
|
||||
describe('orchestration reset CLI handler', () => {
|
||||
beforeEach(() => {
|
||||
callMock.mockReset().mockResolvedValue({ result: { reset: 'all' } })
|
||||
})
|
||||
|
||||
const invoke = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration reset']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
it('sends all: true for a bare `reset` (no scope flag)', async () => {
|
||||
await invoke(new Map())
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', {
|
||||
all: true,
|
||||
tasks: undefined,
|
||||
messages: undefined
|
||||
})
|
||||
})
|
||||
|
||||
it('sends only the tasks scope for --tasks', async () => {
|
||||
await invoke(new Map([['tasks', true]]))
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', {
|
||||
all: undefined,
|
||||
tasks: true,
|
||||
messages: undefined
|
||||
})
|
||||
})
|
||||
|
||||
it('sends only the all scope for --all (no implicit extra scopes)', async () => {
|
||||
await invoke(new Map([['all', true]]))
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', {
|
||||
all: true,
|
||||
tasks: undefined,
|
||||
messages: undefined
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('orchestration send structured payload flags', () => {
|
||||
beforeEach(() => {
|
||||
callMock.mockReset().mockResolvedValue({ result: { message: { id: 'msg_1' } } })
|
||||
@@ -113,6 +73,7 @@ describe('orchestration send structured payload flags', () => {
|
||||
['type', 'worker_done'],
|
||||
['task-id', 'task_1'],
|
||||
['dispatch-id', 'ctx_1'],
|
||||
['outcome', 'succeeded'],
|
||||
['files-modified', 'src/a.ts, src/b.ts'],
|
||||
['report-path', 'reports/done.md']
|
||||
])
|
||||
@@ -129,6 +90,7 @@ describe('orchestration send structured payload flags', () => {
|
||||
payload: JSON.stringify({
|
||||
taskId: 'task_1',
|
||||
dispatchId: 'ctx_1',
|
||||
outcome: 'succeeded',
|
||||
filesModified: ['src/a.ts', 'src/b.ts'],
|
||||
reportPath: 'reports/done.md'
|
||||
}),
|
||||
@@ -151,6 +113,25 @@ describe('orchestration send structured payload flags', () => {
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.send', expect.objectContaining({ body }))
|
||||
})
|
||||
|
||||
it('carries Dispatch authority in the RPC envelope instead of message params', async () => {
|
||||
await invokeSend(
|
||||
new Map<string, string | boolean>([
|
||||
['from', 'term_worker'],
|
||||
['subject', 'alive'],
|
||||
['type', 'heartbeat'],
|
||||
['dispatch-id', 'ctx_1'],
|
||||
['dispatch-capability', 'dcap_secret'],
|
||||
['retry-request', 'mutation_1']
|
||||
])
|
||||
)
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.send',
|
||||
expect.not.objectContaining({ dispatchCapability: expect.anything() }),
|
||||
{ orchestrationCapability: 'dcap_secret', orchestrationRequestId: 'mutation_1' }
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects mixing raw payload with structured payload flags', async () => {
|
||||
await expect(
|
||||
invokeSend(
|
||||
@@ -212,7 +193,8 @@ describe('orchestration send structured payload flags', () => {
|
||||
['from', 'term_worker'],
|
||||
['to', 'term_coord'],
|
||||
['subject', 'done'],
|
||||
['type', 'worker_done']
|
||||
['type', 'worker_done'],
|
||||
['outcome', 'succeeded']
|
||||
])
|
||||
)
|
||||
|
||||
@@ -224,7 +206,7 @@ describe('orchestration send structured payload flags', () => {
|
||||
type: 'worker_done',
|
||||
priority: undefined,
|
||||
threadId: undefined,
|
||||
payload: undefined,
|
||||
payload: JSON.stringify({ outcome: 'succeeded' }),
|
||||
devMode: false
|
||||
})
|
||||
})
|
||||
@@ -236,7 +218,8 @@ describe('orchestration send structured payload flags', () => {
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['subject', 'done'],
|
||||
['type', 'worker_done']
|
||||
['type', 'worker_done'],
|
||||
['outcome', 'succeeded']
|
||||
])
|
||||
)
|
||||
|
||||
@@ -249,7 +232,7 @@ describe('orchestration send structured payload flags', () => {
|
||||
type: 'worker_done',
|
||||
priority: undefined,
|
||||
threadId: undefined,
|
||||
payload: undefined,
|
||||
payload: JSON.stringify({ outcome: 'succeeded' }),
|
||||
devMode: false
|
||||
})
|
||||
})
|
||||
@@ -264,7 +247,8 @@ describe('orchestration send structured payload flags', () => {
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['subject', 'update'],
|
||||
['type', type]
|
||||
['type', type],
|
||||
...(type === 'worker_done' ? ([['outcome', 'succeeded']] as const) : [])
|
||||
])
|
||||
)
|
||||
|
||||
@@ -288,7 +272,8 @@ describe('orchestration send structured payload flags', () => {
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['subject', 'done'],
|
||||
['type', 'worker_done']
|
||||
['type', 'worker_done'],
|
||||
['outcome', 'succeeded']
|
||||
])
|
||||
)
|
||||
|
||||
@@ -308,7 +293,8 @@ describe('orchestration send structured payload flags', () => {
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['subject', 'done'],
|
||||
['type', 'worker_done']
|
||||
['type', 'worker_done'],
|
||||
['outcome', 'succeeded']
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({
|
||||
@@ -328,7 +314,8 @@ describe('orchestration send structured payload flags', () => {
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['subject', 'update'],
|
||||
['type', type]
|
||||
['type', type],
|
||||
...(type === 'worker_done' ? ([['outcome', 'succeeded']] as const) : [])
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({ code: 'no_active_sender_terminal' })
|
||||
@@ -364,7 +351,7 @@ describe('orchestration dispatch coordinator handle', () => {
|
||||
} as never)
|
||||
|
||||
const invokeRun = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration run']({
|
||||
ORCHESTRATION_HANDLERS['orchestration coordinator-start']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
@@ -483,29 +470,18 @@ describe('orchestration dispatch coordinator handle', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('uses a live coordinator handle for orchestration runs', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_stale_coord'
|
||||
process.env.ORCA_PANE_KEY = 'tab_coord:leaf_coord'
|
||||
stubStaleHandleRemint('term_live_coord', {
|
||||
result: { runId: 'run_1', status: 'running' }
|
||||
})
|
||||
getTerminalHandleMock.mockRejectedValue(new Error('active terminal fallback is unsafe'))
|
||||
|
||||
await invokeRun(new Map<string, string | boolean>([['spec', 'run the plan']]))
|
||||
|
||||
expect(callMock).toHaveBeenNthCalledWith(1, 'terminal.show', {
|
||||
terminal: 'term_stale_coord'
|
||||
})
|
||||
expect(callMock).toHaveBeenNthCalledWith(2, 'terminal.resolvePane', {
|
||||
paneKey: 'tab_coord:leaf_coord'
|
||||
})
|
||||
expect(callMock).toHaveBeenNthCalledWith(3, 'orchestration.run', {
|
||||
spec: 'run the plan',
|
||||
from: 'term_live_coord',
|
||||
pollIntervalMs: undefined,
|
||||
maxConcurrent: undefined,
|
||||
worktree: undefined
|
||||
it('retires the legacy coordinator command without runtime effects', async () => {
|
||||
await expect(
|
||||
invokeRun(new Map<string, string | boolean>([['spec', 'run the plan']]))
|
||||
).rejects.toMatchObject({
|
||||
code: 'orchestration_migration_required',
|
||||
data: {
|
||||
reason: 'command_retired',
|
||||
effectsApplied: false,
|
||||
nextCommandArgs: ['skills', 'get', 'orchestration', '--full']
|
||||
}
|
||||
})
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -540,75 +516,58 @@ describe('orchestration task-create caller handle', () => {
|
||||
displayName: undefined,
|
||||
deps: undefined,
|
||||
parent: undefined,
|
||||
run: undefined,
|
||||
callerTerminalHandle: 'term_creator'
|
||||
})
|
||||
})
|
||||
|
||||
it('does not persist a stale env terminal handle as task creator', async () => {
|
||||
it('fails closed when a stale task creator handle cannot be reminted', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_stale'
|
||||
callMock
|
||||
.mockRejectedValueOnce(staleHandleError())
|
||||
.mockResolvedValueOnce({ result: { task: { id: 'task_1', status: 'ready' } } })
|
||||
callMock.mockRejectedValueOnce(staleHandleError())
|
||||
getTerminalHandleMock.mockResolvedValue('term_wrong_active')
|
||||
|
||||
await invokeTaskCreate(new Map<string, string | boolean>([['spec', 'do work']]))
|
||||
await expect(
|
||||
invokeTaskCreate(new Map<string, string | boolean>([['spec', 'do work']]))
|
||||
).rejects.toMatchObject({ code: 'no_active_sender_terminal' })
|
||||
|
||||
expect(callMock).toHaveBeenNthCalledWith(1, 'terminal.show', { terminal: 'term_stale' })
|
||||
expect(getTerminalHandleMock).not.toHaveBeenCalled()
|
||||
expect(callMock).toHaveBeenNthCalledWith(2, 'orchestration.taskCreate', {
|
||||
spec: 'do work',
|
||||
taskTitle: undefined,
|
||||
displayName: undefined,
|
||||
deps: undefined,
|
||||
parent: undefined,
|
||||
callerTerminalHandle: undefined
|
||||
})
|
||||
expect(callMock).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('does not fail task creation when env handle validation cannot inspect the graph', async () => {
|
||||
it('propagates runtime unavailability while proving the bound coordinator', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_creator'
|
||||
callMock
|
||||
.mockRejectedValueOnce(new RuntimeClientError('runtime_unavailable', 'runtime_unavailable'))
|
||||
.mockResolvedValueOnce({ result: { task: { id: 'task_1', status: 'ready' } } })
|
||||
callMock.mockRejectedValueOnce(
|
||||
new RuntimeClientError('runtime_unavailable', 'runtime_unavailable')
|
||||
)
|
||||
|
||||
await invokeTaskCreate(new Map<string, string | boolean>([['spec', 'do work']]))
|
||||
await expect(
|
||||
invokeTaskCreate(new Map<string, string | boolean>([['spec', 'do work']]))
|
||||
).rejects.toMatchObject({ code: 'runtime_unavailable' })
|
||||
|
||||
expect(callMock).toHaveBeenNthCalledWith(1, 'terminal.show', { terminal: 'term_creator' })
|
||||
expect(getTerminalHandleMock).not.toHaveBeenCalled()
|
||||
expect(callMock).toHaveBeenNthCalledWith(2, 'orchestration.taskCreate', {
|
||||
spec: 'do work',
|
||||
taskTitle: undefined,
|
||||
displayName: undefined,
|
||||
deps: undefined,
|
||||
parent: undefined,
|
||||
callerTerminalHandle: undefined
|
||||
})
|
||||
expect(callMock).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('omits caller handle when pane reminting cannot inspect the graph', async () => {
|
||||
it('propagates runtime unavailability while reminting the bound coordinator', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_stale'
|
||||
process.env.ORCA_PANE_KEY = 'tab_creator:leaf_creator'
|
||||
stubStaleHandleRemintFailure(
|
||||
new RuntimeClientError('runtime_unavailable', 'runtime_unavailable')
|
||||
)
|
||||
callMock.mockResolvedValueOnce({ result: { task: { id: 'task_1', status: 'ready' } } })
|
||||
getTerminalHandleMock.mockResolvedValue('term_wrong_active')
|
||||
|
||||
await invokeTaskCreate(new Map<string, string | boolean>([['spec', 'do work']]))
|
||||
await expect(
|
||||
invokeTaskCreate(new Map<string, string | boolean>([['spec', 'do work']]))
|
||||
).rejects.toMatchObject({ code: 'runtime_unavailable' })
|
||||
|
||||
expect(callMock).toHaveBeenNthCalledWith(1, 'terminal.show', { terminal: 'term_stale' })
|
||||
expect(callMock).toHaveBeenNthCalledWith(2, 'terminal.resolvePane', {
|
||||
paneKey: 'tab_creator:leaf_creator'
|
||||
})
|
||||
expect(getTerminalHandleMock).not.toHaveBeenCalled()
|
||||
expect(callMock).toHaveBeenNthCalledWith(3, 'orchestration.taskCreate', {
|
||||
spec: 'do work',
|
||||
taskTitle: undefined,
|
||||
displayName: undefined,
|
||||
deps: undefined,
|
||||
parent: undefined,
|
||||
callerTerminalHandle: undefined
|
||||
})
|
||||
expect(callMock).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
|
||||
it('propagates unexpected caller pane remint failures for task creation', async () => {
|
||||
@@ -665,10 +624,248 @@ describe('orchestration task-create caller handle', () => {
|
||||
displayName: undefined,
|
||||
deps: undefined,
|
||||
parent: undefined,
|
||||
run: undefined,
|
||||
callerTerminalHandle: 'term_live'
|
||||
})
|
||||
})
|
||||
})
|
||||
describe('orchestration timeout flag validation', () => {
|
||||
const invalidTimeoutValues: [string, string | boolean][] = [
|
||||
['missing', true],
|
||||
['empty', ''],
|
||||
['non-numeric', 'not-a-number'],
|
||||
['zero', '0'],
|
||||
['negative', '-1']
|
||||
]
|
||||
|
||||
beforeEach(() => {
|
||||
callMock.mockReset()
|
||||
delete process.env.ORCA_TERMINAL_HANDLE
|
||||
delete process.env.ORCA_PANE_KEY
|
||||
})
|
||||
|
||||
const invokeCheck = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration check']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
const invokeAsk = (flags: Map<string, string | boolean>) =>
|
||||
ORCHESTRATION_HANDLERS['orchestration ask']({
|
||||
flags,
|
||||
client: { call: callMock },
|
||||
cwd: '/tmp/repo',
|
||||
json: true
|
||||
} as never)
|
||||
|
||||
it.each(invalidTimeoutValues)('rejects invalid check --timeout-ms: %s', async (_label, value) => {
|
||||
const flags = new Map<string, string | boolean>([
|
||||
['wait', true],
|
||||
['timeout-ms', value]
|
||||
])
|
||||
|
||||
await expect(invokeCheck(flags)).rejects.toThrow(/--timeout-ms/)
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('passes a parsed check timeout and peek mode into the RPC payload', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({ result: { messages: [], count: 0 } })
|
||||
|
||||
await invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['wait', true],
|
||||
['peek', true],
|
||||
['timeout-ms', '250']
|
||||
])
|
||||
)
|
||||
|
||||
// Why: --peek rides with unread:false so pre-peek runtimes fall back to
|
||||
// the non-consuming all mode instead of the destructive mark-read default.
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.check', {
|
||||
terminal: 'term_worker',
|
||||
unread: false,
|
||||
peek: true,
|
||||
all: undefined,
|
||||
types: undefined,
|
||||
format: undefined,
|
||||
run: undefined,
|
||||
ack: undefined,
|
||||
wait: true,
|
||||
timeoutMs: 250
|
||||
})
|
||||
})
|
||||
|
||||
it('filters already-read rows from a peek response for pre-peek runtimes', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
messages: [
|
||||
{ id: 'msg_old', from_handle: 'a', subject: 'seen', read: 1 },
|
||||
{ id: 'msg_new', from_handle: 'a', subject: 'fresh', read: 0 }
|
||||
],
|
||||
count: 2,
|
||||
formatted: 'banners built from all rows'
|
||||
}
|
||||
})
|
||||
vi.mocked(printResult).mockClear()
|
||||
|
||||
await invokeCheck(new Map<string, string | boolean>([['peek', true]]))
|
||||
|
||||
const response = vi.mocked(printResult).mock.calls[0]?.[0] as {
|
||||
result: { messages: { id: string }[]; count: number; formatted?: string }
|
||||
}
|
||||
expect(response.result.messages.map((m) => m.id)).toEqual(['msg_new'])
|
||||
expect(response.result.count).toBe(1)
|
||||
// Why: the pre-peek runtime built `formatted` from all rows, including
|
||||
// the read one the filter just removed.
|
||||
expect(response.result.formatted).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects combined read modes before calling the runtime', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockClear()
|
||||
|
||||
await expect(
|
||||
invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['unread', true],
|
||||
['peek', true]
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({
|
||||
code: 'invalid_argument',
|
||||
message: expect.stringContaining('read mode')
|
||||
})
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('warns when a pre-peek runtime returned a full 100-row page', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
const rows = Array.from({ length: 100 }, (_, i) => ({
|
||||
id: `msg_${i}`,
|
||||
from_handle: 'a',
|
||||
subject: `s${i}`,
|
||||
read: i === 0 ? 0 : 1
|
||||
}))
|
||||
callMock.mockResolvedValue({ result: { messages: rows, count: 100 } })
|
||||
const errorSpy = vi.spyOn(console, 'error').mockImplementation(() => {})
|
||||
|
||||
await invokeCheck(new Map<string, string | boolean>([['peek', true]]))
|
||||
|
||||
expect(errorSpy).toHaveBeenCalledWith(expect.stringContaining('newest 100 messages'))
|
||||
errorSpy.mockRestore()
|
||||
})
|
||||
|
||||
it('fails --peek --wait against a runtime that returned only read rows', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
messages: [{ id: 'msg_old', from_handle: 'a', subject: 'seen', read: 1 }],
|
||||
count: 1
|
||||
}
|
||||
})
|
||||
|
||||
await expect(
|
||||
invokeCheck(
|
||||
new Map<string, string | boolean>([
|
||||
['peek', true],
|
||||
['wait', true]
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({ code: 'peek_wait_unsupported' })
|
||||
})
|
||||
|
||||
it.each(invalidTimeoutValues)('rejects invalid ask --timeout-ms: %s', async (_label, value) => {
|
||||
const flags = new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['question', 'Proceed?'],
|
||||
['timeout-ms', value]
|
||||
])
|
||||
|
||||
await expect(invokeAsk(flags)).rejects.toThrow(/--timeout-ms/)
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('uses the parsed ask timeout for both runtime wait and client timeout', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
answer: 'yes',
|
||||
messageId: 'msg_1',
|
||||
threadId: 'thread_1',
|
||||
timedOut: false
|
||||
}
|
||||
})
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
|
||||
await invokeAsk(
|
||||
new Map<string, string | boolean>([
|
||||
['to', 'term_coord'],
|
||||
['question', 'Proceed?'],
|
||||
['timeout-ms', '123']
|
||||
])
|
||||
)
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.ask',
|
||||
{
|
||||
to: 'term_coord',
|
||||
run: undefined,
|
||||
question: 'Proceed?',
|
||||
resume: undefined,
|
||||
options: undefined,
|
||||
timeoutMs: 123,
|
||||
from: 'term_worker'
|
||||
},
|
||||
{ timeoutMs: 5_123, orchestrationCapability: undefined }
|
||||
)
|
||||
})
|
||||
|
||||
it('passes an ask resume without creating a new question payload', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
callMock.mockResolvedValue({
|
||||
result: {
|
||||
answer: 'yes',
|
||||
messageId: 'msg_question',
|
||||
threadId: 'msg_question',
|
||||
timedOut: false
|
||||
}
|
||||
})
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
|
||||
await invokeAsk(new Map<string, string | boolean>([['resume', 'msg_question']]))
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.ask',
|
||||
{
|
||||
to: undefined,
|
||||
run: undefined,
|
||||
question: undefined,
|
||||
resume: 'msg_question',
|
||||
options: undefined,
|
||||
timeoutMs: undefined,
|
||||
from: 'term_worker'
|
||||
},
|
||||
{ timeoutMs: 605_000, orchestrationCapability: undefined }
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects ambiguous ask create/resume input before RPC', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_worker'
|
||||
await expect(
|
||||
invokeAsk(
|
||||
new Map<string, string | boolean>([
|
||||
['question', 'new'],
|
||||
['resume', 'msg_old']
|
||||
])
|
||||
)
|
||||
).rejects.toMatchObject({ code: 'invalid_argument' })
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
|
||||
describe('orchestration task-list brief output', () => {
|
||||
it('requests server-side brief and falls back client-side for older runtimes', async () => {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
/* eslint-disable max-lines -- Why: orchestration CLI handlers share flag-parsing helpers and dispatch/preamble logic; splitting by verb would fragment the RuntimeClient call shape without reducing complexity. */
|
||||
import type { CommandHandler } from '../dispatch'
|
||||
import type { RuntimeClient } from '../runtime-client'
|
||||
import { printResult } from '../format'
|
||||
import {
|
||||
getOptionalPositiveIntegerFlag,
|
||||
@@ -14,11 +15,21 @@ import {
|
||||
} from '../../shared/orchestration-ask-timeout'
|
||||
import { abbreviateOrchestrationTasks } from '../../shared/orchestration-task-summary'
|
||||
import { parsePositiveSafeIntegerText } from '../../shared/timer-delay'
|
||||
import type {
|
||||
OrchestrationWorkerReadResult,
|
||||
OrchestrationWorkerReadSource
|
||||
} from '../../shared/orchestration-worker-output'
|
||||
import type { NativeChatMessage } from '../../shared/native-chat-types'
|
||||
import type { RuntimeTerminalRead } from '../../shared/runtime-types'
|
||||
import {
|
||||
orchestrationMigrationData,
|
||||
orchestrationSkillRecoveryData
|
||||
} from '../../shared/orchestration-rpc-contract'
|
||||
|
||||
// Why: 15 s is well under Claude Code's ~2 min Bash-tool silence budget while keeping log volume low. See design doc §3.4.
|
||||
const DEFAULT_KEEPALIVE_INTERVAL_MS = 15_000
|
||||
function getLifecycleGroupRecipientError(type: 'worker_done' | 'heartbeat'): string {
|
||||
return `${type} messages must be sent to a concrete coordinator terminal handle, not a group address.`
|
||||
return `${type} messages belong to one exact Dispatch and cannot target a group address.`
|
||||
}
|
||||
|
||||
// Why: test-only escape hatch so subprocess tests avoid the full 15 s window; bogus values fall back to the default.
|
||||
@@ -83,6 +94,16 @@ type LifecycleSendRejection = {
|
||||
type OrchestrationSendResult =
|
||||
| { message: { id: string }; lifecycle?: LifecycleSendRejection }
|
||||
| { messages: { id: string }[]; recipients: number }
|
||||
| {
|
||||
relay: {
|
||||
messageId: string
|
||||
sequence: number
|
||||
dispatchId: string
|
||||
destination?: 'run_home' | 'worker'
|
||||
accepted: true
|
||||
}
|
||||
lifecycle?: { action: 'completed' | 'failed' }
|
||||
}
|
||||
|
||||
function getOptionalStructuredMessagePayload(
|
||||
flags: Map<string, string | boolean>
|
||||
@@ -90,12 +111,14 @@ function getOptionalStructuredMessagePayload(
|
||||
const rawPayload = getOptionalStringFlag(flags, 'payload')
|
||||
const taskId = getOptionalStringFlag(flags, 'task-id')
|
||||
const dispatchId = getOptionalStringFlag(flags, 'dispatch-id')
|
||||
const outcome = getOptionalStringFlag(flags, 'outcome')
|
||||
const filesModified = getOptionalStringFlag(flags, 'files-modified')
|
||||
const reportPath = getOptionalStringFlag(flags, 'report-path')
|
||||
const phase = getOptionalStringFlag(flags, 'phase')
|
||||
const hasStructuredPayload =
|
||||
taskId !== undefined ||
|
||||
dispatchId !== undefined ||
|
||||
outcome !== undefined ||
|
||||
filesModified !== undefined ||
|
||||
reportPath !== undefined ||
|
||||
phase !== undefined
|
||||
@@ -116,6 +139,15 @@ function getOptionalStructuredMessagePayload(
|
||||
if (dispatchId) {
|
||||
payload.dispatchId = dispatchId
|
||||
}
|
||||
if (outcome) {
|
||||
if (outcome !== 'succeeded' && outcome !== 'failed') {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'Invalid --outcome. Expected succeeded or failed.'
|
||||
)
|
||||
}
|
||||
payload.outcome = outcome
|
||||
}
|
||||
if (filesModified) {
|
||||
payload.filesModified = filesModified
|
||||
.split(',')
|
||||
@@ -163,29 +195,6 @@ async function resolveOrchestrationTerminalHandle(
|
||||
return await getTerminalHandle(flags, cwd, client)
|
||||
}
|
||||
|
||||
async function resolveTaskCreatorTerminalHandle(
|
||||
client: Parameters<CommandHandler>[0]['client']
|
||||
): Promise<string | undefined> {
|
||||
const envHandle = process.env.ORCA_TERMINAL_HANDLE
|
||||
if (!envHandle || envHandle.length === 0) {
|
||||
return undefined
|
||||
}
|
||||
let live: boolean
|
||||
try {
|
||||
live = await isLiveTerminalHandle(envHandle, client)
|
||||
} catch (err) {
|
||||
if (isOptionalTaskCreatorHandleError(err)) {
|
||||
// Why: creator handles are best-effort lineage metadata; graph unavailability must not block task creation.
|
||||
return undefined
|
||||
}
|
||||
throw err
|
||||
}
|
||||
if (live) {
|
||||
return envHandle
|
||||
}
|
||||
return await resolveOrchestrationPaneTerminalHandle(client, { optional: true })
|
||||
}
|
||||
|
||||
async function isLiveTerminalHandle(
|
||||
handle: string,
|
||||
client: Parameters<CommandHandler>[0]['client']
|
||||
@@ -218,11 +227,6 @@ function isNoActiveTerminalError(err: unknown): boolean {
|
||||
return getClientErrorCode(err) === 'no_active_terminal'
|
||||
}
|
||||
|
||||
function isOptionalTaskCreatorHandleError(err: unknown): boolean {
|
||||
const code = getClientErrorCode(err)
|
||||
return code === 'no_active_sender_terminal' || code === 'runtime_unavailable'
|
||||
}
|
||||
|
||||
async function resolveOrchestrationPaneTerminalHandle(
|
||||
client: Parameters<CommandHandler>[0]['client'],
|
||||
options: { optional?: boolean } = {}
|
||||
@@ -310,7 +314,10 @@ function throwNoActiveSenderTerminal(): never {
|
||||
}
|
||||
|
||||
function isDevCliInvocation(): boolean {
|
||||
return process.env.ORCA_USER_DATA_PATH?.includes('orca-dev') ?? false
|
||||
return (
|
||||
process.env.ORCA_DEV_CLI_INVOCATION === '1' ||
|
||||
(process.env.ORCA_USER_DATA_PATH?.includes('orca-dev') ?? false)
|
||||
)
|
||||
}
|
||||
|
||||
function getOptionalPositiveIntegerValueFlag(
|
||||
@@ -340,11 +347,148 @@ function rejectLifecycleGroupRecipient(type: string | undefined, to: string): vo
|
||||
}
|
||||
}
|
||||
|
||||
function callMutation<TResult>(
|
||||
client: RuntimeClient,
|
||||
flags: Map<string, string | boolean>,
|
||||
method: string,
|
||||
params: unknown,
|
||||
options?: { timeoutMs?: number; orchestrationCapability?: string }
|
||||
) {
|
||||
const requestId = getOptionalStringFlag(flags, 'retry-request')
|
||||
if (!requestId) {
|
||||
return options
|
||||
? client.call<TResult>(method, params, options)
|
||||
: client.call<TResult>(method, params)
|
||||
}
|
||||
return client.call<TResult>(method, params, {
|
||||
...options,
|
||||
orchestrationRequestId: requestId
|
||||
})
|
||||
}
|
||||
|
||||
type LegacyWorkerReadResult = {
|
||||
dispatchId: string
|
||||
terminal: RuntimeTerminalRead
|
||||
}
|
||||
|
||||
function formatWorkerRead(value: OrchestrationWorkerReadResult | LegacyWorkerReadResult): string {
|
||||
if (!('source' in value) || value.source === 'terminal') {
|
||||
return value.terminal.tail.join('\n')
|
||||
}
|
||||
return value.transcript.messages.map(formatWorkerTranscriptMessage).join('\n\n')
|
||||
}
|
||||
|
||||
function formatWorkerTranscriptMessage(message: NativeChatMessage): string {
|
||||
const blocks = message.blocks.map((block) => {
|
||||
if (block.type === 'text') {
|
||||
return block.text
|
||||
}
|
||||
if (block.type === 'tool-call') {
|
||||
return `[tool ${block.name}] ${safeJson(block.input)}`
|
||||
}
|
||||
if (block.type === 'tool-result') {
|
||||
return `[tool result${block.isError ? ' error' : ''}] ${block.output}`
|
||||
}
|
||||
return block.url ? `[image] ${block.url}` : `[image omitted]`
|
||||
})
|
||||
return `[${message.role}] ${blocks.join('\n')}`.trimEnd()
|
||||
}
|
||||
|
||||
function safeJson(value: unknown): string {
|
||||
try {
|
||||
return JSON.stringify(value)
|
||||
} catch {
|
||||
return '[unserializable input]'
|
||||
}
|
||||
}
|
||||
|
||||
export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
'orchestration run-create': async ({ flags, client, cwd, json }) => {
|
||||
const from = await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const result = await callMutation<{
|
||||
run: { id: string; objective: string; consumer_generation: number }
|
||||
}>(client, flags, 'orchestration.runCreate', {
|
||||
objective: getRequiredStringFlag(flags, 'objective'),
|
||||
from
|
||||
})
|
||||
printResult(result, json, (r) => `Run ${r.run.id} created and bound: ${r.run.objective}`)
|
||||
},
|
||||
|
||||
'orchestration run-use': async ({ flags, client, cwd, json }) => {
|
||||
const from = await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const result = await callMutation<{
|
||||
run: { id: string; objective: string; consumer_generation: number }
|
||||
}>(client, flags, 'orchestration.runUse', {
|
||||
id: getRequiredStringFlag(flags, 'id'),
|
||||
from
|
||||
})
|
||||
printResult(result, json, (r) => `Using Run ${r.run.id}: ${r.run.objective}`)
|
||||
},
|
||||
|
||||
'orchestration run-current': async ({ flags, client, cwd, json }) => {
|
||||
const from = await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const result = await client.call<{
|
||||
run: { id: string; objective: string } | null
|
||||
}>('orchestration.runCurrent', { from })
|
||||
printResult(result, json, (r) =>
|
||||
r.run ? `${r.run.id} ${r.run.objective}` : 'No Run is bound to this terminal.'
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration run-list': async ({ client, json }) => {
|
||||
const result = await client.call<{
|
||||
runs: { id: string; objective: string; legacy: number }[]
|
||||
}>('orchestration.runList', {})
|
||||
printResult(result, json, (r) =>
|
||||
r.runs.length === 0
|
||||
? 'No Runs found.'
|
||||
: r.runs
|
||||
.map(
|
||||
(run) => `${run.id}${run.legacy ? ' [legacy, inspect only]' : ''} ${run.objective}`
|
||||
)
|
||||
.join('\n')
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration run-show': async ({ flags, client, json }) => {
|
||||
const result = await client.call<{
|
||||
run: {
|
||||
id: string
|
||||
objective: string
|
||||
consumer_generation: number
|
||||
legacy: number
|
||||
created_at: string
|
||||
}
|
||||
}>('orchestration.runShow', { id: getRequiredStringFlag(flags, 'id') })
|
||||
printResult(
|
||||
result,
|
||||
json,
|
||||
(r) =>
|
||||
`${r.run.id}${r.run.legacy ? ' [legacy, inspect only]' : ''} ${r.run.objective}\n` +
|
||||
`consumer generation ${r.run.consumer_generation}; created ${r.run.created_at}`
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration send': async ({ flags, client, cwd, json }) => {
|
||||
const to = getRequiredStringFlag(flags, 'to')
|
||||
const to = getOptionalStringFlag(flags, 'to')
|
||||
const type = getOptionalStringFlag(flags, 'type')
|
||||
rejectLifecycleGroupRecipient(type, to)
|
||||
if (to) {
|
||||
rejectLifecycleGroupRecipient(type, to)
|
||||
}
|
||||
const outcome = getOptionalStringFlag(flags, 'outcome')
|
||||
if (type === 'worker_done' && outcome === undefined && !flags.has('payload')) {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'worker_done requires --outcome succeeded or --outcome failed. No effects were applied.',
|
||||
orchestrationSkillRecoveryData()
|
||||
)
|
||||
}
|
||||
if (type !== 'worker_done' && outcome !== undefined) {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'--outcome is only valid with --type worker_done.'
|
||||
)
|
||||
}
|
||||
|
||||
if (
|
||||
(type === 'worker_done' || type === 'heartbeat') &&
|
||||
@@ -357,9 +501,10 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
|
||||
// Why: lifecycle senders keep ORCA_TERMINAL_HANDLE verbatim — no liveness probe (worker_done must survive the mid-restart window) and no remint (older runtimes require from === the stale assignee_handle).
|
||||
const from = await resolveOrchestrationTerminalHandle(flags, cwd, client, 'from')
|
||||
const result = await client.call<OrchestrationSendResult>('orchestration.send', {
|
||||
const sendParams = {
|
||||
from,
|
||||
to,
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
subject: getRequiredStringFlag(flags, 'subject'),
|
||||
body: getOptionalStringFlag(flags, 'body'),
|
||||
type,
|
||||
@@ -369,7 +514,15 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
// Why: pane key is the remint-stable sender identity the runtime verifies lifecycle ownership against; older runtimes strip it.
|
||||
senderPaneKey: process.env.ORCA_PANE_KEY || undefined,
|
||||
devMode: isDevCliInvocation()
|
||||
})
|
||||
}
|
||||
const dispatchCapability = getOptionalStringFlag(flags, 'dispatch-capability')
|
||||
const result = await callMutation<OrchestrationSendResult>(
|
||||
client,
|
||||
flags,
|
||||
'orchestration.send',
|
||||
sendParams,
|
||||
dispatchCapability ? { orchestrationCapability: dispatchCapability } : undefined
|
||||
)
|
||||
if ('message' in result.result && result.result.lifecycle?.action === 'rejected') {
|
||||
// Why: a rejected lifecycle signal isn't completion; non-zero exit stops workers from treating it as such.
|
||||
process.exitCode = 1
|
||||
@@ -381,6 +534,12 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
}
|
||||
return `Sent ${r.message.id}`
|
||||
}
|
||||
if ('relay' in r) {
|
||||
if (r.relay.destination === 'worker') {
|
||||
return `Queued ${r.relay.messageId} for worker Dispatch ${r.relay.dispatchId}`
|
||||
}
|
||||
return `Queued ${r.relay.messageId} for Run home (Dispatch ${r.relay.dispatchId})`
|
||||
}
|
||||
return `Sent ${r.messages.length} messages to ${r.recipients} recipients`
|
||||
})
|
||||
},
|
||||
@@ -404,17 +563,24 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
messages: MessageSummary[]
|
||||
count: number
|
||||
formatted?: string
|
||||
deliveryId?: string | null
|
||||
runId?: string
|
||||
timedOut?: boolean
|
||||
cancelled?: boolean
|
||||
connectionLost?: boolean
|
||||
}
|
||||
let result: Awaited<ReturnType<typeof client.call<CheckResult>>>
|
||||
try {
|
||||
result = await client.call<CheckResult>('orchestration.check', {
|
||||
result = await callMutation<CheckResult>(client, flags, 'orchestration.check', {
|
||||
terminal,
|
||||
// Why: peek also sends unread:false so pre-peek runtimes degrade to non-consuming all mode instead of destructive mark-read.
|
||||
unread: flags.has('unread') ? true : peek ? false : undefined,
|
||||
peek: peek ? true : undefined,
|
||||
all: flags.has('all') ? true : undefined,
|
||||
types: getOptionalStringFlag(flags, 'types'),
|
||||
inject: flags.has('inject') ? true : undefined,
|
||||
format: flags.has('format') ? true : undefined,
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
ack: getOptionalStringFlag(flags, 'ack'),
|
||||
wait: wait ? true : undefined,
|
||||
timeoutMs
|
||||
})
|
||||
@@ -454,21 +620,36 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
return r.formatted
|
||||
}
|
||||
if (r.count === 0) {
|
||||
if (r.timedOut) {
|
||||
return 'Wait timed out; no messages were consumed.'
|
||||
}
|
||||
if (r.cancelled) {
|
||||
return r.connectionLost
|
||||
? 'Wait cancelled because the connection closed; no messages were consumed.'
|
||||
: 'Wait cancelled; no messages were consumed.'
|
||||
}
|
||||
return 'No messages.'
|
||||
}
|
||||
return r.messages
|
||||
const rendered = r.messages
|
||||
.map((m) => `${m.id} [${m.type ?? 'status'}] from=${m.from_handle} "${m.subject}"`)
|
||||
.join('\n')
|
||||
return r.deliveryId ? `Delivery ${r.deliveryId}\n${rendered}` : rendered
|
||||
})
|
||||
},
|
||||
|
||||
'orchestration reply': async ({ flags, client, cwd, json }) => {
|
||||
const from = await resolveOrchestrationTerminalHandle(flags, cwd, client, 'from')
|
||||
const result = await client.call<{ message: { id: string } }>('orchestration.reply', {
|
||||
id: getRequiredStringFlag(flags, 'id'),
|
||||
body: getRequiredStringFlag(flags, 'body'),
|
||||
from
|
||||
})
|
||||
const result = await callMutation<{ message: { id: string } }>(
|
||||
client,
|
||||
flags,
|
||||
'orchestration.reply',
|
||||
{
|
||||
id: getRequiredStringFlag(flags, 'id'),
|
||||
body: getRequiredStringFlag(flags, 'body'),
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
from
|
||||
}
|
||||
)
|
||||
printResult(result, json, (r) => `Replied ${r.message.id}`)
|
||||
},
|
||||
|
||||
@@ -505,9 +686,11 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
})
|
||||
},
|
||||
|
||||
'orchestration task-create': async ({ flags, client, json }) => {
|
||||
const callerTerminalHandle = await resolveTaskCreatorTerminalHandle(client)
|
||||
const result = await client.call<{ task: { id: string; status: string } }>(
|
||||
'orchestration task-create': async ({ flags, client, cwd, json }) => {
|
||||
const callerTerminalHandle = await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const result = await callMutation<{ task: { id: string; status: string } }>(
|
||||
client,
|
||||
flags,
|
||||
'orchestration.taskCreate',
|
||||
{
|
||||
spec: getRequiredStringFlag(flags, 'spec'),
|
||||
@@ -515,14 +698,19 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
displayName: getOptionalStringFlag(flags, 'display-name'),
|
||||
deps: getOptionalStringFlag(flags, 'deps'),
|
||||
parent: getOptionalStringFlag(flags, 'parent'),
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
callerTerminalHandle
|
||||
}
|
||||
)
|
||||
printResult(result, json, (r) => `Created ${r.task.id} [${r.task.status}]`)
|
||||
},
|
||||
|
||||
'orchestration task-list': async ({ flags, client, json }) => {
|
||||
'orchestration task-list': async ({ flags, client, cwd, json }) => {
|
||||
const brief = flags.has('brief')
|
||||
const run = getOptionalStringFlag(flags, 'run')
|
||||
const callerTerminalHandle = run
|
||||
? undefined
|
||||
: await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const result = await client.call<{
|
||||
tasks: {
|
||||
id: string
|
||||
@@ -535,10 +723,14 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
spec_truncated?: boolean
|
||||
}[]
|
||||
count: number
|
||||
runId?: string
|
||||
legacyReadOnly?: boolean
|
||||
}>('orchestration.taskList', {
|
||||
status: getOptionalStringFlag(flags, 'status'),
|
||||
ready: flags.has('ready') ? true : undefined,
|
||||
brief: brief ? true : undefined
|
||||
brief: brief ? true : undefined,
|
||||
run,
|
||||
callerTerminalHandle
|
||||
})
|
||||
// Why: only older runtimes (no spec_truncated) skip server-side abbreviation and need this client-side fallback.
|
||||
const needsClientAbbreviation =
|
||||
@@ -551,9 +743,9 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
: result
|
||||
printResult(output, json, (r) => {
|
||||
if (r.count === 0) {
|
||||
return 'No tasks.'
|
||||
return r.legacyReadOnly ? 'No legacy tasks (read-only).' : 'No tasks.'
|
||||
}
|
||||
return r.tasks
|
||||
const tasks = r.tasks
|
||||
.map((t) => {
|
||||
const label = t.display_name ?? t.task_title ?? t.spec
|
||||
const head = `${t.id} [${t.status}] ${label.slice(0, 60)}`
|
||||
@@ -563,10 +755,11 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
return head
|
||||
})
|
||||
.join('\n')
|
||||
return r.legacyReadOnly ? `Legacy Run ${r.runId} (read-only)\n${tasks}` : tasks
|
||||
})
|
||||
},
|
||||
|
||||
'orchestration task-update': async ({ flags, client, json }) => {
|
||||
'orchestration task-update': async ({ flags, client, cwd, json }) => {
|
||||
const status = getRequiredStringFlag(flags, 'status')
|
||||
if (!TASK_STATUS_VALUES.includes(status as (typeof TASK_STATUS_VALUES)[number])) {
|
||||
throw new RuntimeClientError(
|
||||
@@ -574,30 +767,149 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
`invalid status '${status}', expected one of: ${TASK_STATUS_VALUES.join(', ')}`
|
||||
)
|
||||
}
|
||||
const result = await client.call<{ task: { id: string; status: string } }>(
|
||||
const result = await callMutation<{ task: { id: string; status: string } }>(
|
||||
client,
|
||||
flags,
|
||||
'orchestration.taskUpdate',
|
||||
{
|
||||
id: getRequiredStringFlag(flags, 'id'),
|
||||
status,
|
||||
result: getOptionalStringFlag(flags, 'result')
|
||||
result: getOptionalStringFlag(flags, 'result'),
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
callerTerminalHandle: await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
}
|
||||
)
|
||||
printResult(result, json, (r) => `Updated ${r.task.id} -> ${r.task.status}`)
|
||||
},
|
||||
|
||||
'orchestration worker-start': async ({ flags, client, cwd, json }) => {
|
||||
const result = await callMutation<{
|
||||
runId: string
|
||||
taskId: string
|
||||
dispatchId: string
|
||||
state: string
|
||||
failedStage?: string
|
||||
lastError?: string
|
||||
effects: unknown[]
|
||||
residualResources: unknown[]
|
||||
}>(client, flags, 'orchestration.workerStart', {
|
||||
task: getRequiredStringFlag(flags, 'task'),
|
||||
on: getOptionalStringFlag(flags, 'on'),
|
||||
worktree: getOptionalStringFlag(flags, 'worktree'),
|
||||
name: getOptionalStringFlag(flags, 'name'),
|
||||
repo: getOptionalStringFlag(flags, 'repo'),
|
||||
baseBranch: getOptionalStringFlag(flags, 'base-branch'),
|
||||
displayName: getOptionalStringFlag(flags, 'display-name'),
|
||||
comment: getOptionalStringFlag(flags, 'comment'),
|
||||
setup: getOptionalStringFlag(flags, 'setup'),
|
||||
agent: getOptionalStringFlag(flags, 'agent'),
|
||||
terminal: getOptionalStringFlag(flags, 'terminal'),
|
||||
retryOf: getOptionalStringFlag(flags, 'retry-of'),
|
||||
timeoutMs: getOptionalPositiveIntegerValueFlag(flags, 'timeout-ms'),
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
from: await resolveCoordinatorTerminalHandle(flags, cwd, client),
|
||||
devMode: isDevCliInvocation()
|
||||
})
|
||||
if (result.result.state !== 'ready') {
|
||||
process.exitCode = 1
|
||||
}
|
||||
printResult(result, json, (worker) => {
|
||||
const base = `Worker ${worker.dispatchId} [${worker.state}] for ${worker.taskId}`
|
||||
return worker.lastError
|
||||
? `${base}\n${worker.failedStage ?? 'start'}: ${worker.lastError}`
|
||||
: base
|
||||
})
|
||||
},
|
||||
|
||||
'orchestration worker-show': async ({ flags, client, json }) => {
|
||||
const result = await client.call<{
|
||||
dispatch: { id: string; task_id: string; status: string }
|
||||
worker: { state: string; stage: string; agent_terminal_handle: string | null }
|
||||
}>('orchestration.workerShow', {
|
||||
dispatch: getRequiredStringFlag(flags, 'dispatch')
|
||||
})
|
||||
printResult(
|
||||
result,
|
||||
json,
|
||||
(value) =>
|
||||
`${value.dispatch.id} task=${value.dispatch.task_id} [${value.worker.state}] stage=${value.worker.stage}`
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration worker-read': async ({ flags, client, json }) => {
|
||||
const cursorFlag = getOptionalStringFlag(flags, 'cursor')
|
||||
const cursor =
|
||||
cursorFlag !== undefined && /^\d+$/.test(cursorFlag)
|
||||
? Number.parseInt(cursorFlag, 10)
|
||||
: cursorFlag
|
||||
const source = getOptionalStringFlag(flags, 'source')
|
||||
if (source && !['auto', 'transcript', 'terminal'].includes(source)) {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'--source must be auto, transcript, or terminal'
|
||||
)
|
||||
}
|
||||
const result = await client.call<OrchestrationWorkerReadResult | LegacyWorkerReadResult>(
|
||||
'orchestration.workerRead',
|
||||
{
|
||||
dispatch: getRequiredStringFlag(flags, 'dispatch'),
|
||||
cursor,
|
||||
limit: getOptionalPositiveIntegerFlag(flags, 'limit'),
|
||||
source: source as OrchestrationWorkerReadSource | undefined
|
||||
}
|
||||
)
|
||||
printResult(result, json, formatWorkerRead)
|
||||
},
|
||||
|
||||
'orchestration worker-stop': async ({ flags, client, json }) => {
|
||||
const result = await callMutation<{
|
||||
dispatchId: string
|
||||
state: string
|
||||
processAction: string
|
||||
lastError?: string
|
||||
}>(client, flags, 'orchestration.workerStop', {
|
||||
dispatch: getRequiredStringFlag(flags, 'dispatch')
|
||||
})
|
||||
if (result.result.state === 'stop_unknown') {
|
||||
process.exitCode = 1
|
||||
}
|
||||
printResult(
|
||||
result,
|
||||
json,
|
||||
(value) =>
|
||||
`Worker ${value.dispatchId} [${value.state}] process=${value.processAction}${value.lastError ? `\n${value.lastError}` : ''}`
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration worker-abandon': async ({ flags, client, json }) => {
|
||||
const result = await callMutation<{
|
||||
dispatchId: string
|
||||
state: string
|
||||
warning: string
|
||||
}>(client, flags, 'orchestration.workerAbandon', {
|
||||
dispatch: getRequiredStringFlag(flags, 'dispatch')
|
||||
})
|
||||
printResult(
|
||||
result,
|
||||
json,
|
||||
(value) => `Worker ${value.dispatchId} [${value.state}]\nWarning: ${value.warning}`
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration dispatch': async ({ flags, client, cwd, json }) => {
|
||||
const from = await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const dryRun = flags.has('dry-run') ? true : undefined
|
||||
const returnPreamble = flags.has('return-preamble') ? true : undefined
|
||||
// Why: --to is only required for non-dry-run; the RPC handler re-enforces.
|
||||
const to = dryRun ? getOptionalStringFlag(flags, 'to') : getRequiredStringFlag(flags, 'to')
|
||||
const result = await client.call<{
|
||||
const result = await callMutation<{
|
||||
dispatch: { id: string; task_id: string; status: string } | null
|
||||
injected?: boolean
|
||||
dryRun?: boolean
|
||||
preamble?: string
|
||||
}>('orchestration.dispatch', {
|
||||
}>(client, flags, 'orchestration.dispatch', {
|
||||
task: getRequiredStringFlag(flags, 'task'),
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
to,
|
||||
from,
|
||||
inject: flags.has('inject') ? true : undefined,
|
||||
@@ -618,23 +930,46 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
const parsedTimeoutMs = getOptionalPositiveIntegerValueFlag(flags, 'timeout-ms')
|
||||
const timeoutMs = clampOrchestrationAskTimeoutMs(parsedTimeoutMs)
|
||||
const from = await resolveOrchestrationTerminalHandle(flags, cwd, client, 'from')
|
||||
const result = await client.call<{
|
||||
const question = getOptionalStringFlag(flags, 'question')
|
||||
const resume = getOptionalStringFlag(flags, 'resume')
|
||||
if ((question ? 1 : 0) + (resume ? 1 : 0) !== 1) {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'Choose exactly one of --question or --resume.'
|
||||
)
|
||||
}
|
||||
if (resume && flags.has('options')) {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'--options is only valid when creating a new question.'
|
||||
)
|
||||
}
|
||||
const result = await callMutation<{
|
||||
answer: string | null
|
||||
messageId: string | null
|
||||
threadId: string
|
||||
timedOut: boolean
|
||||
timeoutMs?: number
|
||||
cancelled?: boolean
|
||||
connectionLost?: boolean
|
||||
}>(
|
||||
client,
|
||||
flags,
|
||||
'orchestration.ask',
|
||||
{
|
||||
to: getRequiredStringFlag(flags, 'to'),
|
||||
question: getRequiredStringFlag(flags, 'question'),
|
||||
to: getOptionalStringFlag(flags, 'to'),
|
||||
run: getOptionalStringFlag(flags, 'run'),
|
||||
question,
|
||||
resume,
|
||||
options: getOptionalStringFlag(flags, 'options'),
|
||||
timeoutMs: parsedTimeoutMs === undefined ? undefined : timeoutMs,
|
||||
from
|
||||
},
|
||||
// Why: extend past timeoutMs so the RPC transport's 60s default doesn't abort before the runtime's own timeout resolves.
|
||||
{ timeoutMs: resolveOrchestrationAskClientTimeoutMs(parsedTimeoutMs) }
|
||||
{
|
||||
timeoutMs: resolveOrchestrationAskClientTimeoutMs(parsedTimeoutMs),
|
||||
orchestrationCapability: getOptionalStringFlag(flags, 'dispatch-capability')
|
||||
}
|
||||
)
|
||||
// Why: bypass printResult so --json emits a bare JSON object (no envelope) pipeable via `jq -r .answer`, unlike other verbs.
|
||||
if (json) {
|
||||
@@ -650,6 +985,16 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
}
|
||||
process.exitCode = 1
|
||||
}
|
||||
if (result.result.cancelled) {
|
||||
if (!json) {
|
||||
console.error(
|
||||
result.result.connectionLost
|
||||
? `ask connection closed (question ${result.result.messageId})`
|
||||
: `ask cancelled (question ${result.result.messageId})`
|
||||
)
|
||||
}
|
||||
process.exitCode = 1
|
||||
}
|
||||
},
|
||||
|
||||
'orchestration dispatch-show': async ({ flags, client, cwd, json }) => {
|
||||
@@ -678,33 +1023,26 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
})
|
||||
},
|
||||
|
||||
'orchestration run': async ({ flags, client, cwd, json }) => {
|
||||
const from = await resolveCoordinatorTerminalHandle(flags, cwd, client)
|
||||
const result = await client.call<{
|
||||
runId: string
|
||||
status: string
|
||||
}>('orchestration.run', {
|
||||
spec: getRequiredStringFlag(flags, 'spec'),
|
||||
from,
|
||||
pollIntervalMs: getOptionalPositiveIntegerFlag(flags, 'poll-interval-ms'),
|
||||
maxConcurrent: getOptionalPositiveIntegerFlag(flags, 'max-concurrent'),
|
||||
worktree: getOptionalStringFlag(flags, 'worktree')
|
||||
})
|
||||
printResult(result, json, (r) => `Run ${r.runId} started (${r.status})`)
|
||||
'orchestration coordinator-start': async () => {
|
||||
throw new RuntimeClientError(
|
||||
'orchestration_migration_required',
|
||||
'The legacy automatic coordinator command is retired. No effects were applied.',
|
||||
orchestrationMigrationData('command_retired')
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration run-stop': async ({ client, json }) => {
|
||||
const result = await client.call<{
|
||||
runId: string
|
||||
stopped: boolean
|
||||
}>('orchestration.runStop', {})
|
||||
printResult(result, json, (r) => `Run ${r.runId} stopped`)
|
||||
'orchestration coordinator-stop': async () => {
|
||||
throw new RuntimeClientError(
|
||||
'orchestration_migration_required',
|
||||
'The legacy automatic coordinator command is retired. No effects were applied.',
|
||||
orchestrationMigrationData('command_retired')
|
||||
)
|
||||
},
|
||||
|
||||
'orchestration gate-create': async ({ flags, client, json }) => {
|
||||
const result = await client.call<{
|
||||
const result = await callMutation<{
|
||||
gate: { id: string; task_id: string; status: string }
|
||||
}>('orchestration.gateCreate', {
|
||||
}>(client, flags, 'orchestration.gateCreate', {
|
||||
task: getRequiredStringFlag(flags, 'task'),
|
||||
question: getRequiredStringFlag(flags, 'question'),
|
||||
options: getOptionalStringFlag(flags, 'options')
|
||||
@@ -717,9 +1055,9 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
},
|
||||
|
||||
'orchestration gate-resolve': async ({ flags, client, json }) => {
|
||||
const result = await client.call<{
|
||||
const result = await callMutation<{
|
||||
gate: { id: string; task_id: string; status: string; resolution: string }
|
||||
}>('orchestration.gateResolve', {
|
||||
}>(client, flags, 'orchestration.gateResolve', {
|
||||
id: getRequiredStringFlag(flags, 'id'),
|
||||
resolution: getRequiredStringFlag(flags, 'resolution')
|
||||
})
|
||||
@@ -745,9 +1083,17 @@ export const ORCHESTRATION_HANDLERS: Record<string, CommandHandler> = {
|
||||
},
|
||||
|
||||
'orchestration reset': async ({ flags, client, json }) => {
|
||||
const hasScopeFlag = flags.has('all') || flags.has('tasks') || flags.has('messages')
|
||||
const result = await client.call<{ reset: string }>('orchestration.reset', {
|
||||
all: flags.has('all') || !hasScopeFlag ? true : undefined,
|
||||
const scopeCount = [flags.has('all'), flags.has('tasks'), flags.has('messages')].filter(
|
||||
Boolean
|
||||
).length
|
||||
if (scopeCount !== 1) {
|
||||
throw new RuntimeClientError(
|
||||
'invalid_argument',
|
||||
'Choose exactly one reset scope: --all, --tasks, or --messages.'
|
||||
)
|
||||
}
|
||||
const result = await callMutation<{ reset: string }>(client, flags, 'orchestration.reset', {
|
||||
all: flags.has('all') ? true : undefined,
|
||||
tasks: flags.has('tasks') ? true : undefined,
|
||||
messages: flags.has('messages') ? true : undefined
|
||||
})
|
||||
|
||||
+17
-3
@@ -85,8 +85,14 @@ Terminals:
|
||||
terminal close Close a terminal pane/session, or its whole tab with --tab
|
||||
|
||||
Orchestration:
|
||||
orchestration run-create Create and bind a lightweight orchestration Run
|
||||
orchestration run-use Bind this coordinator terminal to an existing Run
|
||||
orchestration run-current Show this terminal's bound Run
|
||||
orchestration run-list List lightweight orchestration Runs
|
||||
orchestration run-show Show one lightweight orchestration Run
|
||||
orchestration send Send an inter-agent message
|
||||
orchestration check Check messages for a terminal
|
||||
orchestration check Check the bound Run mailbox
|
||||
orchestration ask Ask the coordinator a blocking question
|
||||
orchestration reply Reply to a message
|
||||
orchestration inbox Show all messages across recipients
|
||||
orchestration task-create Create an orchestration task
|
||||
@@ -94,8 +100,13 @@ Orchestration:
|
||||
orchestration task-update Update a task status
|
||||
orchestration dispatch Dispatch a task to a terminal
|
||||
orchestration dispatch-show Show dispatch context for a task
|
||||
orchestration run Start the coordinator loop
|
||||
orchestration run-stop Stop the active coordinator run
|
||||
orchestration worker-start Start a supervised worker locally or on a connected Orca server
|
||||
orchestration worker-show Inspect one supervised worker
|
||||
orchestration worker-read Read bounded output from one supervised worker
|
||||
orchestration worker-stop Stop one supervised worker
|
||||
orchestration worker-abandon Fence an uncertain worker without claiming it stopped
|
||||
orchestration coordinator-start Start the legacy automatic coordinator loop
|
||||
orchestration coordinator-stop Stop the legacy automatic coordinator loop
|
||||
orchestration gate-create Create a decision gate blocking a task
|
||||
orchestration gate-resolve Resolve a pending decision gate
|
||||
orchestration gate-list List decision gates
|
||||
@@ -419,6 +430,9 @@ function formatCommandFlagHelp(flag: string, commandPath: string[]): string {
|
||||
if (command === 'linear list-issues' && flag === 'cursor') {
|
||||
return '--cursor <cursor> Opaque cursor returned by a previous list-issues page'
|
||||
}
|
||||
if (command === 'orchestration worker-read' && flag === 'cursor') {
|
||||
return '--cursor <cursor> Opaque cursor returned by a previous worker-read page'
|
||||
}
|
||||
if (command === 'linear list-issues' && flag === 'workspace') {
|
||||
return '--workspace <id|all> Connected Linear workspace id, or all'
|
||||
}
|
||||
|
||||
+70
-19
@@ -353,6 +353,15 @@ describe('orca root help', () => {
|
||||
expect(logSpy.mock.calls[0][0]).toContain(
|
||||
'orca terminal create --worktree active --command "codex"'
|
||||
)
|
||||
expect(logSpy.mock.calls[0][0]).toContain(
|
||||
'orchestration worker-start Start a supervised worker locally or on a connected Orca server'
|
||||
)
|
||||
expect(logSpy.mock.calls[0][0]).toContain(
|
||||
'orchestration ask Ask the coordinator a blocking question'
|
||||
)
|
||||
expect(logSpy.mock.calls[0][0]).toContain(
|
||||
'orchestration worker-abandon Fence an uncertain worker without claiming it stopped'
|
||||
)
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
@@ -408,6 +417,20 @@ describe('orca root help', () => {
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('describes worker-read cursors as opaque', async () => {
|
||||
const logSpy = vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
logSpy.mockClear()
|
||||
|
||||
await main(['orchestration', 'worker-read', '--help'], '/tmp/repo')
|
||||
|
||||
const help = String(logSpy.mock.calls[0][0])
|
||||
expect(help).toContain(
|
||||
'--cursor <cursor> Opaque cursor returned by a previous worker-read page'
|
||||
)
|
||||
expect(help).not.toContain('Line cursor from a previous read')
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('advertises Linear issue linking on worktree create and set help', async () => {
|
||||
const logSpy = vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
logSpy.mockClear()
|
||||
@@ -501,6 +524,7 @@ describe('orca root help', () => {
|
||||
describe('orca cli worktree awareness', () => {
|
||||
const originalTerminalHandle = process.env.ORCA_TERMINAL_HANDLE
|
||||
const originalUserDataPath = process.env.ORCA_USER_DATA_PATH
|
||||
const originalDevCliInvocation = process.env.ORCA_DEV_CLI_INVOCATION
|
||||
const originalPairingCode = process.env.ORCA_PAIRING_CODE
|
||||
const originalRemotePairing = process.env.ORCA_REMOTE_PAIRING
|
||||
const originalEnvironment = process.env.ORCA_ENVIRONMENT
|
||||
@@ -511,6 +535,7 @@ describe('orca cli worktree awareness', () => {
|
||||
callMock.mockReset()
|
||||
delete process.env.ORCA_TERMINAL_HANDLE
|
||||
delete process.env.ORCA_USER_DATA_PATH
|
||||
delete process.env.ORCA_DEV_CLI_INVOCATION
|
||||
delete process.env.ORCA_WORKSPACE_ID
|
||||
delete process.env.ORCA_WORKTREE_ID
|
||||
// Isolate the pane key so claude-teams tests that set it don't leak a
|
||||
@@ -555,6 +580,11 @@ describe('orca cli worktree awareness', () => {
|
||||
} else {
|
||||
process.env.ORCA_USER_DATA_PATH = originalUserDataPath
|
||||
}
|
||||
if (originalDevCliInvocation === undefined) {
|
||||
delete process.env.ORCA_DEV_CLI_INVOCATION
|
||||
} else {
|
||||
process.env.ORCA_DEV_CLI_INVOCATION = originalDevCliInvocation
|
||||
}
|
||||
if (originalPairingCode === undefined) {
|
||||
delete process.env.ORCA_PAIRING_CODE
|
||||
} else {
|
||||
@@ -3489,17 +3519,11 @@ describe('orca cli worktree awareness', () => {
|
||||
expect(logSpy).toHaveBeenCalledWith('Sent 2 messages to 2 recipients')
|
||||
})
|
||||
|
||||
it('passes all reset scope explicitly for no-flag orchestration reset', async () => {
|
||||
callMock.mockResolvedValueOnce(okFixture('req_reset', { reset: 'all' }))
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
|
||||
it('rejects no-flag orchestration reset before calling the runtime', async () => {
|
||||
await main(['orchestration', 'reset'], '/tmp/repo')
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', {
|
||||
all: true,
|
||||
tasks: undefined,
|
||||
messages: undefined
|
||||
})
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
expect(process.exitCode).toBe(1)
|
||||
})
|
||||
|
||||
it.each([
|
||||
@@ -3517,16 +3541,6 @@ describe('orca cli worktree awareness', () => {
|
||||
args: ['orchestration', 'reset', '--messages'],
|
||||
params: { all: undefined, tasks: undefined, messages: true },
|
||||
reset: 'messages'
|
||||
},
|
||||
{
|
||||
args: ['orchestration', 'reset', '--tasks', '--messages'],
|
||||
params: { all: undefined, tasks: true, messages: true },
|
||||
reset: 'tasks'
|
||||
},
|
||||
{
|
||||
args: ['orchestration', 'reset', '--all', '--tasks'],
|
||||
params: { all: true, tasks: true, messages: undefined },
|
||||
reset: 'all'
|
||||
}
|
||||
])('passes explicit reset flags through for $args', async ({ args, params, reset }) => {
|
||||
callMock.mockResolvedValueOnce(okFixture('req_reset', { reset }))
|
||||
@@ -3537,6 +3551,16 @@ describe('orca cli worktree awareness', () => {
|
||||
expect(callMock).toHaveBeenCalledWith('orchestration.reset', params)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['orchestration', 'reset', '--tasks', '--messages'],
|
||||
['orchestration', 'reset', '--all', '--tasks']
|
||||
])('rejects conflicting reset scopes for $args', async (...args) => {
|
||||
await main(args, '/tmp/repo')
|
||||
|
||||
expect(callMock).not.toHaveBeenCalled()
|
||||
expect(process.exitCode).toBe(1)
|
||||
})
|
||||
|
||||
it('rejects unknown task-update status with an enum-aware error', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_coord'
|
||||
const logSpy = vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
@@ -3629,6 +3653,33 @@ describe('orca cli worktree awareness', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('passes dev mode from an explicit dev CLI marker with a custom profile path', async () => {
|
||||
process.env.ORCA_TERMINAL_HANDLE = 'term_sender'
|
||||
process.env.ORCA_USER_DATA_PATH = '/tmp/federation-acceptance-profile'
|
||||
process.env.ORCA_DEV_CLI_INVOCATION = '1'
|
||||
callMock.mockResolvedValueOnce({
|
||||
id: 'req_dispatch',
|
||||
ok: true,
|
||||
result: {
|
||||
dispatch: { id: 'ctx_1', task_id: 'task_1', status: 'dispatched' }
|
||||
},
|
||||
_meta: {
|
||||
runtimeId: 'runtime-1'
|
||||
}
|
||||
})
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {})
|
||||
|
||||
await main(
|
||||
['orchestration', 'dispatch', '--task', 'task_1', '--to', 'term_worker', '--inject'],
|
||||
'/tmp/repo'
|
||||
)
|
||||
|
||||
expect(callMock).toHaveBeenCalledWith(
|
||||
'orchestration.dispatch',
|
||||
expect.objectContaining({ devMode: true })
|
||||
)
|
||||
})
|
||||
|
||||
it('uses the resolved enclosing worktree for terminal consumers', async () => {
|
||||
queueFixtures(
|
||||
callMock,
|
||||
|
||||
@@ -3,6 +3,7 @@ import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { createServer, type Socket } from 'node:net'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY } from '../shared/protocol-version'
|
||||
import { RuntimeClient, RuntimeRpcFailureError } from './runtime-client'
|
||||
import { launchOrcaApp } from './runtime/launch'
|
||||
|
||||
@@ -74,6 +75,87 @@ function findUnusedPid(seed = 200_000): number {
|
||||
// Windows does not support Unix domain sockets in the same way, causing
|
||||
// EACCES errors on listen(), so the suite is skipped on that platform.
|
||||
describe.skipIf(process.platform === 'win32')('RuntimeClient', () => {
|
||||
it('adds an opaque durable request ID only to orchestration mutations', async () => {
|
||||
const userDataPath = mkdtempSync(join(tmpdir(), 'orca-runtime-client-'))
|
||||
const endpoint = join(userDataPath, 'runtime.sock')
|
||||
const requests: Record<string, unknown>[] = []
|
||||
const server = createServer((socket) => {
|
||||
sockets.add(socket)
|
||||
socket.once('close', () => sockets.delete(socket))
|
||||
socket.once('data', (data) => {
|
||||
const request = JSON.parse(String(data).trim()) as Record<string, unknown>
|
||||
requests.push(request)
|
||||
const result =
|
||||
request.method === 'status.get'
|
||||
? { capabilities: [ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY] }
|
||||
: {}
|
||||
socket.write(
|
||||
`${JSON.stringify({
|
||||
id: request.id,
|
||||
ok: true,
|
||||
result,
|
||||
_meta: { runtimeId: 'runtime-1' }
|
||||
})}\n`
|
||||
)
|
||||
})
|
||||
})
|
||||
servers.add(server)
|
||||
await new Promise<void>((resolve) => server.listen(endpoint, resolve))
|
||||
writeMetadata(userDataPath, endpoint)
|
||||
|
||||
const client = new RuntimeClient(userDataPath, 500)
|
||||
await client.call(
|
||||
'orchestration.send',
|
||||
{ subject: 'hello' },
|
||||
{
|
||||
orchestrationRequestId: 'mutation_explicit'
|
||||
}
|
||||
)
|
||||
await client.call('orchestration.taskList', {})
|
||||
|
||||
expect(requests[0]?.method).toBe('status.get')
|
||||
expect(requests[1]?.orchestrationRequestId).toBe('mutation_explicit')
|
||||
expect(requests[1]?.orchestrationContractVersion).toBe(1)
|
||||
expect(requests[2]?.orchestrationRequestId).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects an old local runtime before sending an orchestration mutation', async () => {
|
||||
const userDataPath = mkdtempSync(join(tmpdir(), 'orca-runtime-client-'))
|
||||
const endpoint = join(userDataPath, 'runtime.sock')
|
||||
const requests: Record<string, unknown>[] = []
|
||||
const server = createServer((socket) => {
|
||||
sockets.add(socket)
|
||||
socket.once('close', () => sockets.delete(socket))
|
||||
socket.once('data', (data) => {
|
||||
const request = JSON.parse(String(data).trim()) as Record<string, unknown>
|
||||
requests.push(request)
|
||||
socket.write(
|
||||
`${JSON.stringify({
|
||||
id: request.id,
|
||||
ok: true,
|
||||
result: { capabilities: [] },
|
||||
_meta: { runtimeId: 'runtime-1' }
|
||||
})}\n`
|
||||
)
|
||||
})
|
||||
})
|
||||
servers.add(server)
|
||||
await new Promise<void>((resolve) => server.listen(endpoint, resolve))
|
||||
writeMetadata(userDataPath, endpoint)
|
||||
|
||||
const client = new RuntimeClient(userDataPath, 500)
|
||||
await expect(client.call('orchestration.send', { subject: 'hello' })).rejects.toMatchObject({
|
||||
code: 'orchestration_migration_required',
|
||||
data: {
|
||||
reason: 'runtime_capability_missing',
|
||||
effectsApplied: false,
|
||||
nextCommandArgs: ['skills', 'get', 'orchestration', '--full']
|
||||
}
|
||||
})
|
||||
expect(requests).toHaveLength(1)
|
||||
expect(requests[0]?.method).toBe('status.get')
|
||||
})
|
||||
|
||||
it('returns the full RPC envelope for successful calls', async () => {
|
||||
const userDataPath = mkdtempSync(join(tmpdir(), 'orca-runtime-client-'))
|
||||
const endpoint = join(userDataPath, 'runtime.sock')
|
||||
|
||||
+78
-10
@@ -1,4 +1,10 @@
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import type { CliStatusResult, RuntimeStatus } from '../../shared/runtime-types'
|
||||
import type { RuntimeOrchestrationEnvelope } from '../../shared/runtime-rpc-envelope'
|
||||
import {
|
||||
isOrchestrationMutation,
|
||||
orchestrationMigrationData
|
||||
} from '../../shared/orchestration-rpc-contract'
|
||||
import { parsePairingCode, type PairingOffer } from '../../shared/pairing'
|
||||
import { launchOrcaApp } from './launch'
|
||||
import { getDefaultUserDataPath, readMetadata } from './metadata'
|
||||
@@ -10,6 +16,8 @@ import { markEnvironmentUsed, resolveEnvironmentPairingOffer } from './environme
|
||||
import { describeRuntimeCompatBlock, evaluateRuntimeCompat } from '../../shared/protocol-compat'
|
||||
import {
|
||||
MIN_COMPATIBLE_RUNTIME_SERVER_VERSION,
|
||||
ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY,
|
||||
ORCHESTRATION_CONTRACT_VERSION,
|
||||
RUNTIME_PROTOCOL_VERSION
|
||||
} from '../../shared/protocol-version'
|
||||
|
||||
@@ -27,6 +35,7 @@ export class RuntimeClient {
|
||||
private readonly remotePairing: PairingOffer | null
|
||||
private readonly environmentSelector: string | null
|
||||
private remoteCompatChecked = false
|
||||
private orchestrationContractCheck: Promise<void> | null = null
|
||||
|
||||
// Why: browser commands trigger first-time session init (agent-browser connect +
|
||||
// CDP proxy setup) which can take 15-30s. 60s accommodates cold start without
|
||||
@@ -50,21 +59,39 @@ export class RuntimeClient {
|
||||
async call<TResult>(
|
||||
method: string,
|
||||
params?: unknown,
|
||||
options?: {
|
||||
timeoutMs?: number
|
||||
}
|
||||
options?: { timeoutMs?: number } & RuntimeOrchestrationEnvelope
|
||||
): Promise<RuntimeRpcSuccess<TResult>> {
|
||||
const effectiveTimeoutMs = options?.timeoutMs ?? this.resolveMethodTimeoutMs(method, params)
|
||||
const orchestrationMutation = isOrchestrationMutation(method, params)
|
||||
if (orchestrationMutation) {
|
||||
await this.ensureOrchestrationContractCompatible(effectiveTimeoutMs)
|
||||
}
|
||||
const orchestrationRequestId = orchestrationMutation
|
||||
? (options?.orchestrationRequestId ?? randomUUID())
|
||||
: undefined
|
||||
const envelope = {
|
||||
orchestrationCapability: options?.orchestrationCapability,
|
||||
orchestrationContractVersion: method.startsWith('orchestration.')
|
||||
? ORCHESTRATION_CONTRACT_VERSION
|
||||
: undefined,
|
||||
orchestrationRequestId
|
||||
}
|
||||
if (this.remotePairing) {
|
||||
if (method !== 'status.get') {
|
||||
await this.ensureRemoteRuntimeCompatible(effectiveTimeoutMs)
|
||||
}
|
||||
const response = await sendWebSocketRequest<TResult>(
|
||||
this.remotePairing,
|
||||
method,
|
||||
params,
|
||||
effectiveTimeoutMs
|
||||
)
|
||||
let response
|
||||
try {
|
||||
response = await sendWebSocketRequest<TResult>(
|
||||
this.remotePairing,
|
||||
method,
|
||||
params,
|
||||
effectiveTimeoutMs,
|
||||
envelope
|
||||
)
|
||||
} catch (error) {
|
||||
throw attachMutationRecovery(error, orchestrationRequestId)
|
||||
}
|
||||
if (response.ok === false) {
|
||||
throw new RuntimeRpcFailureError(response)
|
||||
}
|
||||
@@ -76,7 +103,12 @@ export class RuntimeClient {
|
||||
return response
|
||||
}
|
||||
const metadata = readMetadata(this.userDataPath)
|
||||
const response = await sendRequest<TResult>(metadata, method, params, effectiveTimeoutMs)
|
||||
let response
|
||||
try {
|
||||
response = await sendRequest<TResult>(metadata, method, params, effectiveTimeoutMs, envelope)
|
||||
} catch (error) {
|
||||
throw attachMutationRecovery(error, orchestrationRequestId)
|
||||
}
|
||||
if (response.ok === false) {
|
||||
throw new RuntimeRpcFailureError(response)
|
||||
}
|
||||
@@ -165,6 +197,28 @@ export class RuntimeClient {
|
||||
}
|
||||
}
|
||||
|
||||
private async ensureOrchestrationContractCompatible(timeoutMs: number): Promise<void> {
|
||||
if (!this.orchestrationContractCheck) {
|
||||
this.orchestrationContractCheck = this.checkOrchestrationContractCompatibility(timeoutMs)
|
||||
}
|
||||
await this.orchestrationContractCheck
|
||||
}
|
||||
|
||||
private async checkOrchestrationContractCompatibility(timeoutMs: number): Promise<void> {
|
||||
const response = await this.call<RuntimeStatus>('status.get', undefined, { timeoutMs })
|
||||
if (this.remotePairing) {
|
||||
this.assertRemoteRuntimeStatusCompatible(response.result)
|
||||
this.remoteCompatChecked = true
|
||||
}
|
||||
if (!response.result.capabilities?.includes(ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY)) {
|
||||
throw new RuntimeClientError(
|
||||
'orchestration_migration_required',
|
||||
'The connected Orca runtime does not support the current orchestration contract. No effects were applied.',
|
||||
orchestrationMigrationData('runtime_capability_missing')
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
private assertRemoteRuntimeStatusCompatible(status: RuntimeStatus): void {
|
||||
const verdict = evaluateRuntimeCompat({
|
||||
clientProtocolVersion: RUNTIME_PROTOCOL_VERSION,
|
||||
@@ -213,6 +267,20 @@ export class RuntimeClient {
|
||||
}
|
||||
}
|
||||
|
||||
function attachMutationRecovery(error: unknown, requestId: string | undefined): unknown {
|
||||
if (!requestId || !(error instanceof RuntimeClientError)) {
|
||||
return error
|
||||
}
|
||||
return new RuntimeClientError(
|
||||
error.code,
|
||||
`${error.message} Orchestration mutation request ID: ${requestId}.`,
|
||||
{
|
||||
...(error.data && typeof error.data === 'object' ? error.data : {}),
|
||||
orchestrationRequestId: requestId
|
||||
}
|
||||
)
|
||||
}
|
||||
|
||||
function throwDesktopActivationBlocked(): never {
|
||||
throw new RuntimeClientError(
|
||||
'desktop_activation_blocked',
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { createConnection } from 'node:net'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { findTransport, type RuntimeMetadata } from '../../shared/runtime-bootstrap'
|
||||
import type { RuntimeOrchestrationEnvelope } from '../../shared/runtime-rpc-envelope'
|
||||
import { isKeepaliveFrame, RuntimeRpcEnvelopeSchema } from './envelope-schema'
|
||||
import { RuntimeClientError, type RuntimeRpcResponse } from './types'
|
||||
import { MAX_TIMER_DELAY_MS, isSafeTimerDelayMs } from '../../shared/timer-delay'
|
||||
@@ -9,7 +10,8 @@ export async function sendRequest<TResult>(
|
||||
metadata: RuntimeMetadata,
|
||||
method: string,
|
||||
params: unknown,
|
||||
timeoutMs: number
|
||||
timeoutMs: number,
|
||||
envelope?: RuntimeOrchestrationEnvelope
|
||||
): Promise<RuntimeRpcResponse<TResult>> {
|
||||
if (!isSafeTimerDelayMs(timeoutMs)) {
|
||||
throw new RuntimeClientError(
|
||||
@@ -183,7 +185,10 @@ export async function sendRequest<TResult>(
|
||||
id: requestId,
|
||||
authToken: metadata.authToken,
|
||||
method,
|
||||
params
|
||||
params,
|
||||
orchestrationCapability: envelope?.orchestrationCapability,
|
||||
orchestrationContractVersion: envelope?.orchestrationContractVersion,
|
||||
orchestrationRequestId: envelope?.orchestrationRequestId
|
||||
})}\n`
|
||||
)
|
||||
})
|
||||
|
||||
@@ -162,6 +162,29 @@ describe('CLI remote WebSocket transport', () => {
|
||||
message: expect.stringContaining('server is too old')
|
||||
})
|
||||
})
|
||||
|
||||
it('blocks orchestration mutations when a remote runtime lacks the contract capability', async () => {
|
||||
const runtime = await startTestRuntime('runtime-old-orchestration', { capabilities: [] })
|
||||
servers.push(runtime)
|
||||
const client = new RuntimeClient(
|
||||
'/tmp/unused',
|
||||
5_000,
|
||||
encodePairingOffer({
|
||||
v: 2,
|
||||
endpoint: runtime.endpoint,
|
||||
deviceToken: runtime.deviceToken,
|
||||
publicKeyB64: runtime.publicKeyB64
|
||||
})
|
||||
)
|
||||
|
||||
await expect(client.call('orchestration.send', { subject: 'hello' })).rejects.toMatchObject({
|
||||
code: 'orchestration_migration_required',
|
||||
data: {
|
||||
reason: 'runtime_capability_missing',
|
||||
effectsApplied: false
|
||||
}
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
async function startTestRuntime(
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import type { PairingOffer } from '../../shared/pairing'
|
||||
import type { RuntimeOrchestrationEnvelope } from '../../shared/runtime-rpc-envelope'
|
||||
import {
|
||||
RemoteRuntimeClientError,
|
||||
sendRemoteRuntimeRequest
|
||||
@@ -9,10 +10,11 @@ export async function sendWebSocketRequest<TResult>(
|
||||
pairing: PairingOffer,
|
||||
method: string,
|
||||
params: unknown,
|
||||
timeoutMs: number
|
||||
timeoutMs: number,
|
||||
envelope?: RuntimeOrchestrationEnvelope
|
||||
): Promise<RuntimeRpcResponse<TResult>> {
|
||||
try {
|
||||
return await sendRemoteRuntimeRequest<TResult>(pairing, method, params, timeoutMs)
|
||||
return await sendRemoteRuntimeRequest<TResult>(pairing, method, params, timeoutMs, envelope)
|
||||
} catch (error) {
|
||||
if (error instanceof RemoteRuntimeClientError) {
|
||||
throw new RuntimeClientError(error.code, error.message)
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
import { GLOBAL_FLAGS, type CommandSpec } from '../args'
|
||||
|
||||
export const ORCHESTRATION_WORKER_COMMAND_SPECS: CommandSpec[] = [
|
||||
{
|
||||
path: ['orchestration', 'worker-start'],
|
||||
summary: 'Start one supervised worker on the Run home or a connected Orca server',
|
||||
usage:
|
||||
'orca orchestration worker-start --task <task_id> [--on <saved-environment>] [--worktree <current|selector|new-child|new-top-level>] (--agent <agent> | --terminal <handle>) [--name <name>] [--repo <selector>] [--base-branch <ref>] [--display-name <text>] [--comment <text>] [--setup <run|skip|inherit>] [--retry-of <dispatch_id>] [--timeout-ms <n>] [--run <run_id>] [--from <handle>] [--retry-request <id>] [--json]',
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'task',
|
||||
'on',
|
||||
'worktree',
|
||||
'name',
|
||||
'repo',
|
||||
'base-branch',
|
||||
'display-name',
|
||||
'comment',
|
||||
'setup',
|
||||
'agent',
|
||||
'terminal',
|
||||
'retry-of',
|
||||
'timeout-ms',
|
||||
'run',
|
||||
'from',
|
||||
'retry-request'
|
||||
],
|
||||
notes: [
|
||||
'Current and existing worktrees never rerun setup; a fresh agent terminal is created unless --terminal is explicit.',
|
||||
'New worktrees use agent-first creation and default --setup to run. Repository start-immediately runs setup beside the agent; wait-for-setup gates agent readiness and task input.',
|
||||
'Creation flags (--name, --repo, --base-branch, --display-name, --comment, --setup) are rejected for current/existing worktrees. Use exact --repo on the selected server; project/host convenience routing remains on worktree create.',
|
||||
'--on selects only the worker server; the Run and this command remain on the current Orca server.',
|
||||
'Remote current and new-child are invalid; discover an exact remote selector or use new-top-level.',
|
||||
'--retry-of links the replacement attempt but does not inherit placement; repeat the intended --on/worktree and --agent/terminal choices.',
|
||||
'The call exits 0 only for ready. Failed or outcome_unknown exits 1 and JSON includes stage/failedStage, setup, effects, residualResources, and recovery commands when needed.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'worker-show'],
|
||||
summary: 'Inspect one supervised worker Dispatch',
|
||||
usage: 'orca orchestration worker-show --dispatch <dispatch_id> [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'dispatch']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'worker-read'],
|
||||
summary: 'Read bounded output from one supervised worker',
|
||||
usage:
|
||||
'orca orchestration worker-read --dispatch <dispatch_id> [--source <auto|transcript|terminal>] [--cursor <cursor>] [--limit <n>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'dispatch', 'source', 'cursor', 'limit'],
|
||||
notes: [
|
||||
'The default auto source uses an exact hook-reported transcript when available and otherwise returns labeled terminal output.',
|
||||
'A returned cursor is pinned to the exact source; start a fresh read if Orca reports source_changed.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'worker-stop'],
|
||||
summary: 'Fence and stop only one supervised agent terminal',
|
||||
usage:
|
||||
'orca orchestration worker-stop --dispatch <dispatch_id> [--retry-request <id>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'dispatch', 'retry-request'],
|
||||
notes: ['Never deletes the worktree, setup terminal, configured tabs, or unrelated processes.']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'worker-abandon'],
|
||||
summary: 'Fence a worker without claiming its process stopped',
|
||||
usage:
|
||||
'orca orchestration worker-abandon --dispatch <dispatch_id> [--retry-request <id>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'dispatch', 'retry-request'],
|
||||
notes: ['Retains all possibly-live resources and performs no process or filesystem action.']
|
||||
}
|
||||
]
|
||||
+126
-31
@@ -1,15 +1,51 @@
|
||||
import type { CommandSpec } from '../args'
|
||||
import { GLOBAL_FLAGS } from '../args'
|
||||
import { ORCHESTRATION_WORKER_COMMAND_SPECS } from './orchestration-worker-specs'
|
||||
|
||||
export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
{
|
||||
path: ['orchestration', 'run-create'],
|
||||
summary: 'Create and bind a lightweight orchestration Run',
|
||||
usage: 'orca orchestration run-create --objective <text> [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'objective', 'from', 'retry-request'],
|
||||
notes: [
|
||||
'A Run is a namespace and home inbox. It never schedules or places workers.',
|
||||
'--retry-request is only for exact recovery after an unknown mutation result.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'run-use'],
|
||||
summary: 'Bind this coordinator terminal to an existing Run',
|
||||
usage: 'orca orchestration run-use --id <run_id> [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'from', 'retry-request']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'run-current'],
|
||||
summary: 'Show the Run bound to this coordinator terminal',
|
||||
usage: 'orca orchestration run-current [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'from']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'run-list'],
|
||||
summary: 'List lightweight orchestration Runs',
|
||||
usage: 'orca orchestration run-list [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'run-show'],
|
||||
summary: 'Show one lightweight orchestration Run',
|
||||
usage: 'orca orchestration run-show --id <run_id> [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'send'],
|
||||
summary: 'Send an inter-agent message',
|
||||
usage:
|
||||
'orca orchestration send --to <handle> --subject <text> [--from <handle>] [--body <text>] [--type <type>] [--priority <level>] [--thread-id <id>] [--payload <json>] [--task-id <id>] [--dispatch-id <id>] [--files-modified <csv>] [--report-path <path>] [--phase <text>] [--json]',
|
||||
'orca orchestration send --subject <text> [--to <run:id|dispatch:id|legacy_handle>] [--run <run_id>] [--from <handle>] [--body <text>] [--type <type>] [--priority <level>] [--thread-id <id>] [--payload <json>] [--task-id <id>] [--dispatch-id <id>] [--outcome <succeeded|failed>] [--files-modified <csv>] [--report-path <path>] [--phase <text>] [--json]',
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'to',
|
||||
'run',
|
||||
'from',
|
||||
'subject',
|
||||
'body',
|
||||
@@ -19,13 +55,19 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
'payload',
|
||||
'task-id',
|
||||
'dispatch-id',
|
||||
'dispatch-capability',
|
||||
'retry-request',
|
||||
'outcome',
|
||||
'files-modified',
|
||||
'report-path',
|
||||
'phase'
|
||||
],
|
||||
notes: [
|
||||
'On Windows PowerShell, quote group addresses such as --to "@all" or --to "@worktree:<id>".',
|
||||
'worker_done and heartbeat must target a concrete coordinator terminal handle; use status for broadcast updates.',
|
||||
"worker_done and heartbeat are exact-Dispatch signals and cannot target groups; omit --to to use the Dispatch's Run mailbox.",
|
||||
'worker_done requires --outcome succeeded or --outcome failed.',
|
||||
'From an active Dispatch, an omitted recipient defaults to its owning Run mailbox.',
|
||||
'Use --to dispatch:<id> for attempt-specific coordinator guidance; Orca durably relays it to a connected worker server.',
|
||||
'A worker_done with the active task/dispatch IDs completes that task only from the dispatched pane. When stable pane identity is unavailable, the sender handle must exactly match the dispatch assignee; injected preambles include the correct --from value.',
|
||||
'Prefer --task-id/--dispatch-id/etc. over raw --payload JSON in worker commands; PowerShell strips JSON quotes easily.'
|
||||
]
|
||||
@@ -34,8 +76,9 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
path: ['orchestration', 'check'],
|
||||
summary: 'Check messages for a terminal',
|
||||
usage:
|
||||
'orca orchestration check [--terminal <handle>] [--unread | --peek | --all] [--types <type,...>] [--inject] [--wait] [--timeout-ms <n>] [--json]\n' +
|
||||
' --unread (default): return only unread messages and mark them read.\n' +
|
||||
'orca orchestration check [--terminal <handle>] [--run <run_id>] [--ack <delivery_id>] [--unread | --peek | --all] [--types <type,...>] [--format] [--wait] [--timeout-ms <n>] [--json]\n' +
|
||||
" default: return the bound Run's oldest unacknowledged FIFO batch.\n" +
|
||||
' --ack: acknowledge the prior whole batch before checking/waiting.\n' +
|
||||
' --peek: return only unread messages without marking them read.\n' +
|
||||
' --all: return every message for the handle; does not mark read.\n' +
|
||||
' --wait: block until a matching message arrives or --timeout-ms expires.\n' +
|
||||
@@ -46,23 +89,29 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'terminal',
|
||||
'run',
|
||||
'ack',
|
||||
'unread',
|
||||
'peek',
|
||||
'all',
|
||||
'types',
|
||||
'inject',
|
||||
'format',
|
||||
'wait',
|
||||
'timeout-ms'
|
||||
'timeout-ms',
|
||||
'retry-request'
|
||||
],
|
||||
notes: [
|
||||
'On Windows PowerShell, quote comma-separated type filters, e.g. --types "worker_done,escalation".'
|
||||
'On Windows PowerShell, quote comma-separated type filters, e.g. --types "worker_done,escalation".',
|
||||
'--format renders the returned rows as local text only; it never writes to another terminal.',
|
||||
'A bound Run replays the same Delivery until --ack; process every message before acknowledging.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'reply'],
|
||||
summary: 'Reply to a message',
|
||||
usage: 'orca orchestration reply --id <msg_id> --body <text> [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'body', 'from']
|
||||
usage:
|
||||
'orca orchestration reply --id <msg_id> --body <text> [--run <run_id>] [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'body', 'run', 'from', 'retry-request']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'inbox'],
|
||||
@@ -74,30 +123,52 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
path: ['orchestration', 'task-create'],
|
||||
summary: 'Create an orchestration task',
|
||||
usage:
|
||||
'orca orchestration task-create --spec <text> [--task-title <text>] [--display-name <text>] [--deps <json_array>] [--parent <task_id>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'spec', 'task-title', 'display-name', 'deps', 'parent']
|
||||
'orca orchestration task-create --spec <text> [--task-title <text>] [--display-name <text>] [--deps <json_array>] [--parent <task_id>] [--run <run_id>] [--from <handle>] [--json]',
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'spec',
|
||||
'task-title',
|
||||
'display-name',
|
||||
'deps',
|
||||
'parent',
|
||||
'run',
|
||||
'from',
|
||||
'retry-request'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'task-list'],
|
||||
summary: 'List orchestration tasks',
|
||||
usage: 'orca orchestration task-list [--status <status>] [--ready] [--brief] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'status', 'ready', 'brief'],
|
||||
usage:
|
||||
'orca orchestration task-list [--status <status>] [--ready] [--brief] [--run <run_id>] [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'status', 'ready', 'brief', 'run', 'from'],
|
||||
notes: ['--brief collapses whitespace and caps each spec at 160 characters.']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'task-update'],
|
||||
summary: 'Update a task status',
|
||||
usage:
|
||||
'orca orchestration task-update --id <task_id> --status <status> [--result <json>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'status', 'result'],
|
||||
'orca orchestration task-update --id <task_id> --status <status> [--result <json>] [--run <run_id>] [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'status', 'result', 'run', 'from', 'retry-request'],
|
||||
notes: ['Valid --status values: pending, ready, dispatched, completed, failed, blocked.']
|
||||
},
|
||||
...ORCHESTRATION_WORKER_COMMAND_SPECS,
|
||||
{
|
||||
path: ['orchestration', 'dispatch'],
|
||||
summary: 'Dispatch a task to a terminal',
|
||||
usage:
|
||||
'orca orchestration dispatch --task <task_id> --to <handle> [--from <handle>] [--inject] [--dry-run] [--return-preamble] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'task', 'to', 'from', 'inject', 'dry-run', 'return-preamble']
|
||||
'orca orchestration dispatch --task <task_id> --to <handle> [--from <handle>] [--run <run_id>] [--inject] [--dry-run] [--return-preamble] [--json]',
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'task',
|
||||
'to',
|
||||
'from',
|
||||
'run',
|
||||
'inject',
|
||||
'dry-run',
|
||||
'return-preamble',
|
||||
'retry-request'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'dispatch-show'],
|
||||
@@ -110,14 +181,30 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
path: ['orchestration', 'ask'],
|
||||
summary: 'Ask the coordinator a question and block until answered',
|
||||
usage:
|
||||
'orca orchestration ask --to <handle> --question <text> [--options <csv>] [--timeout-ms <n>] [--from <handle>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'to', 'question', 'options', 'timeout-ms', 'from']
|
||||
'orca orchestration ask (--question <text> | --resume <message_id>) [--to <run:id>] [--run <run_id>] [--options <csv>] [--timeout-ms <n>] [--from <handle>] [--json]',
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'to',
|
||||
'run',
|
||||
'question',
|
||||
'resume',
|
||||
'dispatch-capability',
|
||||
'options',
|
||||
'timeout-ms',
|
||||
'from',
|
||||
'retry-request'
|
||||
],
|
||||
notes: [
|
||||
'From an active Dispatch, a new question defaults to its owning Run mailbox.',
|
||||
'Timeout leaves the question pending; resume with the original message ID.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'run'],
|
||||
summary: 'Start the coordinator loop',
|
||||
path: ['orchestration', 'coordinator-start'],
|
||||
aliases: [['orchestration', 'run']],
|
||||
summary: 'Retired: load the current orchestration skill',
|
||||
usage:
|
||||
'orca orchestration run --spec <text> [--from <handle>] [--poll-interval-ms <n>] [--max-concurrent <n>] [--worktree <selector>] [--json]',
|
||||
'orca orchestration coordinator-start --spec <text> [--from <handle>] [--poll-interval-ms <n>] [--max-concurrent <n>] [--worktree <selector>] [--json]',
|
||||
allowedFlags: [
|
||||
...GLOBAL_FLAGS,
|
||||
'spec',
|
||||
@@ -125,26 +212,34 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
'poll-interval-ms',
|
||||
'max-concurrent',
|
||||
'worktree'
|
||||
],
|
||||
notes: [
|
||||
'This command performs no effects and returns the exact `skills get orchestration --full` recovery action.',
|
||||
'Use the lightweight Run, Task, and worker-start primitives described by the current skill.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'run-stop'],
|
||||
summary: 'Stop the active coordinator run',
|
||||
usage: 'orca orchestration run-stop [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS]
|
||||
path: ['orchestration', 'coordinator-stop'],
|
||||
aliases: [['orchestration', 'run-stop']],
|
||||
summary: 'Retired: load the current orchestration skill',
|
||||
usage: 'orca orchestration coordinator-stop [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS],
|
||||
notes: [
|
||||
'This command performs no effects and returns the exact `skills get orchestration --full` recovery action.'
|
||||
]
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'gate-create'],
|
||||
summary: 'Create a decision gate blocking a task',
|
||||
usage:
|
||||
'orca orchestration gate-create --task <task_id> --question <text> [--options <json_array>] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'task', 'question', 'options']
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'task', 'question', 'options', 'retry-request']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'gate-resolve'],
|
||||
summary: 'Resolve a pending decision gate',
|
||||
usage: 'orca orchestration gate-resolve --id <gate_id> --resolution <text> [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'resolution']
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'id', 'resolution', 'retry-request']
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'gate-list'],
|
||||
@@ -154,8 +249,8 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
|
||||
},
|
||||
{
|
||||
path: ['orchestration', 'reset'],
|
||||
summary: 'Reset orchestration state (one scope; bare command resets all)',
|
||||
usage: 'orca orchestration reset [--all | --tasks | --messages] [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'all', 'tasks', 'messages']
|
||||
summary: 'Reset one explicit orchestration state scope',
|
||||
usage: 'orca orchestration reset (--all | --tasks | --messages) [--json]',
|
||||
allowedFlags: [...GLOBAL_FLAGS, 'all', 'tasks', 'messages', 'retry-request']
|
||||
}
|
||||
]
|
||||
|
||||
+30
-1
@@ -41,6 +41,13 @@ import { resolveConsent } from './telemetry/consent'
|
||||
import { triggerStartupNotificationRegistration } from './ipc/notifications'
|
||||
import { OrcaRuntimeService, type RuntimeWorktreeLifecycleEvent } from './runtime/orca-runtime'
|
||||
import { loadAgentSessionClaimSigner } from './runtime/agent-session-claim-identity'
|
||||
import {
|
||||
fingerprintOrchestrationPeer,
|
||||
type OrchestrationEnvironmentTransport
|
||||
} from './runtime/orchestration/environment-transport'
|
||||
import { callRuntimeEnvironment } from './ipc/runtime-environment-transport-routing'
|
||||
import { resolveEnvironment } from '../shared/runtime-environment-store'
|
||||
import { getPreferredPairingOffer } from '../shared/runtime-environments'
|
||||
import { OrcaRuntimeRpcServer } from './runtime/runtime-rpc'
|
||||
import { resolveAdvertisedPairingEndpoint } from './runtime/pairing-endpoint'
|
||||
import { ServeReadinessPublisher } from './server/serve-readiness'
|
||||
@@ -2117,6 +2124,27 @@ app.whenReady().then(async () => {
|
||||
.filter((account) => !activeIds.has(account.id))
|
||||
.map((account) => ({ id: account.id, managedHomePath: account.managedHomePath }))
|
||||
})
|
||||
const orchestrationEnvironmentTransport: OrchestrationEnvironmentTransport = {
|
||||
resolve: (selector) => {
|
||||
const environment = resolveEnvironment(app.getPath('userData'), selector)
|
||||
const pairing = getPreferredPairingOffer(environment)
|
||||
return {
|
||||
environmentId: environment.id,
|
||||
name: environment.name,
|
||||
peerFingerprint: fingerprintOrchestrationPeer(pairing.publicKeyB64)
|
||||
}
|
||||
},
|
||||
call: (selector, method, params, timeoutMs, envelope) =>
|
||||
callRuntimeEnvironment(
|
||||
app.getPath('userData'),
|
||||
selector,
|
||||
method,
|
||||
params,
|
||||
timeoutMs,
|
||||
undefined,
|
||||
envelope
|
||||
)
|
||||
}
|
||||
const runtimeService = new OrcaRuntimeService(store, stats, {
|
||||
agentSessionClaimSigner: loadAgentSessionClaimSigner(
|
||||
getProfileUserDataPath(),
|
||||
@@ -2155,7 +2183,8 @@ app.whenReady().then(async () => {
|
||||
systemCodexHomePath: resolveHostCodexSessionSourceHome(store!.getSettings())
|
||||
}),
|
||||
buildAgentHookPtyEnv: () =>
|
||||
isAgentStatusHooksEnabled(store?.getSettings()) ? agentHookServer.buildPtyEnv() : {}
|
||||
isAgentStatusHooksEnabled(store?.getSettings()) ? agentHookServer.buildPtyEnv() : {},
|
||||
orchestrationEnvironmentTransport
|
||||
})
|
||||
runtime = runtimeService
|
||||
publishProviderSessionChanges(agentHookServer.getProviderSessionIdentities())
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
import { REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY } from '../../shared/protocol-version'
|
||||
import { sendRemoteRuntimeRequest } from '../../shared/remote-runtime-client'
|
||||
import { markEnvironmentUsed } from '../../shared/runtime-environment-store'
|
||||
import type {
|
||||
getPreferredPairingOffer,
|
||||
KnownRuntimeEnvironment
|
||||
} from '../../shared/runtime-environments'
|
||||
import type { RuntimeStatus } from '../../shared/runtime-types'
|
||||
|
||||
const sharedControlSupport = new Map<string, { cacheKey: string; check: Promise<boolean> }>()
|
||||
|
||||
export function resetSharedControlSupport(): void {
|
||||
sharedControlSupport.clear()
|
||||
}
|
||||
|
||||
export function clearSharedControlSupport(environmentId: string): void {
|
||||
sharedControlSupport.delete(environmentId)
|
||||
}
|
||||
|
||||
export async function supportsSharedControl(
|
||||
userDataPath: string,
|
||||
environment: KnownRuntimeEnvironment,
|
||||
pairing: ReturnType<typeof getPreferredPairingOffer>,
|
||||
timeoutMs: number
|
||||
): Promise<boolean> {
|
||||
const cacheKey = getSharedControlSupportCacheKey(environment, pairing)
|
||||
const cached = sharedControlSupport.get(environment.id)
|
||||
if (cached?.cacheKey === cacheKey) {
|
||||
return cached.check
|
||||
}
|
||||
let resolvedCacheKey = cacheKey
|
||||
const check = (async () => {
|
||||
const response = await sendRemoteRuntimeRequest<RuntimeStatus>(
|
||||
pairing,
|
||||
'status.get',
|
||||
undefined,
|
||||
timeoutMs
|
||||
)
|
||||
if (response.ok === true) {
|
||||
markEnvironmentUsed(userDataPath, environment.id, { runtimeId: response._meta.runtimeId })
|
||||
resolvedCacheKey = getSharedControlSupportCacheKey(
|
||||
environment,
|
||||
pairing,
|
||||
response._meta.runtimeId
|
||||
)
|
||||
return (
|
||||
response.result.capabilities?.includes(REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY) === true
|
||||
)
|
||||
}
|
||||
return false
|
||||
})()
|
||||
// Why: support belongs to the saved pairing/runtime identity, not its mutable display name.
|
||||
sharedControlSupport.set(environment.id, { cacheKey, check })
|
||||
try {
|
||||
const supported = await check
|
||||
const cachedAfterCheck = sharedControlSupport.get(environment.id)
|
||||
if (cachedAfterCheck?.check === check && cachedAfterCheck.cacheKey !== resolvedCacheKey) {
|
||||
sharedControlSupport.set(environment.id, { cacheKey: resolvedCacheKey, check })
|
||||
}
|
||||
return supported
|
||||
} catch (error) {
|
||||
if (sharedControlSupport.get(environment.id)?.check === check) {
|
||||
sharedControlSupport.delete(environment.id)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
function getSharedControlSupportCacheKey(
|
||||
environment: KnownRuntimeEnvironment,
|
||||
pairing: ReturnType<typeof getPreferredPairingOffer>,
|
||||
runtimeId = environment.runtimeId
|
||||
): string {
|
||||
return [
|
||||
runtimeId ?? 'unknown-runtime',
|
||||
pairing.endpoint,
|
||||
pairing.deviceToken,
|
||||
pairing.publicKeyB64
|
||||
].join('\0')
|
||||
}
|
||||
@@ -1,11 +1,10 @@
|
||||
import {
|
||||
getPreferredPairingOffer,
|
||||
type KnownRuntimeEnvironment
|
||||
} from '../../shared/runtime-environments'
|
||||
import { getPreferredPairingOffer } from '../../shared/runtime-environments'
|
||||
import { resolveEnvironment, markEnvironmentUsed } from '../../shared/runtime-environment-store'
|
||||
import type { RuntimeRpcResponse } from '../../shared/runtime-rpc-envelope'
|
||||
import type {
|
||||
RuntimeOrchestrationEnvelope,
|
||||
RuntimeRpcResponse
|
||||
} from '../../shared/runtime-rpc-envelope'
|
||||
import type { RuntimeStatus } from '../../shared/runtime-types'
|
||||
import { REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY } from '../../shared/protocol-version'
|
||||
import {
|
||||
sendRemoteRuntimeRequest,
|
||||
subscribeRemoteRuntimeRequest,
|
||||
@@ -22,14 +21,15 @@ import {
|
||||
import { attachRemoteControlDiagnostics } from './runtime-environment-status-diagnostics'
|
||||
import { runtimeEnvironmentRevisionFailure } from './runtime-environment-revision-guard'
|
||||
import { withTailscaleHintForResponse } from './runtime-environment-tailscale-response'
|
||||
import {
|
||||
clearSharedControlSupport,
|
||||
resetSharedControlSupport,
|
||||
supportsSharedControl
|
||||
} from './runtime-environment-shared-control-support'
|
||||
|
||||
const DEFAULT_REMOTE_RUNTIME_TIMEOUT_MS = 15_000
|
||||
const sharedControlSupport = new Map<string, { cacheKey: string; check: Promise<boolean> }>()
|
||||
|
||||
export const resetSharedControlSupport = (): void => sharedControlSupport.clear()
|
||||
|
||||
export const clearSharedControlSupport = (environmentId: string): void =>
|
||||
void sharedControlSupport.delete(environmentId)
|
||||
export { clearSharedControlSupport, resetSharedControlSupport }
|
||||
|
||||
export async function getRuntimeEnvironmentStatus(
|
||||
userDataPath: string,
|
||||
@@ -81,7 +81,8 @@ export async function callRuntimeEnvironment(
|
||||
method: string,
|
||||
params: unknown,
|
||||
timeoutMs?: number,
|
||||
expectedEnvironmentPairingRevision?: number
|
||||
expectedEnvironmentPairingRevision?: number,
|
||||
envelope?: RuntimeOrchestrationEnvelope
|
||||
): Promise<RuntimeRpcResponse<unknown>> {
|
||||
const environment = resolveEnvironment(userDataPath, selector)
|
||||
// Why: connection failures reject (they don't resolve as ok:false), so the
|
||||
@@ -104,6 +105,17 @@ export async function callRuntimeEnvironment(
|
||||
const pairing = getPreferredPairingOffer(currentEnvironment)
|
||||
endpoint = pairing.endpoint
|
||||
const effectiveTimeoutMs = timeoutMs ?? DEFAULT_REMOTE_RUNTIME_TIMEOUT_MS
|
||||
if (envelope) {
|
||||
const response = await sendRemoteRuntimeRequest(
|
||||
pairing,
|
||||
method,
|
||||
params,
|
||||
effectiveTimeoutMs,
|
||||
envelope
|
||||
)
|
||||
markEnvironmentUsedFromResponse(userDataPath, currentEnvironment.id, response)
|
||||
return response
|
||||
}
|
||||
if (shouldUseCachedRequestConnection(method)) {
|
||||
const response = await sendRemoteRuntimeConnectionRequest(
|
||||
currentEnvironment.id,
|
||||
@@ -257,66 +269,3 @@ function shouldUseSharedControlSubscription(method: string): boolean {
|
||||
method === 'files.watch'
|
||||
)
|
||||
}
|
||||
|
||||
async function supportsSharedControl(
|
||||
userDataPath: string,
|
||||
environment: KnownRuntimeEnvironment,
|
||||
pairing: ReturnType<typeof getPreferredPairingOffer>,
|
||||
timeoutMs: number
|
||||
): Promise<boolean> {
|
||||
const cacheKey = getSharedControlSupportCacheKey(environment, pairing)
|
||||
const cached = sharedControlSupport.get(environment.id)
|
||||
if (cached?.cacheKey === cacheKey) {
|
||||
return cached.check
|
||||
}
|
||||
let resolvedCacheKey = cacheKey
|
||||
const check = (async () => {
|
||||
const response = await sendRemoteRuntimeRequest<RuntimeStatus>(
|
||||
pairing,
|
||||
'status.get',
|
||||
undefined,
|
||||
timeoutMs
|
||||
)
|
||||
if (response.ok === true) {
|
||||
markEnvironmentUsed(userDataPath, environment.id, { runtimeId: response._meta.runtimeId })
|
||||
resolvedCacheKey = getSharedControlSupportCacheKey(
|
||||
environment,
|
||||
pairing,
|
||||
response._meta.runtimeId
|
||||
)
|
||||
return (
|
||||
response.result.capabilities?.includes(REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY) === true
|
||||
)
|
||||
}
|
||||
return false
|
||||
})()
|
||||
// Why: the same saved host can be re-paired or point at a different runtime
|
||||
// binary over time; capability support belongs to that pairing/runtime identity.
|
||||
sharedControlSupport.set(environment.id, { cacheKey, check })
|
||||
try {
|
||||
const supported = await check
|
||||
const cachedAfterCheck = sharedControlSupport.get(environment.id)
|
||||
if (cachedAfterCheck?.check === check && cachedAfterCheck.cacheKey !== resolvedCacheKey) {
|
||||
sharedControlSupport.set(environment.id, { cacheKey: resolvedCacheKey, check })
|
||||
}
|
||||
return supported
|
||||
} catch (error) {
|
||||
if (sharedControlSupport.get(environment.id)?.check === check) {
|
||||
sharedControlSupport.delete(environment.id)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
function getSharedControlSupportCacheKey(
|
||||
environment: KnownRuntimeEnvironment,
|
||||
pairing: ReturnType<typeof getPreferredPairingOffer>,
|
||||
runtimeId = environment.runtimeId
|
||||
): string {
|
||||
return [
|
||||
runtimeId ?? 'unknown-runtime',
|
||||
pairing.endpoint,
|
||||
pairing.deviceToken,
|
||||
pairing.publicKeyB64
|
||||
].join('\0')
|
||||
}
|
||||
|
||||
@@ -49,6 +49,8 @@ export async function readNativeChatTranscriptTailFile(
|
||||
consumedTo: number
|
||||
hasMore: boolean
|
||||
beforeOffset: number
|
||||
malformedRecordCount?: number
|
||||
oversizedRecordCount?: number
|
||||
}> {
|
||||
const end = Math.min((await stat(filePath)).size, endOffset ?? Number.MAX_SAFE_INTEGER)
|
||||
if (end === 0) {
|
||||
@@ -59,6 +61,9 @@ export async function readNativeChatTranscriptTailFile(
|
||||
let lineBytes = 0
|
||||
let lineOversized = false
|
||||
let lifecycle: NativeChatTurnLifecycle | undefined
|
||||
let malformedRecordCount = 0
|
||||
let oversizedRecordCount = 0
|
||||
let ignoreNextMalformedRecord = false
|
||||
try {
|
||||
const consumedTo = includeTrailingLine ? end : await findLastCompleteLineEnd(handle, end)
|
||||
if (consumedTo === 0) {
|
||||
@@ -67,6 +72,7 @@ export async function readNativeChatTranscriptTailFile(
|
||||
const newestFirst: { message: NativeChatMessage; offset: number }[] = []
|
||||
const finalByte = Buffer.allocUnsafe(1)
|
||||
await handle.read(finalByte, 0, 1, consumedTo - 1)
|
||||
ignoreNextMalformedRecord = finalByte[0] !== 0x0a
|
||||
let cursor = consumedTo - (finalByte[0] === 0x0a ? 1 : 0)
|
||||
while (cursor > 0 && newestFirst.length <= limit) {
|
||||
const start = Math.max(0, cursor - TAIL_CHUNK_BYTES)
|
||||
@@ -101,7 +107,9 @@ export async function readNativeChatTranscriptTailFile(
|
||||
...(lifecycle ? { lifecycle } : {}),
|
||||
consumedTo,
|
||||
hasMore: limit > 0 && chronological.length > limit,
|
||||
beforeOffset: selected[0]?.offset ?? end
|
||||
beforeOffset: selected[0]?.offset ?? end,
|
||||
...(malformedRecordCount > 0 ? { malformedRecordCount } : {}),
|
||||
...(oversizedRecordCount > 0 ? { oversizedRecordCount } : {})
|
||||
}
|
||||
} finally {
|
||||
await handle.close()
|
||||
@@ -115,6 +123,7 @@ export async function readNativeChatTranscriptTailFile(
|
||||
if (lineBytes > MAX_NATIVE_CHAT_TRANSCRIPT_RECORD_BYTES) {
|
||||
lineParts.length = 0
|
||||
lineOversized = true
|
||||
oversizedRecordCount++
|
||||
return
|
||||
}
|
||||
lineParts.push(part)
|
||||
@@ -137,6 +146,17 @@ export async function readNativeChatTranscriptTailFile(
|
||||
if (!line) {
|
||||
return
|
||||
}
|
||||
try {
|
||||
JSON.parse(line)
|
||||
} catch {
|
||||
if (ignoreNextMalformedRecord) {
|
||||
ignoreNextMalformedRecord = false
|
||||
return
|
||||
}
|
||||
malformedRecordCount++
|
||||
return
|
||||
}
|
||||
ignoreNextMalformedRecord = false
|
||||
const fallbackId = transcriptFallbackId(filePath, lineOffset)
|
||||
// Why: scan the same bounded JSONL window for provider-authored lifecycle
|
||||
// records so reconnect snapshots can replay completion without guessing
|
||||
|
||||
@@ -1012,6 +1012,7 @@ class InMemoryOrchestrationMessages {
|
||||
this.sequence += 1
|
||||
const row: MessageRow = {
|
||||
id: `msg_${this.sequence}`,
|
||||
run_id: 'run_test',
|
||||
from_handle: msg.from,
|
||||
to_handle: msg.to,
|
||||
subject: msg.subject,
|
||||
@@ -12468,6 +12469,102 @@ describe('OrcaRuntimeService', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it('observes setup command completion without waiting for its interactive shell to exit', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
spawn: vi.fn().mockResolvedValue({ id: 'pty-setup' }),
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null
|
||||
})
|
||||
runtime.attachWindow(1)
|
||||
runtime.syncWindowGraph(1, { tabs: [], leaves: [] })
|
||||
const { handle } = await runtime.createTerminal(`path:${TEST_WORKTREE_PATH}`)
|
||||
;(
|
||||
runtime as unknown as { setupCompletionTokenByPtyId: Map<string, string> }
|
||||
).setupCompletionTokenByPtyId.set('pty-setup', 'token-live')
|
||||
|
||||
const waiting = runtime.waitForSetupTerminalCompletion(handle)
|
||||
runtime.onPtyData(
|
||||
'pty-setup',
|
||||
'setup failed\r\n__ORCA_SETUP_COMPLETE__:token-live:17\r\nPS>',
|
||||
100
|
||||
)
|
||||
|
||||
await expect(waiting).resolves.toEqual({ exitCode: 17 })
|
||||
await expect(runtime.readTerminal(handle)).resolves.toMatchObject({ status: 'running' })
|
||||
})
|
||||
|
||||
it('replays fast setup completion emitted before its observer is registered', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
spawn: vi.fn().mockResolvedValue({ id: 'pty-fast-setup' }),
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null
|
||||
})
|
||||
runtime.attachWindow(1)
|
||||
runtime.syncWindowGraph(1, { tabs: [], leaves: [] })
|
||||
const { handle } = await runtime.createTerminal(`path:${TEST_WORKTREE_PATH}`)
|
||||
;(
|
||||
runtime as unknown as { setupCompletionTokenByPtyId: Map<string, string> }
|
||||
).setupCompletionTokenByPtyId.set('pty-fast-setup', 'token-fast')
|
||||
runtime.onPtyData(
|
||||
'pty-fast-setup',
|
||||
'__ORCA_SETUP_COMPLETE__:wrong:9\r\n__ORCA_SETUP_COMPLETE__:token-fast:0\r\n$',
|
||||
100
|
||||
)
|
||||
|
||||
await expect(runtime.waitForSetupTerminalCompletion(handle)).resolves.toEqual({ exitCode: 0 })
|
||||
})
|
||||
|
||||
it('falls back to setup terminal exit when no completion signal is available', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
spawn: vi.fn().mockResolvedValue({ id: 'pty-legacy-setup' }),
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null
|
||||
})
|
||||
runtime.attachWindow(1)
|
||||
runtime.syncWindowGraph(1, { tabs: [], leaves: [] })
|
||||
const { handle } = await runtime.createTerminal(`path:${TEST_WORKTREE_PATH}`)
|
||||
|
||||
const waiting = runtime.waitForSetupTerminalCompletion(handle)
|
||||
runtime.onPtyExit('pty-legacy-setup', 9)
|
||||
|
||||
await expect(waiting).resolves.toEqual({ exitCode: 9 })
|
||||
})
|
||||
|
||||
it('keeps observing after an uncertain setup terminal status', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
spawn: vi.fn().mockResolvedValue({ id: 'pty-uncertain-setup' }),
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null
|
||||
})
|
||||
runtime.attachWindow(1)
|
||||
runtime.syncWindowGraph(1, { tabs: [], leaves: [] })
|
||||
const { handle } = await runtime.createTerminal(`path:${TEST_WORKTREE_PATH}`)
|
||||
;(
|
||||
runtime as unknown as { setupCompletionTokenByPtyId: Map<string, string> }
|
||||
).setupCompletionTokenByPtyId.set('pty-uncertain-setup', 'token-uncertain')
|
||||
vi.spyOn(runtime, 'waitForTerminal').mockResolvedValue({
|
||||
handle,
|
||||
condition: 'exit',
|
||||
satisfied: false,
|
||||
status: 'unknown',
|
||||
exitCode: null
|
||||
})
|
||||
|
||||
const waiting = runtime.waitForSetupTerminalCompletion(handle)
|
||||
await Promise.resolve()
|
||||
runtime.onPtyData('pty-uncertain-setup', '__ORCA_SETUP_COMPLETE__:token-uncertain:0\r\n', 100)
|
||||
|
||||
await expect(waiting).resolves.toEqual({ exitCode: 0 })
|
||||
})
|
||||
|
||||
it('drops retained PTY transcript memory when a background terminal exits', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
@@ -17290,6 +17387,83 @@ describe('OrcaRuntimeService', () => {
|
||||
expect(writes).toEqual(['still writable'])
|
||||
})
|
||||
|
||||
it('preserves runtime-created PTY process identity after graph unavailable', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
spawn: vi.fn().mockResolvedValue({ id: 'pty-bg' }),
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null
|
||||
})
|
||||
runtime.attachWindow(1)
|
||||
runtime.syncWindowGraph(1, { tabs: [], leaves: [] })
|
||||
const { handle } = await runtime.createTerminal(`path:${TEST_WORKTREE_PATH}`)
|
||||
const incarnation = runtime.getTerminalProcessIncarnation(handle)
|
||||
|
||||
runtime.markGraphUnavailable(1)
|
||||
|
||||
expect(runtime.getTerminalProcessIncarnation(handle)).toBe(incarnation)
|
||||
})
|
||||
|
||||
it('preserves PTY process identity while a renderer surface detaches and reattaches', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
spawn: vi.fn().mockResolvedValue({
|
||||
id: 'pty-bg',
|
||||
incarnationId: 'incarnation-bg'
|
||||
}),
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null
|
||||
})
|
||||
runtime.attachWindow(1)
|
||||
runtime.syncWindowGraph(1, { tabs: [], leaves: [] })
|
||||
const created = await runtime.createTerminal(`path:${TEST_WORKTREE_PATH}`)
|
||||
const [tabId, leafId] = created.paneKey?.split(':') ?? []
|
||||
if (!tabId || !leafId) {
|
||||
throw new Error('expected stable pane identity')
|
||||
}
|
||||
const syncSurface = (ptyId: string | null): void => {
|
||||
runtime.syncWindowGraph(1, {
|
||||
tabs: [
|
||||
{
|
||||
tabId,
|
||||
worktreeId: TEST_WORKTREE_ID,
|
||||
title: 'Codex',
|
||||
activeLeafId: leafId,
|
||||
layout: null
|
||||
}
|
||||
],
|
||||
leaves: [
|
||||
{
|
||||
tabId,
|
||||
worktreeId: TEST_WORKTREE_ID,
|
||||
leafId,
|
||||
paneRuntimeId: 1,
|
||||
ptyId,
|
||||
paneTitle: 'Codex'
|
||||
}
|
||||
]
|
||||
})
|
||||
}
|
||||
|
||||
syncSurface('pty-bg')
|
||||
await runtime.listTerminals()
|
||||
const before = runtime.getTerminalProcessIncarnation(created.handle)
|
||||
syncSurface(null)
|
||||
syncSurface('pty-bg')
|
||||
await runtime.listTerminals()
|
||||
|
||||
expect(runtime.getTerminalProcessIncarnation(created.handle)).toBe(before)
|
||||
|
||||
runtime.registerPty('pty-bg', TEST_WORKTREE_ID, null, {
|
||||
tabId,
|
||||
leafId,
|
||||
incarnationId: 'incarnation-replacement'
|
||||
})
|
||||
expect(runtime.getTerminalProcessIncarnation(created.handle)).not.toBe(before)
|
||||
})
|
||||
|
||||
it('recognizes runtime-created PTY handles with agent launch titles', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
runtime.setPtyController({
|
||||
@@ -27108,7 +27282,7 @@ describe('OrcaRuntimeService', () => {
|
||||
|
||||
const waitPromise = runtime.waitForMessage('term_abc', { timeoutMs: 5000 })
|
||||
runtime.notifyMessageArrived('term_abc')
|
||||
await waitPromise
|
||||
await expect(waitPromise).resolves.toBe('notified')
|
||||
})
|
||||
|
||||
it('does not resolve type-filtered message waiters for unrelated message types', async () => {
|
||||
@@ -27149,13 +27323,38 @@ describe('OrcaRuntimeService', () => {
|
||||
})
|
||||
|
||||
it('resolves message waiters on timeout when no message arrives', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
vi.useFakeTimers()
|
||||
try {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
const wait = runtime.waitForMessage('term_abc', { timeoutMs: 100 })
|
||||
|
||||
const start = Date.now()
|
||||
await runtime.waitForMessage('term_abc', { timeoutMs: 100 })
|
||||
const elapsed = Date.now() - start
|
||||
expect(elapsed).toBeGreaterThanOrEqual(90)
|
||||
expect(elapsed).toBeLessThan(500)
|
||||
await vi.advanceTimersByTimeAsync(99)
|
||||
let settled = false
|
||||
void wait.then(() => {
|
||||
settled = true
|
||||
})
|
||||
await Promise.resolve()
|
||||
expect(settled).toBe(false)
|
||||
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await expect(wait).resolves.toBe('timed_out')
|
||||
} finally {
|
||||
vi.useRealTimers()
|
||||
}
|
||||
})
|
||||
|
||||
it('allows only one exclusive mailbox waiter and supports explicit cancellation', async () => {
|
||||
const runtime = new OrcaRuntimeService(store)
|
||||
const first = runtime.waitForMessage('run:run_1', {
|
||||
timeoutMs: 5000,
|
||||
exclusive: true
|
||||
})
|
||||
|
||||
await expect(
|
||||
runtime.waitForMessage('run:run_1', { timeoutMs: 5000, exclusive: true })
|
||||
).resolves.toBe('waiter_exists')
|
||||
runtime.cancelMessageWaiters('run:run_1')
|
||||
await expect(first).resolves.toBe('cancelled')
|
||||
})
|
||||
|
||||
it('rejects leaf PTY waits when the request signal aborts', async () => {
|
||||
@@ -32908,11 +33107,13 @@ describe('OrcaRuntimeService', () => {
|
||||
}
|
||||
])
|
||||
|
||||
await runtime.createManagedWorktree({
|
||||
const result = await runtime.createManagedWorktree({
|
||||
repoSelector: 'id:repo-1',
|
||||
name: 'runtime-headless-parallel',
|
||||
setupDecision: 'run',
|
||||
startup: { command: 'claude' }
|
||||
startup: { command: 'claude' },
|
||||
observeSetupCompletion: true,
|
||||
awaitTerminalProvisioning: true
|
||||
})
|
||||
|
||||
// Why: setup now spawns fire-and-forget on a later tick; wait for both PTYs.
|
||||
@@ -32920,8 +33121,14 @@ describe('OrcaRuntimeService', () => {
|
||||
expect(spawn).toHaveBeenNthCalledWith(1, expect.objectContaining({ command: 'claude' }))
|
||||
expect(spawn).toHaveBeenNthCalledWith(
|
||||
2,
|
||||
expect.objectContaining({ command: 'bash /tmp/repo/.git/orca/setup-runner.sh' })
|
||||
expect.objectContaining({
|
||||
command: expect.stringContaining('__ORCA_SETUP_COMPLETE__:')
|
||||
})
|
||||
)
|
||||
expect(result.setupReceipt).toMatchObject({
|
||||
state: 'running',
|
||||
terminalHandle: expect.stringMatching(/^term_/)
|
||||
})
|
||||
})
|
||||
|
||||
it('creates the first terminal for CLI-created worktrees without activating them', async () => {
|
||||
@@ -33133,10 +33340,12 @@ describe('OrcaRuntimeService', () => {
|
||||
const result = await runtime.createManagedWorktree({
|
||||
repoSelector: 'id:repo-1',
|
||||
name: 'runtime-cli-setup-skip',
|
||||
setupDecision: 'skip'
|
||||
setupDecision: 'skip',
|
||||
awaitTerminalProvisioning: true
|
||||
})
|
||||
|
||||
expect(result.warning).toBeUndefined()
|
||||
expect(result.setupReceipt).toMatchObject({ requested: 'skip', state: 'skipped' })
|
||||
expect(createSetupRunnerScript).not.toHaveBeenCalled()
|
||||
expect(spawn).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
@@ -33694,7 +33903,8 @@ describe('OrcaRuntimeService', () => {
|
||||
name: 'runtime-startup-setup-split',
|
||||
startupDraft: 'https://github.com/stablyai/orca/issues/123',
|
||||
setupDecision: 'run',
|
||||
activate: true
|
||||
activate: true,
|
||||
awaitTerminalProvisioning: true
|
||||
})
|
||||
|
||||
await vi.waitFor(() => expect(spawn).toHaveBeenCalledTimes(2))
|
||||
@@ -33732,6 +33942,10 @@ describe('OrcaRuntimeService', () => {
|
||||
const mainEnv = (spawn.mock.calls[0]![0] as { env?: Record<string, string> }).env ?? {}
|
||||
const setupEnv = (spawn.mock.calls[1]![0] as { env?: Record<string, string> }).env ?? {}
|
||||
expect(result.setup).toBeUndefined()
|
||||
expect(result.setupReceipt).toMatchObject({
|
||||
state: 'running',
|
||||
terminalHandle: expect.stringMatching(/^term_/)
|
||||
})
|
||||
expect(mainEnv.ORCA_TAB_ID).toBeDefined()
|
||||
expect(mainEnv.ORCA_PANE_KEY).toBeDefined()
|
||||
expect(setupEnv.ORCA_TAB_ID).toBe(mainEnv.ORCA_TAB_ID)
|
||||
|
||||
@@ -96,7 +96,23 @@ import { mkdir, readFile, readdir, rm, stat } from 'node:fs/promises'
|
||||
import { resolveWorktreeCreateBase } from '../worktree-create-base'
|
||||
import { resolveWorktreeAddBaseRef } from '../../shared/worktree-base-ref'
|
||||
import { OrchestrationDb } from './orchestration/db'
|
||||
import { OrchestrationError } from './orchestration/orchestration-error'
|
||||
import {
|
||||
buildObservedSetupCommand,
|
||||
createSetupCompletionScanner
|
||||
} from './orchestration/setup-completion-signal'
|
||||
import type { RuntimeOrchestrationEnvelope } from '../../shared/runtime-rpc-envelope'
|
||||
import {
|
||||
isOrchestrationMutation,
|
||||
orchestrationMigrationData
|
||||
} from '../../shared/orchestration-rpc-contract'
|
||||
import type {
|
||||
OrchestrationEnvironmentTransport,
|
||||
OrchestrationWorkerServer
|
||||
} from './orchestration/environment-transport'
|
||||
import { syncFederatedDispatch } from './orchestration/federation-sync'
|
||||
import { formatMessagesForInjection } from './orchestration/formatter'
|
||||
import { selectExactWorkerProviderSession } from './orchestration/worker-provider-session'
|
||||
import type {
|
||||
Automation,
|
||||
AutomationCreateInput,
|
||||
@@ -183,6 +199,7 @@ import type {
|
||||
AgentProviderSessionMetadata,
|
||||
SleepingAgentLaunchConfig
|
||||
} from '../../shared/agent-session-resume'
|
||||
import type { ExactWorkerProviderSession } from '../../shared/orchestration-worker-output'
|
||||
import type { RuntimeClientEvent } from '../../shared/runtime-client-events'
|
||||
import { toRuntimeActivateWorktreeEvent } from '../../shared/runtime-client-events'
|
||||
import {
|
||||
@@ -332,6 +349,8 @@ import {
|
||||
BROWSER_HEADLESS_RUNTIME_CAPABILITY,
|
||||
BROWSER_CERTIFICATE_TRUST_RUNTIME_CAPABILITY,
|
||||
MIN_COMPATIBLE_RUNTIME_CLIENT_VERSION,
|
||||
ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY,
|
||||
ORCHESTRATION_CONTRACT_VERSION,
|
||||
REMOTE_RUNTIME_SHARED_CONTROL_CAPABILITY,
|
||||
RUNTIME_CAPABILITIES,
|
||||
RUNTIME_PROTOCOL_VERSION,
|
||||
@@ -1698,11 +1717,13 @@ type TerminalWaiter = {
|
||||
type MessageWaiter = {
|
||||
handle: string
|
||||
typeFilter: string[] | undefined
|
||||
resolve: (result: void) => void
|
||||
resolve: (result: MessageWaitResult) => void
|
||||
timeout: NodeJS.Timeout | null
|
||||
abortCleanup: (() => void) | null
|
||||
}
|
||||
|
||||
export type MessageWaitResult = 'notified' | 'timed_out' | 'cancelled' | 'waiter_exists'
|
||||
|
||||
function omitUndefinedProperties<T extends Record<string, unknown>>(value: T): Partial<T> {
|
||||
return Object.fromEntries(
|
||||
Object.entries(value).filter(([, entry]) => entry !== undefined)
|
||||
@@ -2438,6 +2459,10 @@ export class OrcaRuntimeService {
|
||||
private readonly runtimeId = randomUUID()
|
||||
private readonly startedAt = Date.now()
|
||||
private readonly store: RuntimeStore | null
|
||||
private readonly orchestrationEnvironmentTransport: OrchestrationEnvironmentTransport | null
|
||||
private readonly orchestrationFederationTimers = new Map<string, ReturnType<typeof setInterval>>()
|
||||
private readonly orchestrationFederationSyncs = new Map<string, Promise<void>>()
|
||||
private readonly orchestrationFederationWarnings = new Set<string>()
|
||||
private rendererGraphEpoch = 0
|
||||
private graphStatus: RuntimeGraphStatus = 'unavailable'
|
||||
private authoritativeWindowId: number | null = null
|
||||
@@ -2563,6 +2588,7 @@ export class OrcaRuntimeService {
|
||||
// Why: startup draft paste can subscribe after the agent already emitted its
|
||||
// ready marker. Keep a bounded raw buffer so fast startup output is replayed.
|
||||
private recentPtyOutputById = new Map<string, RecentPtyOutputBuffer>()
|
||||
private setupCompletionTokenByPtyId = new Map<string, string>()
|
||||
// Why: mobile clients need to know when the desktop restores a terminal
|
||||
// from mobile-fit so they can update their UI. These listeners are
|
||||
// invoked from resizeForClient and onClientDisconnected/onPtyExit.
|
||||
@@ -2969,6 +2995,7 @@ export class OrcaRuntimeService {
|
||||
buildAgentHookPtyEnv?: () => Record<string, string>
|
||||
getDesktopWindowStatus?: () => RuntimeDesktopWindowStatus
|
||||
agentSessionClaimSigner?: AgentSessionClaimSigner
|
||||
orchestrationEnvironmentTransport?: OrchestrationEnvironmentTransport
|
||||
}
|
||||
) {
|
||||
this.store = store
|
||||
@@ -2980,6 +3007,7 @@ export class OrcaRuntimeService {
|
||||
this.clientSessionTabSelections.setPersistListener((state) => {
|
||||
this.store?.setMobileClientTabSelections?.(state)
|
||||
})
|
||||
this.orchestrationEnvironmentTransport = deps?.orchestrationEnvironmentTransport ?? null
|
||||
if (stats) {
|
||||
this.stats = stats
|
||||
this.agentDetector = new AgentDetector(stats)
|
||||
@@ -3440,6 +3468,136 @@ export class OrcaRuntimeService {
|
||||
return this.runtimeId
|
||||
}
|
||||
|
||||
resolveOrchestrationWorkerServer(selector: string): OrchestrationWorkerServer {
|
||||
if (!this.orchestrationEnvironmentTransport) {
|
||||
throw new OrchestrationError(
|
||||
'server_required',
|
||||
'Connected-server orchestration is unavailable in this runtime.'
|
||||
)
|
||||
}
|
||||
return this.orchestrationEnvironmentTransport.resolve(selector)
|
||||
}
|
||||
|
||||
async callOrchestrationWorkerServer(
|
||||
selector: string,
|
||||
method: string,
|
||||
params: unknown,
|
||||
timeoutMs?: number,
|
||||
envelope?: RuntimeOrchestrationEnvelope
|
||||
): Promise<unknown> {
|
||||
if (!this.orchestrationEnvironmentTransport) {
|
||||
throw new OrchestrationError(
|
||||
'server_required',
|
||||
'Connected-server orchestration is unavailable in this runtime.'
|
||||
)
|
||||
}
|
||||
if (isOrchestrationMutation(method, params)) {
|
||||
const statusResponse = await this.orchestrationEnvironmentTransport.call(
|
||||
selector,
|
||||
'status.get',
|
||||
undefined,
|
||||
timeoutMs
|
||||
)
|
||||
if (statusResponse.ok === false) {
|
||||
throw new OrchestrationError(
|
||||
statusResponse.error.code,
|
||||
statusResponse.error.message,
|
||||
statusResponse.error.data
|
||||
)
|
||||
}
|
||||
const status = statusResponse.result as RuntimeStatus
|
||||
if (!status.capabilities?.includes(ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY)) {
|
||||
throw new OrchestrationError(
|
||||
'orchestration_migration_required',
|
||||
'The connected worker server does not support the current orchestration contract. No effects were applied.',
|
||||
orchestrationMigrationData('runtime_capability_missing')
|
||||
)
|
||||
}
|
||||
}
|
||||
const response = await this.orchestrationEnvironmentTransport.call(
|
||||
selector,
|
||||
method,
|
||||
params,
|
||||
timeoutMs,
|
||||
method.startsWith('orchestration.')
|
||||
? { ...envelope, orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION }
|
||||
: envelope
|
||||
)
|
||||
if (response.ok === false) {
|
||||
throw new OrchestrationError(response.error.code, response.error.message, response.error.data)
|
||||
}
|
||||
return response.result
|
||||
}
|
||||
|
||||
async syncOrchestrationFederation(runId?: string): Promise<void> {
|
||||
if (!this.orchestrationEnvironmentTransport) {
|
||||
return
|
||||
}
|
||||
const dispatches = this.getOrchestrationDb().listActiveFederatedDispatches(runId)
|
||||
await Promise.allSettled(
|
||||
dispatches.map((dispatch) => this.syncOrchestrationFederatedDispatch(dispatch.dispatch_id))
|
||||
)
|
||||
}
|
||||
|
||||
private syncOrchestrationFederatedDispatch(dispatchId: string): Promise<void> {
|
||||
const current = this.orchestrationFederationSyncs.get(dispatchId)
|
||||
if (current) {
|
||||
return current
|
||||
}
|
||||
const sync = syncFederatedDispatch(this, dispatchId)
|
||||
.then(() => {
|
||||
this.orchestrationFederationWarnings.delete(dispatchId)
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
if (!this.orchestrationFederationWarnings.has(dispatchId)) {
|
||||
console.warn(`[orchestration] Federation sync failed for ${dispatchId}:`, error)
|
||||
this.orchestrationFederationWarnings.add(dispatchId)
|
||||
}
|
||||
throw error
|
||||
})
|
||||
.finally(() => {
|
||||
this.orchestrationFederationSyncs.delete(dispatchId)
|
||||
})
|
||||
this.orchestrationFederationSyncs.set(dispatchId, sync)
|
||||
return sync
|
||||
}
|
||||
|
||||
ensureOrchestrationFederationRelay(runId?: string): void {
|
||||
if (!this.orchestrationEnvironmentTransport) {
|
||||
return
|
||||
}
|
||||
for (const dispatch of this.getOrchestrationDb().listActiveFederatedDispatches(runId)) {
|
||||
if (this.orchestrationFederationTimers.has(dispatch.dispatch_id)) {
|
||||
continue
|
||||
}
|
||||
const tick = () => {
|
||||
const worker = this.getOrchestrationDb().getWorkerDispatch(dispatch.dispatch_id)
|
||||
if (!worker || !['starting', 'ready', 'stopping'].includes(worker.state)) {
|
||||
const activeTimer = this.orchestrationFederationTimers.get(dispatch.dispatch_id)
|
||||
if (activeTimer) {
|
||||
clearInterval(activeTimer)
|
||||
}
|
||||
this.orchestrationFederationTimers.delete(dispatch.dispatch_id)
|
||||
this.orchestrationFederationWarnings.delete(dispatch.dispatch_id)
|
||||
return
|
||||
}
|
||||
void this.syncOrchestrationFederatedDispatch(dispatch.dispatch_id).catch(() => undefined)
|
||||
}
|
||||
const timer = setInterval(tick, 1_000)
|
||||
timer.unref?.()
|
||||
this.orchestrationFederationTimers.set(dispatch.dispatch_id, timer)
|
||||
tick()
|
||||
}
|
||||
}
|
||||
|
||||
stopOrchestrationFederationRelay(): void {
|
||||
for (const timer of this.orchestrationFederationTimers.values()) {
|
||||
clearInterval(timer)
|
||||
}
|
||||
this.orchestrationFederationTimers.clear()
|
||||
this.orchestrationFederationWarnings.clear()
|
||||
}
|
||||
|
||||
getStartedAt(): number {
|
||||
return this.startedAt
|
||||
}
|
||||
@@ -10901,6 +11059,7 @@ export class OrcaRuntimeService {
|
||||
this.resizeListeners.delete(ptyId)
|
||||
this.lastRendererSizes.delete(ptyId)
|
||||
this.recentPtyOutputById.delete(ptyId)
|
||||
this.setupCompletionTokenByPtyId.delete(ptyId)
|
||||
this.clearWaitBlockedCheckState(ptyId)
|
||||
this.recentPtyPathCandidatesById.delete(ptyId)
|
||||
this.ptyOutputSequenceById.delete(ptyId)
|
||||
@@ -13568,6 +13727,64 @@ export class OrcaRuntimeService {
|
||||
}
|
||||
}
|
||||
|
||||
getTerminalProcessIncarnation(handle: string): string | null {
|
||||
const live = this.getLivePtyForHandle(handle)
|
||||
const record = live?.record ?? this.handles.get(handle)
|
||||
if (!record?.ptyId) {
|
||||
return null
|
||||
}
|
||||
const incarnationId = live?.pty.incarnationId ?? this.ptysById.get(record.ptyId)?.incarnationId
|
||||
if (incarnationId) {
|
||||
return `${record.ptyId}:${incarnationId}`
|
||||
}
|
||||
// Why: legacy providers may omit process incarnation; retain the prior restart-degraded fence.
|
||||
return `${this.runtimeId}:${record.ptyId}:${record.ptyGeneration}`
|
||||
}
|
||||
|
||||
getExactWorkerProviderSession(
|
||||
handle: string,
|
||||
observedAfter: number
|
||||
): ExactWorkerProviderSession | null {
|
||||
const paneKey = this.getTerminalPaneKey(handle)
|
||||
const processIncarnation = this.getTerminalProcessIncarnation(handle)
|
||||
if (!paneKey || !processIncarnation) {
|
||||
return null
|
||||
}
|
||||
let connectionId: string | null | undefined
|
||||
let launchToken: string | null | undefined
|
||||
try {
|
||||
const ptyId = this.getTerminalAgentStatusPtyId(handle)
|
||||
const pty = this.ptysById.get(ptyId)
|
||||
connectionId = pty?.connectionId ?? null
|
||||
launchToken = pty?.launchToken ?? null
|
||||
} catch {
|
||||
// Exact worker validation rejects this in production; test/legacy providers may not expose PTY metadata.
|
||||
connectionId = undefined
|
||||
launchToken = undefined
|
||||
}
|
||||
return selectExactWorkerProviderSession({
|
||||
paneKey,
|
||||
processIncarnation,
|
||||
connectionId,
|
||||
launchToken,
|
||||
observedAfter,
|
||||
statuses: this.getAgentStatusSnapshotFn?.() ?? []
|
||||
})
|
||||
}
|
||||
|
||||
validateOrchestrationAgentLauncher(agent: TuiAgent): void {
|
||||
const settings = this.store?.getSettings()
|
||||
if (!settings) {
|
||||
throw new Error('runtime_unavailable')
|
||||
}
|
||||
if (!isTuiAgentEnabled(agent, settings.disabledTuiAgents)) {
|
||||
throw new OrchestrationError(
|
||||
'agent_unconfigured',
|
||||
`Agent launcher ${agent} is disabled or unavailable.`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
resolveTerminalPane(paneKey: string, expectedWorktreeId?: string): RuntimeTerminalResolvePane {
|
||||
// Why: the renderer context menu only knows the stable pane key; main owns
|
||||
// the runtime terminal handle that agents and CLI commands can address.
|
||||
@@ -14485,6 +14702,62 @@ export class OrcaRuntimeService {
|
||||
})
|
||||
}
|
||||
|
||||
async waitForSetupTerminalCompletion(handle: string): Promise<{ exitCode: number | null }> {
|
||||
const ptyId = this.getLivePtyForHandle(handle)?.pty.ptyId
|
||||
if (!ptyId) {
|
||||
throw new Error('terminal_handle_stale')
|
||||
}
|
||||
const completionToken = this.setupCompletionTokenByPtyId.get(ptyId)
|
||||
const exitAbort = new AbortController()
|
||||
return await new Promise<{ exitCode: number | null }>((resolve, reject) => {
|
||||
let settled = false
|
||||
let unsubscribe: (() => void) | null = null
|
||||
const cleanup = (): void => {
|
||||
unsubscribe?.()
|
||||
exitAbort.abort()
|
||||
}
|
||||
const finish = (exitCode: number | null): void => {
|
||||
if (settled) {
|
||||
return
|
||||
}
|
||||
settled = true
|
||||
cleanup()
|
||||
this.setupCompletionTokenByPtyId.delete(ptyId)
|
||||
resolve({ exitCode })
|
||||
}
|
||||
const fail = (error: unknown): void => {
|
||||
if (settled) {
|
||||
return
|
||||
}
|
||||
settled = true
|
||||
cleanup()
|
||||
reject(error)
|
||||
}
|
||||
const scanner = completionToken ? createSetupCompletionScanner(completionToken, finish) : null
|
||||
|
||||
if (scanner) {
|
||||
unsubscribe = this.subscribeToTerminalData(ptyId, scanner.scan)
|
||||
}
|
||||
// Why: setup can finish before the observer is registered on fast local worktrees.
|
||||
const replay = this.recentPtyOutputById.get(ptyId)?.read()
|
||||
if (scanner && replay) {
|
||||
scanner.scan(replay)
|
||||
}
|
||||
if (!settled) {
|
||||
void this.waitForTerminal(handle, {
|
||||
condition: 'exit',
|
||||
signal: exitAbort.signal
|
||||
})
|
||||
.then((wait) => {
|
||||
if (wait.satisfied && wait.condition === 'exit' && wait.status === 'exited') {
|
||||
finish(wait.exitCode)
|
||||
}
|
||||
})
|
||||
.catch(fail)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
async getWorktreePs(limit = DEFAULT_WORKTREE_PS_LIMIT): Promise<{
|
||||
worktrees: RuntimeWorktreePsSummary[]
|
||||
totalCount: number
|
||||
@@ -18172,16 +18445,18 @@ export class OrcaRuntimeService {
|
||||
primaryTerminalHandle?: string | null
|
||||
hasStartupTerminal: boolean
|
||||
setupCommandPlatform: 'windows' | 'posix'
|
||||
observeSetupCompletion?: boolean
|
||||
// Why: when the agent startup is sequenced to wait for setup
|
||||
// (waitForAgentStartup), the startup PTY runs a wrapper that already embeds
|
||||
// the setup command. Pass that wrapped command through so the Setup tab runs
|
||||
// the same script the agent is waiting on instead of a bare runner.
|
||||
wrappedSetupCommand?: string
|
||||
}): Promise<{ setupSpawned: boolean }> {
|
||||
}): Promise<{ setupSpawned: boolean; setupTerminalHandle: string | null }> {
|
||||
if (!this.ptyController?.spawn) {
|
||||
return { setupSpawned: false }
|
||||
return { setupSpawned: false, setupTerminalHandle: null }
|
||||
}
|
||||
let setupSpawned = false
|
||||
let setupTerminalHandle: string | null = null
|
||||
try {
|
||||
const defaultTabHandles = await this.createDefaultTabTerminals(
|
||||
args.worktreeSelector,
|
||||
@@ -18200,25 +18475,41 @@ export class OrcaRuntimeService {
|
||||
primaryTerminalHandle = terminal.handle
|
||||
}
|
||||
if (args.setup) {
|
||||
const completionToken =
|
||||
args.observeSetupCompletion && !args.wrappedSetupCommand ? randomUUID() : null
|
||||
const observedCommand = completionToken
|
||||
? buildObservedSetupCommand(
|
||||
args.setup.runnerScriptPath,
|
||||
args.setupCommandPlatform,
|
||||
completionToken
|
||||
)
|
||||
: null
|
||||
const setupCommand =
|
||||
args.wrappedSetupCommand ??
|
||||
observedCommand?.command ??
|
||||
buildSetupRunnerCommand(args.setup.runnerScriptPath, args.setupCommandPlatform)
|
||||
const setupEnv = { ...args.setup.envVars, ...observedCommand?.env }
|
||||
const shouldSplitSetup =
|
||||
primaryTerminalHandle &&
|
||||
(setupLaunchMode === 'split-vertical' || setupLaunchMode === 'split-horizontal')
|
||||
await (shouldSplitSetup
|
||||
const setupTerminal = await (shouldSplitSetup
|
||||
? this.splitTerminal(primaryTerminalHandle!, {
|
||||
direction: setupLaunchMode === 'split-horizontal' ? 'horizontal' : 'vertical',
|
||||
command: setupCommand,
|
||||
env: args.setup.envVars,
|
||||
env: setupEnv,
|
||||
activate: false
|
||||
})
|
||||
: this.createTerminal(args.worktreeSelector, {
|
||||
title: 'Setup',
|
||||
command: setupCommand,
|
||||
env: args.setup.envVars
|
||||
env: setupEnv
|
||||
}))
|
||||
setupTerminalHandle = setupTerminal.handle
|
||||
setupSpawned = true
|
||||
const ptyId = this.getLivePtyForHandle(setupTerminal.handle)?.pty.ptyId
|
||||
if (completionToken && ptyId) {
|
||||
this.setupCompletionTokenByPtyId.set(ptyId, completionToken)
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err)
|
||||
@@ -18226,7 +18517,7 @@ export class OrcaRuntimeService {
|
||||
`[worktree-create] Failed to create setup/default terminals for ${args.worktreePath}: ${message}`
|
||||
)
|
||||
}
|
||||
return { setupSpawned }
|
||||
return { setupSpawned, setupTerminalHandle }
|
||||
}
|
||||
|
||||
private async waitForStartupFollowupReady(
|
||||
@@ -18360,6 +18651,8 @@ export class OrcaRuntimeService {
|
||||
runHooks?: boolean
|
||||
activate?: boolean
|
||||
setupDecision?: 'run' | 'skip' | 'inherit'
|
||||
awaitTerminalProvisioning?: boolean
|
||||
observeSetupCompletion?: boolean
|
||||
createdWithAgent?: TuiAgent
|
||||
startupAgent?: TuiAgent
|
||||
startupPrompt?: string
|
||||
@@ -19160,6 +19453,7 @@ export class OrcaRuntimeService {
|
||||
// RPC return value must omit setup so the client does not spawn it a second
|
||||
// time. Mirrors the wait-for-agent setup contract from #6298.
|
||||
let didSpawnSetup = false
|
||||
let setupTerminalHandle: string | null = null
|
||||
let startupTerminalHandle: string | null = null
|
||||
let startupTerminalTabId: string | null = null
|
||||
let startupTerminalPaneKey: string | null = null
|
||||
@@ -19250,11 +19544,13 @@ export class OrcaRuntimeService {
|
||||
? 'windows'
|
||||
: 'posix'
|
||||
: 'posix',
|
||||
observeSetupCompletion: args.observeSetupCompletion,
|
||||
// Why: carry the wait-for-agent wrapped setup command (#6298) so the
|
||||
// Setup tab runs the same script the sequenced agent waits on.
|
||||
...(wrappedSetupCommandStr ? { wrappedSetupCommand: wrappedSetupCommandStr } : {})
|
||||
})
|
||||
didSpawnSetup = provisioned.setupSpawned
|
||||
setupTerminalHandle = provisioned.setupTerminalHandle
|
||||
}
|
||||
// Why: when runtime spawned setup, omit it from activation. When setup
|
||||
// spawn failed, fall through with the wrapped command so renderer
|
||||
@@ -19290,7 +19586,7 @@ export class OrcaRuntimeService {
|
||||
} else if (this.ptyController?.spawn && (setup || defaultTabs || didSpawnStartup)) {
|
||||
// Why: inactive terminal materialization matches normal worktree creation,
|
||||
// but setup/default tab failures must not gate automation dispatch.
|
||||
void this.provisionManagedWorktreeTerminals({
|
||||
const provisioning = this.provisionManagedWorktreeTerminals({
|
||||
worktreeSelector: `id:${worktree.id}`,
|
||||
worktreeId: worktree.id,
|
||||
worktreePath,
|
||||
@@ -19303,12 +19599,20 @@ export class OrcaRuntimeService {
|
||||
? 'windows'
|
||||
: 'posix'
|
||||
: 'posix',
|
||||
observeSetupCompletion: args.observeSetupCompletion,
|
||||
...(wrappedSetupCommandStr ? { wrappedSetupCommand: wrappedSetupCommandStr } : {})
|
||||
})
|
||||
// Why: runtime owns setup spawning here, so the RPC result must omit setup
|
||||
// to keep the headless/mobile caller from launching it a second time.
|
||||
if (setup) {
|
||||
didSpawnSetup = true
|
||||
if (args.awaitTerminalProvisioning) {
|
||||
const provisioned = await provisioning
|
||||
didSpawnSetup = provisioned.setupSpawned
|
||||
setupTerminalHandle = provisioned.setupTerminalHandle
|
||||
} else {
|
||||
void provisioning
|
||||
if (setup) {
|
||||
didSpawnSetup = true
|
||||
}
|
||||
}
|
||||
} else if (this.ptyController?.spawn) {
|
||||
try {
|
||||
@@ -19348,6 +19652,25 @@ export class OrcaRuntimeService {
|
||||
},
|
||||
...(lineageInput ? { lineage, workspaceLineage, warnings: lineageWarnings } : {}),
|
||||
...(returnedSetup ? { setup: returnedSetup } : {}),
|
||||
...(args.awaitTerminalProvisioning
|
||||
? {
|
||||
setupReceipt: {
|
||||
requested: effectiveDecision,
|
||||
hookFound: Boolean(hooks?.scripts.setup),
|
||||
startupPolicy: setup?.waitForAgentStartup
|
||||
? ('wait-for-setup' as const)
|
||||
: ('start-immediately' as const),
|
||||
state: !hooks?.scripts.setup
|
||||
? ('not_configured' as const)
|
||||
: effectiveDecision === 'skip' || !shouldRunSetup
|
||||
? ('skipped' as const)
|
||||
: didSpawnSetup
|
||||
? ('running' as const)
|
||||
: ('spawn_failed' as const),
|
||||
...(setupTerminalHandle ? { terminalHandle: setupTerminalHandle } : {})
|
||||
}
|
||||
}
|
||||
: {}),
|
||||
...(defaultTabs ? { defaultTabs } : {}),
|
||||
...(warning ? { warning } : {}),
|
||||
...(addResult.localBaseRefRefresh
|
||||
@@ -19397,6 +19720,8 @@ export class OrcaRuntimeService {
|
||||
runHooks?: boolean
|
||||
activate?: boolean
|
||||
setupDecision?: 'run' | 'skip' | 'inherit'
|
||||
awaitTerminalProvisioning?: boolean
|
||||
observeSetupCompletion?: boolean
|
||||
createdWithAgent?: TuiAgent
|
||||
pendingFirstAgentMessageRename?: boolean
|
||||
automationProvenance?: AutomationWorkspaceProvenance
|
||||
@@ -19474,6 +19799,7 @@ export class OrcaRuntimeService {
|
||||
// Why: same no-double-spawn contract as the local path — once runtime
|
||||
// provisions setup, omit it from activation and the RPC result.
|
||||
let didSpawnSetup = false
|
||||
let setupTerminalHandle: string | null = null
|
||||
let startupTerminalHandle: string | null = null
|
||||
let startupTerminalTabId: string | null = null
|
||||
let startupTerminalPaneKey: string | null = null
|
||||
@@ -19557,11 +19883,13 @@ export class OrcaRuntimeService {
|
||||
? 'windows'
|
||||
: 'posix'
|
||||
: 'posix',
|
||||
observeSetupCompletion: args.observeSetupCompletion,
|
||||
// Why: carry the wait-for-agent wrapped setup command (#6298) so the
|
||||
// remote Setup tab runs the same script the sequenced agent waits on.
|
||||
...(wrappedSetupCommandStr ? { wrappedSetupCommand: wrappedSetupCommandStr } : {})
|
||||
})
|
||||
didSpawnSetup = provisioned.setupSpawned
|
||||
setupTerminalHandle = provisioned.setupTerminalHandle
|
||||
}
|
||||
// Why: omit setup from activation when runtime spawned it; on spawn
|
||||
// failure fall through with the wrapped command so renderer retries.
|
||||
@@ -19602,7 +19930,7 @@ export class OrcaRuntimeService {
|
||||
) {
|
||||
// Why: inactive terminal materialization matches normal worktree creation,
|
||||
// but setup/default tab failures must not gate automation dispatch.
|
||||
void this.provisionManagedWorktreeTerminals({
|
||||
const provisioning = this.provisionManagedWorktreeTerminals({
|
||||
worktreeSelector: `path:${result.worktree.path}`,
|
||||
worktreeId: result.worktree.id,
|
||||
worktreePath: result.worktree.path,
|
||||
@@ -19615,12 +19943,20 @@ export class OrcaRuntimeService {
|
||||
? 'windows'
|
||||
: 'posix'
|
||||
: 'posix',
|
||||
observeSetupCompletion: args.observeSetupCompletion,
|
||||
...(wrappedSetupCommandStr ? { wrappedSetupCommand: wrappedSetupCommandStr } : {})
|
||||
})
|
||||
// Why: runtime owns setup spawning here, so omit setup from the RPC result
|
||||
// to keep the headless/mobile caller from launching it a second time.
|
||||
if (result.setup) {
|
||||
didSpawnSetup = true
|
||||
if (args.awaitTerminalProvisioning) {
|
||||
const provisioned = await provisioning
|
||||
didSpawnSetup = provisioned.setupSpawned
|
||||
setupTerminalHandle = provisioned.setupTerminalHandle
|
||||
} else {
|
||||
void provisioning
|
||||
if (result.setup) {
|
||||
didSpawnSetup = true
|
||||
}
|
||||
}
|
||||
} else if (!shouldActivate && this.ptyController?.spawn) {
|
||||
try {
|
||||
@@ -19665,7 +20001,27 @@ export class OrcaRuntimeService {
|
||||
}
|
||||
: resultForRenderer
|
||||
|
||||
return warning ? { ...resultWithStartupTerminal, warning } : resultWithStartupTerminal
|
||||
const requestedSetupDecision = args.runHooks ? 'run' : (args.setupDecision ?? 'inherit')
|
||||
const setupReceipt = {
|
||||
requested: requestedSetupDecision,
|
||||
hookFound: Boolean(result.setup),
|
||||
startupPolicy: result.setup?.waitForAgentStartup
|
||||
? ('wait-for-setup' as const)
|
||||
: ('start-immediately' as const),
|
||||
state:
|
||||
requestedSetupDecision === 'skip'
|
||||
? ('skipped' as const)
|
||||
: !result.setup
|
||||
? ('not_configured' as const)
|
||||
: didSpawnSetup
|
||||
? ('running' as const)
|
||||
: ('spawn_failed' as const),
|
||||
...(setupTerminalHandle ? { terminalHandle: setupTerminalHandle } : {})
|
||||
}
|
||||
const resultWithSetupReceipt = args.awaitTerminalProvisioning
|
||||
? { ...resultWithStartupTerminal, setupReceipt }
|
||||
: resultWithStartupTerminal
|
||||
return warning ? { ...resultWithSetupReceipt, warning } : resultWithSetupReceipt
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -25598,6 +25954,7 @@ export class OrcaRuntimeService {
|
||||
this.advancePtyLifecycleGeneration(ptyId)
|
||||
this.ptysById.delete(ptyId)
|
||||
this.recentPtyOutputById.delete(ptyId)
|
||||
this.setupCompletionTokenByPtyId.delete(ptyId)
|
||||
this.clearWaitBlockedCheckState(ptyId)
|
||||
this.recentPtyPathCandidatesById.delete(ptyId)
|
||||
this.ptyOutputSequenceById.delete(ptyId)
|
||||
@@ -27057,15 +27414,25 @@ export class OrcaRuntimeService {
|
||||
if (messageType && waiter.typeFilter && !waiter.typeFilter.includes(messageType)) {
|
||||
continue
|
||||
}
|
||||
this.resolveMessageWaiter(waiter)
|
||||
this.resolveMessageWaiter(waiter, 'notified')
|
||||
}
|
||||
}
|
||||
|
||||
waitForMessage(
|
||||
handle: string,
|
||||
options?: { typeFilter?: string[]; timeoutMs?: number; signal?: AbortSignal }
|
||||
): Promise<void> {
|
||||
options?: {
|
||||
typeFilter?: string[]
|
||||
timeoutMs?: number
|
||||
signal?: AbortSignal
|
||||
exclusive?: boolean
|
||||
}
|
||||
): Promise<MessageWaitResult> {
|
||||
return new Promise((resolve) => {
|
||||
const currentWaiters = this.messageWaitersByHandle.get(handle)
|
||||
if (options?.exclusive && currentWaiters && currentWaiters.size > 0) {
|
||||
resolve('waiter_exists')
|
||||
return
|
||||
}
|
||||
const timeoutMs = options?.timeoutMs ?? MESSAGE_WAIT_DEFAULT_TIMEOUT_MS
|
||||
|
||||
const waiter: MessageWaiter = {
|
||||
@@ -27080,11 +27447,11 @@ export class OrcaRuntimeService {
|
||||
const signal = options?.signal
|
||||
const onAbort = (): void => {
|
||||
this.removeMessageWaiter(waiter)
|
||||
resolve()
|
||||
resolve('cancelled')
|
||||
}
|
||||
if (signal) {
|
||||
if (signal.aborted) {
|
||||
resolve()
|
||||
resolve('cancelled')
|
||||
return
|
||||
}
|
||||
waiter.abortCleanup = () => signal.removeEventListener('abort', onAbort)
|
||||
@@ -27093,7 +27460,7 @@ export class OrcaRuntimeService {
|
||||
|
||||
waiter.timeout = setTimeout(() => {
|
||||
this.removeMessageWaiter(waiter)
|
||||
resolve()
|
||||
resolve('timed_out')
|
||||
}, timeoutMs)
|
||||
|
||||
let waiters = this.messageWaitersByHandle.get(handle)
|
||||
@@ -27105,9 +27472,19 @@ export class OrcaRuntimeService {
|
||||
})
|
||||
}
|
||||
|
||||
private resolveMessageWaiter(waiter: MessageWaiter): void {
|
||||
cancelMessageWaiters(handle: string): void {
|
||||
const waiters = this.messageWaitersByHandle.get(handle)
|
||||
if (!waiters) {
|
||||
return
|
||||
}
|
||||
for (const waiter of [...waiters]) {
|
||||
this.resolveMessageWaiter(waiter, 'cancelled')
|
||||
}
|
||||
}
|
||||
|
||||
private resolveMessageWaiter(waiter: MessageWaiter, result: MessageWaitResult): void {
|
||||
this.removeMessageWaiter(waiter)
|
||||
waiter.resolve()
|
||||
waiter.resolve(result)
|
||||
}
|
||||
|
||||
private removeMessageWaiter(waiter: MessageWaiter): void {
|
||||
|
||||
@@ -19,7 +19,7 @@ import { spawn } from 'node:child_process'
|
||||
import { existsSync, mkdtempSync, rmSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { OrcaRuntimeService } from './orca-runtime'
|
||||
import { OrchestrationDb } from './orchestration/db'
|
||||
import { OrcaRuntimeRpcServer } from './runtime-rpc'
|
||||
@@ -198,6 +198,15 @@ describeIfBuilt('orca orchestration reset subprocess', () => {
|
||||
const runtime = new OrcaRuntimeService()
|
||||
const db = new OrchestrationDb(':memory:')
|
||||
runtime.setOrchestrationDb(db)
|
||||
const coordinatorPaneKey = 'tab_cli:11111111-1111-4111-8111-111111111111'
|
||||
vi.spyOn(runtime, 'getTerminalPaneKey').mockImplementation((handle) =>
|
||||
handle === 'term_cli' ? coordinatorPaneKey : null
|
||||
)
|
||||
db.createRun({
|
||||
objective: 'CLI reset subprocess fixture',
|
||||
coordinatorHandle: 'term_cli',
|
||||
coordinatorPaneKey
|
||||
})
|
||||
const server = new OrcaRuntimeRpcServer({ runtime, userDataPath })
|
||||
await server.start()
|
||||
|
||||
@@ -218,6 +227,8 @@ describeIfBuilt('orca orchestration reset subprocess', () => {
|
||||
'task-create',
|
||||
'--spec',
|
||||
'throwaway task',
|
||||
'--from',
|
||||
'term_cli',
|
||||
'--json'
|
||||
])
|
||||
expect(create.exitCode, create.stderr).toBe(0)
|
||||
@@ -253,18 +264,41 @@ describeIfBuilt('orca orchestration reset subprocess', () => {
|
||||
expect(db.getInbox()).toHaveLength(1)
|
||||
expect(db.listTasks()).toHaveLength(0)
|
||||
|
||||
// Why: task resets intentionally remove Runs, so recreate the caller's
|
||||
// normal binding before exercising task creation again.
|
||||
db.createRun({
|
||||
objective: 'CLI reset subprocess fixture after reset',
|
||||
coordinatorHandle: 'term_cli',
|
||||
coordinatorPaneKey
|
||||
})
|
||||
const recreate = await runBuiltCli(userDataPath, [
|
||||
'orchestration',
|
||||
'task-create',
|
||||
'--spec',
|
||||
'throwaway task after partial reset',
|
||||
'--from',
|
||||
'term_cli',
|
||||
'--json'
|
||||
])
|
||||
expect(recreate.exitCode, recreate.stderr).toBe(0)
|
||||
expect(db.getInbox()).toHaveLength(1)
|
||||
expect(db.listTasks()).toHaveLength(1)
|
||||
|
||||
const resetAll = await runBuiltCli(userDataPath, ['orchestration', 'reset', '--json'])
|
||||
const bareReset = await runBuiltCli(userDataPath, ['orchestration', 'reset', '--json'])
|
||||
expect(bareReset.exitCode).toBe(1)
|
||||
expect(JSON.parse(bareReset.stdout)).toMatchObject({
|
||||
ok: false,
|
||||
error: { code: 'invalid_argument' }
|
||||
})
|
||||
expect(db.getInbox()).toHaveLength(1)
|
||||
expect(db.listTasks()).toHaveLength(1)
|
||||
|
||||
const resetAll = await runBuiltCli(userDataPath, [
|
||||
'orchestration',
|
||||
'reset',
|
||||
'--all',
|
||||
'--json'
|
||||
])
|
||||
expect(resetAll.exitCode, resetAll.stderr).toBe(0)
|
||||
expect(JSON.parse(resetAll.stdout)).toMatchObject({ ok: true, result: { reset: 'all' } })
|
||||
expect(db.getInbox()).toHaveLength(0)
|
||||
|
||||
@@ -10,7 +10,7 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
|
||||
=== CLI COMMANDS ===
|
||||
|
||||
# Report task completion (REQUIRED when done — even on failure).
|
||||
# Report the terminal task outcome (REQUIRED exactly once).
|
||||
#
|
||||
# RULE: --body must be a 3-sentence executive summary (what you did,
|
||||
# what you found, what's left). Never send an empty body; the coordinator
|
||||
@@ -18,14 +18,15 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
# If you produced a long-form artifact, include its path as
|
||||
# payload.reportPath so the coordinator can find it without a file search.
|
||||
#
|
||||
# RULE: send worker_done exactly once. Failure is still a worker_done
|
||||
# with subject like "Failed: <reason>" — never silently exit.
|
||||
# RULE: send worker_done exactly once. Use --outcome succeeded when the
|
||||
# requested work is done, or replace it with --outcome failed when it is not.
|
||||
# Never encode failure only in prose and never silently exit.
|
||||
# Include BOTH taskId and dispatchId in the payload so a late completion
|
||||
# from a failed retry cannot complete the current dispatch.
|
||||
orca orchestration send --to term_COORD --from term_WORKER \\
|
||||
orca orchestration send --from term_WORKER \\
|
||||
--type worker_done --subject "<short status>" \\
|
||||
--body "<3-sentence summary: what you did, what you found, what's left>" \\
|
||||
--task-id task_SNAP --dispatch-id ctx_SNAP \\
|
||||
--task-id task_SNAP --dispatch-id ctx_SNAP --outcome succeeded \\
|
||||
--files-modified "path/a,path/b" \\
|
||||
--report-path "<optional: path to the full artifact>"
|
||||
|
||||
@@ -39,7 +40,7 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
# attributes the heartbeat to the specific dispatch context, not just
|
||||
# the task, so a straggler heartbeat from a previously-failed dispatch
|
||||
# cannot mask a hung retry.
|
||||
orca orchestration send --to term_COORD --from term_WORKER \\
|
||||
orca orchestration send --from term_WORKER \\
|
||||
--type heartbeat --subject "alive" \\
|
||||
--task-id task_SNAP --dispatch-id ctx_SNAP \\
|
||||
--phase "<short: investigating|implementing|reviewing|waiting>"
|
||||
@@ -52,18 +53,18 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
# coordinator cannot see and cannot answer — your session will hang forever
|
||||
# waiting on a human. Every interactive question goes through \`ask\` below.
|
||||
#
|
||||
# The \`ask\` verb is a thin wrapper: it sends a decision_gate message and
|
||||
# blocks on \`check --wait\` until the coordinator replies, then prints the
|
||||
# reply body. Use it anywhere you would otherwise have reached for
|
||||
# AskUserQuestion.
|
||||
orca orchestration ask --to term_COORD --from term_WORKER \\
|
||||
# The \`ask\` verb durably records a question in this Dispatch's Run and
|
||||
# blocks until the coordinator replies, then prints the reply body. If the
|
||||
# call times out or disconnects, resume with the returned message ID instead
|
||||
# of creating a duplicate question.
|
||||
orca orchestration ask --from term_WORKER \\
|
||||
--question "<your question>" \\
|
||||
--options "<optional,comma,separated>" \\
|
||||
--timeout-ms 600000
|
||||
|
||||
# Escalate a blocker or failure (pre-completion, when you need the
|
||||
# coordinator to do something before you can continue):
|
||||
orca orchestration send --to term_COORD --from term_WORKER \\
|
||||
orca orchestration send --from term_WORKER \\
|
||||
--type escalation --subject "Blocked: <reason>" \\
|
||||
--body "<details>" \\
|
||||
--task-id task_SNAP
|
||||
|
||||
@@ -98,6 +98,7 @@ function insertWorkerDone(
|
||||
payload: JSON.stringify({
|
||||
taskId: params.taskId,
|
||||
dispatchId,
|
||||
outcome: 'succeeded',
|
||||
...(params.filesModified ? { filesModified: params.filesModified } : {})
|
||||
}),
|
||||
senderPaneKey:
|
||||
@@ -192,7 +193,7 @@ describe('Coordinator', () => {
|
||||
to: 'coord',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id })
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' })
|
||||
})
|
||||
|
||||
reconcileLifecycleMessage(db, msg)
|
||||
@@ -214,7 +215,11 @@ describe('Coordinator', () => {
|
||||
|
||||
const task = db.createTask({ spec: 'duplicate completion' })
|
||||
const dispatch = db.createDispatchContext(task.id, 'term_a')
|
||||
const payload = JSON.stringify({ taskId: task.id, dispatchId: dispatch.id })
|
||||
const payload = JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
const first = db.insertMessage({
|
||||
from: 'term_a',
|
||||
to: 'coord',
|
||||
@@ -569,7 +574,11 @@ describe('Coordinator', () => {
|
||||
to: 'coord',
|
||||
subject: 'Late done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: staleCtx.id })
|
||||
payload: JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: staleCtx.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
})
|
||||
|
||||
const staleCoordinator = new Coordinator(db, runtime, {
|
||||
@@ -621,7 +630,7 @@ describe('Coordinator', () => {
|
||||
to: 'coord',
|
||||
subject: 'Done after restart',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: ctx.id }),
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: ctx.id, outcome: 'succeeded' }),
|
||||
senderPaneKey: `tab_after:${leafId}`
|
||||
})
|
||||
|
||||
|
||||
@@ -242,6 +242,7 @@ export class Coordinator {
|
||||
case 'dispatch':
|
||||
case 'handoff':
|
||||
case 'merge_ready':
|
||||
case 'question':
|
||||
break
|
||||
}
|
||||
}
|
||||
@@ -255,6 +256,10 @@ export class Coordinator {
|
||||
if (!this.state.completedTasks.includes(result.taskId)) {
|
||||
this.state.completedTasks.push(result.taskId)
|
||||
}
|
||||
return
|
||||
}
|
||||
if (result.action === 'failed' && !this.state.failedTasks.includes(result.taskId)) {
|
||||
this.state.failedTasks.push(result.taskId)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -3,7 +3,7 @@ import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import Database from '../../sqlite/sync-database'
|
||||
import { OrchestrationDb } from './db'
|
||||
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
|
||||
import type { MessageType } from './db'
|
||||
|
||||
// Overwrites the datetime('now')-seeded timestamps with explicit fixture values
|
||||
@@ -942,6 +942,8 @@ describe('OrchestrationDb', () => {
|
||||
|
||||
// v1 data preserved
|
||||
expect(d.getMessageById('msg_v1')?.subject).toBe('pre-migration')
|
||||
expect(d.getMessageById('msg_v1')?.run_id).toBe(LEGACY_RUN_ID)
|
||||
expect(d.getRun(LEGACY_RUN_ID)).toMatchObject({ legacy: 1 })
|
||||
})
|
||||
|
||||
it('adds pane-identity columns (v6) and persists them', () => {
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,26 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import type {
|
||||
RuntimeOrchestrationEnvelope,
|
||||
RuntimeRpcResponse
|
||||
} from '../../../shared/runtime-rpc-envelope'
|
||||
|
||||
export type OrchestrationWorkerServer = {
|
||||
environmentId: string
|
||||
name: string
|
||||
peerFingerprint: string
|
||||
}
|
||||
|
||||
export type OrchestrationEnvironmentTransport = {
|
||||
resolve(selector: string): OrchestrationWorkerServer
|
||||
call(
|
||||
selector: string,
|
||||
method: string,
|
||||
params: unknown,
|
||||
timeoutMs?: number,
|
||||
envelope?: RuntimeOrchestrationEnvelope
|
||||
): Promise<RuntimeRpcResponse<unknown>>
|
||||
}
|
||||
|
||||
export function fingerprintOrchestrationPeer(publicKeyB64: string): string {
|
||||
return createHash('sha256').update(Buffer.from(publicKeyB64, 'base64')).digest('base64url')
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
import { MESSAGE_TYPES, type MessagePriority, type MessageType } from './types'
|
||||
import type { OrchestrationDb } from './db'
|
||||
import { OrchestrationError } from './orchestration-error'
|
||||
|
||||
const MESSAGE_TYPE_SET = new Set<MessageType>(MESSAGE_TYPES)
|
||||
|
||||
export type FederatedControlMessage = {
|
||||
from: string
|
||||
subject: string
|
||||
body: string
|
||||
type: MessageType
|
||||
priority: MessagePriority
|
||||
threadId: string | null
|
||||
payload: string | null
|
||||
}
|
||||
|
||||
export function encodeFederatedControlMessage(message: FederatedControlMessage): string {
|
||||
return JSON.stringify(message)
|
||||
}
|
||||
|
||||
export function parseFederatedControlMessage(payload: string): FederatedControlMessage {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(payload)
|
||||
} catch {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated control message is invalid JSON.')
|
||||
}
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated control message is invalid.')
|
||||
}
|
||||
const message = parsed as Partial<FederatedControlMessage>
|
||||
if (
|
||||
typeof message.from !== 'string' ||
|
||||
typeof message.subject !== 'string' ||
|
||||
typeof message.body !== 'string' ||
|
||||
typeof message.type !== 'string' ||
|
||||
!MESSAGE_TYPE_SET.has(message.type as MessageType)
|
||||
) {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated control message is incomplete.')
|
||||
}
|
||||
return {
|
||||
from: message.from,
|
||||
subject: message.subject,
|
||||
body: message.body,
|
||||
type: message.type as MessageType,
|
||||
priority:
|
||||
message.priority === 'high' || message.priority === 'urgent' ? message.priority : 'normal',
|
||||
threadId: typeof message.threadId === 'string' ? message.threadId : null,
|
||||
payload: typeof message.payload === 'string' ? message.payload : null
|
||||
}
|
||||
}
|
||||
|
||||
export function importFederatedControlMessage(
|
||||
db: OrchestrationDb,
|
||||
params: {
|
||||
dispatchId: string
|
||||
messageId: string
|
||||
payload: string
|
||||
}
|
||||
): { imported: boolean; type: MessageType } {
|
||||
const message = parseFederatedControlMessage(params.payload)
|
||||
const recipient = `dispatch:${params.dispatchId}`
|
||||
const existing = db.getMessageById(params.messageId)
|
||||
if (existing) {
|
||||
if (
|
||||
existing.to_handle !== recipient ||
|
||||
existing.from_handle !== message.from ||
|
||||
existing.subject !== message.subject ||
|
||||
existing.body !== message.body ||
|
||||
existing.type !== message.type ||
|
||||
existing.priority !== message.priority ||
|
||||
existing.thread_id !== message.threadId ||
|
||||
existing.payload !== message.payload
|
||||
) {
|
||||
throw new OrchestrationError(
|
||||
'request_mismatch',
|
||||
`Federated control message ${params.messageId} conflicts with an existing message.`
|
||||
)
|
||||
}
|
||||
return { imported: false, type: message.type }
|
||||
}
|
||||
db.insertMessage({
|
||||
id: params.messageId,
|
||||
from: message.from,
|
||||
to: recipient,
|
||||
subject: message.subject,
|
||||
body: message.body,
|
||||
type: message.type,
|
||||
priority: message.priority,
|
||||
threadId: message.threadId ?? undefined,
|
||||
payload: message.payload ?? undefined
|
||||
})
|
||||
return { imported: true, type: message.type }
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { parseRelayedMessage } from './federation-sync'
|
||||
|
||||
describe('federation relay parsing', () => {
|
||||
it('accepts a supported message type', () => {
|
||||
expect(
|
||||
parseRelayedMessage(
|
||||
JSON.stringify({ subject: 'done', body: 'Finished', type: 'worker_done' })
|
||||
)
|
||||
).toMatchObject({ type: 'worker_done', priority: 'normal' })
|
||||
})
|
||||
|
||||
it('rejects an unsupported type before it reaches the database constraint', () => {
|
||||
expect(() =>
|
||||
parseRelayedMessage(JSON.stringify({ subject: 'bad', body: 'Blocked', type: 'invented' }))
|
||||
).toThrowError('Federated relay message type invented is not supported.')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,253 @@
|
||||
import {
|
||||
MESSAGE_TYPES,
|
||||
type MessagePriority,
|
||||
type MessageType,
|
||||
type WorkerReportOutcome
|
||||
} from './types'
|
||||
import type { OrcaRuntimeService } from '../orca-runtime'
|
||||
import { OrchestrationError } from './orchestration-error'
|
||||
|
||||
const MESSAGE_TYPE_SET = new Set<MessageType>(MESSAGE_TYPES)
|
||||
|
||||
function isMessageType(value: unknown): value is MessageType {
|
||||
return typeof value === 'string' && MESSAGE_TYPE_SET.has(value as MessageType)
|
||||
}
|
||||
|
||||
type PulledRelayItem = {
|
||||
dispatch_id: string
|
||||
direction: 'to_home'
|
||||
sequence: number
|
||||
message_id: string
|
||||
kind: string
|
||||
payload: string
|
||||
}
|
||||
|
||||
type RelayedMessage = {
|
||||
from: string
|
||||
subject: string
|
||||
body: string
|
||||
type: MessageType
|
||||
priority: MessagePriority
|
||||
threadId: string | null
|
||||
payload: string | null
|
||||
}
|
||||
|
||||
export async function syncFederatedDispatch(
|
||||
runtime: OrcaRuntimeService,
|
||||
dispatchId: string
|
||||
): Promise<{ imported: number; acknowledgedThrough: number }> {
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const federated = db.getFederatedDispatch(dispatchId)
|
||||
const dispatch = db.getDispatchContextById(dispatchId)
|
||||
if (!federated || !dispatch) {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_not_found',
|
||||
`Federated Dispatch ${dispatchId} was not found.`
|
||||
)
|
||||
}
|
||||
const currentServer = runtime.resolveOrchestrationWorkerServer(federated.environment_id)
|
||||
if (currentServer.peerFingerprint !== federated.peer_fingerprint) {
|
||||
throw new OrchestrationError(
|
||||
'peer_changed',
|
||||
`Saved environment ${federated.environment_name} now identifies a different Orca server.`
|
||||
)
|
||||
}
|
||||
|
||||
const pulled = (await runtime.callOrchestrationWorkerServer(
|
||||
federated.environment_id,
|
||||
'orchestration.federationPull',
|
||||
{
|
||||
dispatchId,
|
||||
afterSequence: federated.to_home_imported_sequence,
|
||||
limit: 50
|
||||
},
|
||||
15_000
|
||||
)) as { runtimeEpoch: string; items: PulledRelayItem[] }
|
||||
let cursor = federated.to_home_imported_sequence
|
||||
let imported = 0
|
||||
for (const item of pulled.items) {
|
||||
if (item.dispatch_id !== dispatchId || item.sequence !== cursor + 1) {
|
||||
throw new OrchestrationError(
|
||||
'operation_unknown',
|
||||
`Federated relay for ${dispatchId} is not contiguous after sequence ${cursor}.`
|
||||
)
|
||||
}
|
||||
const message = parseRelayedMessage(item.payload)
|
||||
const stored = db.importFederatedRelayItem({
|
||||
dispatchId,
|
||||
sequence: item.sequence,
|
||||
message: {
|
||||
id: item.message_id,
|
||||
runId: dispatch.run_id,
|
||||
from: `dispatch:${dispatchId}`,
|
||||
to: `run:${dispatch.run_id}`,
|
||||
subject: message.subject,
|
||||
body: message.body,
|
||||
type: message.type,
|
||||
priority: message.priority,
|
||||
threadId: message.threadId ?? undefined,
|
||||
payload: message.payload ?? undefined
|
||||
},
|
||||
lifecycle: parseFederatedLifecycle(message, item.message_id, dispatchId, dispatch.task_id)
|
||||
})
|
||||
cursor = item.sequence
|
||||
runtime.notifyMessageArrived(stored.message.to_handle, stored.message.type)
|
||||
imported += stored.duplicate ? 0 : 1
|
||||
}
|
||||
|
||||
if (cursor > 0) {
|
||||
await runtime.callOrchestrationWorkerServer(
|
||||
federated.environment_id,
|
||||
'orchestration.federationAck',
|
||||
{ dispatchId, throughSequence: cursor },
|
||||
15_000,
|
||||
{ orchestrationRequestId: `relay_ack_${dispatchId}_${cursor}` }
|
||||
)
|
||||
}
|
||||
const toWorker =
|
||||
db.getWorkerDispatch(dispatchId)?.state === 'ready'
|
||||
? db.listPendingFederationRelay(dispatchId, 'to_worker')
|
||||
: []
|
||||
if (toWorker.length > 0) {
|
||||
const delivered = (await runtime.callOrchestrationWorkerServer(
|
||||
federated.environment_id,
|
||||
'orchestration.federationImport',
|
||||
{ dispatchId, items: toWorker },
|
||||
15_000,
|
||||
{
|
||||
orchestrationRequestId: `relay_import_${dispatchId}_${toWorker.at(-1)?.sequence ?? 0}`
|
||||
}
|
||||
)) as { acknowledgedThrough: number }
|
||||
db.acknowledgeFederationRelay({
|
||||
dispatchId,
|
||||
direction: 'to_worker',
|
||||
throughSequence: delivered.acknowledgedThrough
|
||||
})
|
||||
}
|
||||
return { imported, acknowledgedThrough: cursor }
|
||||
}
|
||||
|
||||
export function parseRelayedMessage(payload: string): RelayedMessage {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(payload)
|
||||
} catch {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated relay payload is invalid JSON.')
|
||||
}
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated relay payload is not a message.')
|
||||
}
|
||||
const message = parsed as Partial<RelayedMessage>
|
||||
if (typeof message.subject !== 'string' || typeof message.body !== 'string') {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated relay message is incomplete.')
|
||||
}
|
||||
if (!isMessageType(message.type)) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
`Federated relay message type ${String(message.type)} is not supported.`
|
||||
)
|
||||
}
|
||||
return {
|
||||
from: typeof message.from === 'string' ? message.from : 'remote-worker',
|
||||
subject: message.subject,
|
||||
body: message.body,
|
||||
type: message.type,
|
||||
priority:
|
||||
message.priority === 'high' || message.priority === 'urgent' ? message.priority : 'normal',
|
||||
threadId: typeof message.threadId === 'string' ? message.threadId : null,
|
||||
payload: typeof message.payload === 'string' ? message.payload : null
|
||||
}
|
||||
}
|
||||
|
||||
function parseFederatedLifecycle(
|
||||
message: RelayedMessage,
|
||||
messageId: string,
|
||||
dispatchId: string,
|
||||
taskId: string
|
||||
):
|
||||
| { kind: 'none' }
|
||||
| { kind: 'heartbeat'; at: string }
|
||||
| {
|
||||
kind: 'worker_report'
|
||||
taskId: string
|
||||
outcome: WorkerReportOutcome
|
||||
result: string
|
||||
}
|
||||
| { kind: 'rejected'; code: string; reason: string } {
|
||||
if (message.type === 'heartbeat') {
|
||||
return { kind: 'heartbeat', at: new Date().toISOString() }
|
||||
}
|
||||
if (message.type !== 'worker_done') {
|
||||
return { kind: 'none' }
|
||||
}
|
||||
let payload
|
||||
try {
|
||||
payload = parseWorkerReportPayload(message.payload)
|
||||
} catch (error) {
|
||||
return {
|
||||
kind: 'rejected',
|
||||
code: 'invalid_payload',
|
||||
reason: error instanceof Error ? error.message : String(error)
|
||||
}
|
||||
}
|
||||
if (payload.dispatchId !== dispatchId || payload.taskId !== taskId) {
|
||||
return {
|
||||
kind: 'rejected',
|
||||
code: 'task_dispatch_mismatch',
|
||||
reason: `Federated report does not match Dispatch ${dispatchId}.`
|
||||
}
|
||||
}
|
||||
const result = JSON.stringify({
|
||||
provenance: 'worker_report',
|
||||
outcome: payload.outcome,
|
||||
messageId,
|
||||
reportedBy: `dispatch:${dispatchId}`,
|
||||
subject: message.subject,
|
||||
body: message.body,
|
||||
completedBy: `dispatch:${dispatchId}`,
|
||||
filesModified: payload.filesModified,
|
||||
reportPath: payload.reportPath,
|
||||
completedAt: new Date().toISOString()
|
||||
})
|
||||
return {
|
||||
kind: 'worker_report',
|
||||
taskId: payload.taskId,
|
||||
outcome: payload.outcome,
|
||||
result
|
||||
}
|
||||
}
|
||||
|
||||
function parseWorkerReportPayload(payload: string | null): {
|
||||
taskId: string
|
||||
dispatchId: string
|
||||
outcome: WorkerReportOutcome
|
||||
filesModified: string[]
|
||||
reportPath: string | null
|
||||
} {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = payload ? JSON.parse(payload) : null
|
||||
} catch {
|
||||
parsed = null
|
||||
}
|
||||
if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated worker report is invalid.')
|
||||
}
|
||||
const report = parsed as Record<string, unknown>
|
||||
if (
|
||||
typeof report.taskId !== 'string' ||
|
||||
typeof report.dispatchId !== 'string' ||
|
||||
(report.outcome !== 'succeeded' && report.outcome !== 'failed')
|
||||
) {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated worker report is incomplete.')
|
||||
}
|
||||
return {
|
||||
taskId: report.taskId,
|
||||
dispatchId: report.dispatchId,
|
||||
outcome: report.outcome,
|
||||
filesModified: Array.isArray(report.filesModified)
|
||||
? report.filesModified.filter((file): file is string => typeof file === 'string')
|
||||
: [],
|
||||
reportPath: typeof report.reportPath === 'string' ? report.reportPath : null
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ import type { MessageRow } from './types'
|
||||
function makeMessage(overrides: Partial<MessageRow> = {}): MessageRow {
|
||||
return {
|
||||
id: 'msg_test1',
|
||||
run_id: 'run_test',
|
||||
from_handle: 'term_abc123',
|
||||
to_handle: 'term_coord',
|
||||
subject: 'Auth API implementation complete',
|
||||
|
||||
@@ -17,7 +17,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id })
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' })
|
||||
})
|
||||
|
||||
expect(reconcileLifecycleMessage(db, message, (line) => logs.push(line))).toMatchObject({
|
||||
@@ -43,7 +43,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id }),
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' }),
|
||||
senderPaneKey: `tab_w:${LEAF_A}`
|
||||
})
|
||||
|
||||
@@ -51,6 +51,94 @@ describe('lifecycle reconciliation', () => {
|
||||
expect(db.getTask(task.id)?.status).toBe('completed')
|
||||
})
|
||||
|
||||
it('fails both the dispatch and task from an authenticated failed worker report', () => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const task = db.createTask({ spec: 'work' })
|
||||
const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`)
|
||||
const message = db.insertMessage({
|
||||
from: 'term_worker',
|
||||
to: 'term_coordinator',
|
||||
subject: 'Failed: tests cannot start',
|
||||
body: 'I attempted the work. The required service is unavailable. No files changed.',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'failed',
|
||||
filesModified: []
|
||||
}),
|
||||
senderPaneKey: `tab_w:${LEAF_A}`
|
||||
})
|
||||
|
||||
expect(reconcileLifecycleMessage(db, message)).toEqual({
|
||||
action: 'failed',
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id
|
||||
})
|
||||
expect(db.getTask(task.id)).toMatchObject({ status: 'failed' })
|
||||
expect(db.getDispatchContextById(dispatch.id)).toMatchObject({ status: 'failed' })
|
||||
expect(JSON.parse(db.getTask(task.id)?.result ?? '{}')).toMatchObject({
|
||||
provenance: 'worker_report',
|
||||
outcome: 'failed',
|
||||
messageId: message.id
|
||||
})
|
||||
})
|
||||
|
||||
it('replays an identical terminal outcome without mutating settled state', () => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const task = db.createTask({ spec: 'work' })
|
||||
const dispatch = db.createDispatchContext(task.id, 'term_worker')
|
||||
const makeMessage = () =>
|
||||
db.insertMessage({
|
||||
from: 'term_worker',
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
})
|
||||
|
||||
expect(reconcileLifecycleMessage(db, makeMessage()).action).toBe('completed')
|
||||
const result = db.getTask(task.id)?.result
|
||||
expect(reconcileLifecycleMessage(db, makeMessage()).action).toBe('completed')
|
||||
expect(db.getTask(task.id)?.result).toBe(result)
|
||||
})
|
||||
|
||||
it.each([
|
||||
{ payload: undefined, code: 'invalid_payload' },
|
||||
{ payload: '{', code: 'invalid_payload' },
|
||||
{
|
||||
payload: JSON.stringify({ dispatchId: 'ctx_1', outcome: 'succeeded' }),
|
||||
code: 'missing_task_id'
|
||||
},
|
||||
{
|
||||
payload: JSON.stringify({ taskId: 'task_1', outcome: 'succeeded' }),
|
||||
code: 'missing_dispatch_id'
|
||||
},
|
||||
{
|
||||
payload: JSON.stringify({ taskId: 'task_1', dispatchId: 'ctx_1', outcome: 'maybe' }),
|
||||
code: 'invalid_outcome'
|
||||
}
|
||||
])('rejects malformed worker reports with $code', ({ payload, code }) => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const message = db.insertMessage({
|
||||
from: 'term_worker',
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload
|
||||
})
|
||||
|
||||
expect(reconcileLifecycleMessage(db, message)).toMatchObject({ action: 'rejected', code })
|
||||
expect(db.getMessageById(message.id)).toMatchObject({
|
||||
priority: 'high',
|
||||
subject: 'Rejected worker_done: Done'
|
||||
})
|
||||
})
|
||||
|
||||
it('completes worker_done from the same leaf after a pane break-out changed the tab half', () => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const task = db.createTask({ spec: 'work' })
|
||||
@@ -62,7 +150,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id }),
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' }),
|
||||
senderPaneKey: `tab_old:${LEAF_A}`
|
||||
})
|
||||
|
||||
@@ -79,7 +167,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id }),
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' }),
|
||||
senderPaneKey: 'tab_w:42'
|
||||
})
|
||||
|
||||
@@ -96,7 +184,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id }),
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' }),
|
||||
senderPaneKey: `tab_w2:${LEAF_B}`
|
||||
})
|
||||
|
||||
@@ -142,6 +230,7 @@ describe('lifecycle reconciliation', () => {
|
||||
payload: JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded',
|
||||
_orcaLifecycleRejection: {
|
||||
code: 'sender_not_assignee',
|
||||
reason: 'caller supplied'
|
||||
@@ -167,7 +256,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id })
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' })
|
||||
})
|
||||
|
||||
expect(reconcileLifecycleMessage(db, message)).toMatchObject({
|
||||
@@ -186,7 +275,11 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: acceptedTask.id, dispatchId: acceptedDispatch.id })
|
||||
payload: JSON.stringify({
|
||||
taskId: acceptedTask.id,
|
||||
dispatchId: acceptedDispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
})
|
||||
expect(reconcileLifecycleMessage(db, accepted).action).toBe('completed')
|
||||
|
||||
@@ -197,7 +290,11 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: rejectedTask.id, dispatchId: rejectedDispatch.id })
|
||||
payload: JSON.stringify({
|
||||
taskId: rejectedTask.id,
|
||||
dispatchId: rejectedDispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
})
|
||||
expect(reconcileLifecycleMessage(db, rejected)).toMatchObject({
|
||||
action: 'rejected',
|
||||
@@ -211,7 +308,11 @@ describe('lifecycle reconciliation', () => {
|
||||
const parent = db.createTask({ spec: 'parent' })
|
||||
const child = db.createTask({ spec: 'child', deps: [parent.id] })
|
||||
const dispatch = db.createDispatchContext(parent.id, 'term_worker', `tab_w:${LEAF_A}`)
|
||||
const payload = JSON.stringify({ taskId: parent.id, dispatchId: dispatch.id })
|
||||
const payload = JSON.stringify({
|
||||
taskId: parent.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
|
||||
const foreign = db.insertMessage({
|
||||
from: 'term_coordinator',
|
||||
@@ -243,7 +344,11 @@ describe('lifecycle reconciliation', () => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const task = db.createTask({ spec: 'work' })
|
||||
const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`)
|
||||
const payload = JSON.stringify({ taskId: task.id, dispatchId: dispatch.id })
|
||||
const payload = JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
const owner = db.insertMessage({
|
||||
from: 'term_worker',
|
||||
to: 'term_coordinator',
|
||||
@@ -280,7 +385,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id }),
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' }),
|
||||
senderPaneKey: `tab_w2:${LEAF_B}`
|
||||
})
|
||||
|
||||
@@ -390,7 +495,7 @@ describe('lifecycle reconciliation', () => {
|
||||
to: 'term_coordinator',
|
||||
subject: 'Done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id })
|
||||
payload: JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, outcome: 'succeeded' })
|
||||
})
|
||||
|
||||
reconcileLifecycleMessage(db, done)
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import type { OrchestrationDb } from './db'
|
||||
import type { MessageRow } from './types'
|
||||
import type { MessageRow, WorkerReportOutcome } from './types'
|
||||
import { parsePaneKey } from '../../../shared/stable-pane-id'
|
||||
|
||||
// Why: the tab half can change on pane break-out, while opaque legacy keys
|
||||
@@ -35,11 +35,25 @@ export type LifecycleReconciliationResult =
|
||||
| { action: 'suppressed' }
|
||||
| LifecycleRejectionResult
|
||||
| { action: 'completed'; taskId: string; dispatchId: string }
|
||||
| { action: 'failed'; taskId: string; dispatchId: string }
|
||||
| { action: 'heartbeat_recorded'; dispatchId: string }
|
||||
|
||||
export type LifecycleRejectionCode =
|
||||
| 'sender_not_assignee'
|
||||
| 'dispatch_capability_invalid'
|
||||
| 'invalid_payload'
|
||||
| 'missing_task_id'
|
||||
| 'missing_dispatch_id'
|
||||
| 'invalid_outcome'
|
||||
| 'unknown_task'
|
||||
| 'unknown_dispatch'
|
||||
| 'task_dispatch_mismatch'
|
||||
| 'inactive_dispatch'
|
||||
| 'stale_dispatch'
|
||||
|
||||
export type LifecycleRejectionResult = {
|
||||
action: 'rejected'
|
||||
code: 'sender_not_assignee'
|
||||
code: LifecycleRejectionCode
|
||||
reason: string
|
||||
}
|
||||
|
||||
@@ -54,7 +68,11 @@ function parseObjectPayload(msg: MessageRow, onInvalidJson: () => void): Record<
|
||||
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(msg.payload)
|
||||
return parsed && typeof parsed === 'object' ? (parsed as Record<string, unknown>) : {}
|
||||
if (parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {
|
||||
return parsed as Record<string, unknown>
|
||||
}
|
||||
onInvalidJson()
|
||||
return {}
|
||||
} catch {
|
||||
onInvalidJson()
|
||||
return {}
|
||||
@@ -68,7 +86,7 @@ function getPersistedLifecycleRejection(
|
||||
if (
|
||||
!rejection ||
|
||||
typeof rejection !== 'object' ||
|
||||
(rejection as { code?: unknown }).code !== 'sender_not_assignee' ||
|
||||
typeof (rejection as { code?: unknown }).code !== 'string' ||
|
||||
typeof (rejection as { reason?: unknown }).reason !== 'string'
|
||||
) {
|
||||
return undefined
|
||||
@@ -77,7 +95,7 @@ function getPersistedLifecycleRejection(
|
||||
// also prevents caller-supplied markers from turning lifecycle sends into success.
|
||||
return {
|
||||
action: 'rejected',
|
||||
code: 'sender_not_assignee',
|
||||
code: (rejection as { code: LifecycleRejectionCode }).code,
|
||||
reason: (rejection as { reason: string }).reason
|
||||
}
|
||||
}
|
||||
@@ -98,6 +116,7 @@ export function reconcileLifecycleMessage(
|
||||
case 'escalation':
|
||||
case 'handoff':
|
||||
case 'decision_gate':
|
||||
case 'question':
|
||||
return { action: 'ignored' }
|
||||
}
|
||||
}
|
||||
@@ -142,7 +161,7 @@ function reconcileHeartbeatMessage(
|
||||
// a hung assignee behind another agent's timer.
|
||||
const reason = buildLifecycleAuthorityRejectionReason(dispatchId, dispatch, msg)
|
||||
onLog(`Heartbeat rejected: ${reason}`)
|
||||
db.convertLifecycleMessageToRejection(msg.id, reason)
|
||||
db.convertLifecycleMessageToRejection(msg.id, 'sender_not_assignee', reason)
|
||||
return { action: 'rejected', code: 'sender_not_assignee', reason }
|
||||
}
|
||||
|
||||
@@ -159,7 +178,9 @@ function reconcileWorkerDoneMessage(
|
||||
): LifecycleReconciliationResult {
|
||||
onLog(`Worker done: ${msg.from_handle} — ${msg.subject}`)
|
||||
|
||||
let invalidPayload = false
|
||||
const payload = parseObjectPayload(msg, () => {
|
||||
invalidPayload = true
|
||||
onLog(`Warning: invalid payload in worker_done from ${msg.from_handle}`)
|
||||
})
|
||||
const persistedRejection = getPersistedLifecycleRejection(payload)
|
||||
@@ -169,58 +190,83 @@ function reconcileWorkerDoneMessage(
|
||||
onLog(`Warning: worker_done rejected: ${persistedRejection.reason}`)
|
||||
return persistedRejection
|
||||
}
|
||||
if (invalidPayload || !msg.payload) {
|
||||
return rejectLifecycleMessage(
|
||||
db,
|
||||
msg,
|
||||
'invalid_payload',
|
||||
'worker_done requires a JSON object payload.',
|
||||
onLog
|
||||
)
|
||||
}
|
||||
|
||||
const taskId = payload.taskId
|
||||
if (typeof taskId !== 'string' || taskId.length === 0) {
|
||||
onLog(`Warning: worker_done without taskId from ${msg.from_handle}`)
|
||||
return { action: 'ignored' }
|
||||
return rejectLifecycleMessage(db, msg, 'missing_task_id', 'worker_done requires taskId.', onLog)
|
||||
}
|
||||
|
||||
const dispatchId = payload.dispatchId
|
||||
if (typeof dispatchId !== 'string' || dispatchId.length === 0) {
|
||||
onLog(`Warning: worker_done without dispatchId from ${msg.from_handle}`)
|
||||
return { action: 'ignored' }
|
||||
return rejectLifecycleMessage(
|
||||
db,
|
||||
msg,
|
||||
'missing_dispatch_id',
|
||||
'worker_done requires dispatchId.',
|
||||
onLog
|
||||
)
|
||||
}
|
||||
|
||||
const outcome = payload.outcome
|
||||
if (outcome !== 'succeeded' && outcome !== 'failed') {
|
||||
return rejectLifecycleMessage(
|
||||
db,
|
||||
msg,
|
||||
'invalid_outcome',
|
||||
'worker_done requires outcome=succeeded or outcome=failed.',
|
||||
onLog
|
||||
)
|
||||
}
|
||||
|
||||
const task = db.getTask(taskId)
|
||||
if (!task) {
|
||||
onLog(`Warning: worker_done for unknown task ${taskId}`)
|
||||
return { action: 'ignored' }
|
||||
return rejectLifecycleMessage(
|
||||
db,
|
||||
msg,
|
||||
'unknown_task',
|
||||
`worker_done references unknown task ${taskId}.`,
|
||||
onLog
|
||||
)
|
||||
}
|
||||
|
||||
// Why: taskId alone is not a completion authority; retried tasks can have
|
||||
// stale worker_done messages racing the current active dispatch.
|
||||
const dispatch = db.getDispatchContextById(dispatchId)
|
||||
if (!dispatch) {
|
||||
onLog(`Warning: worker_done for unknown dispatch ${dispatchId}`)
|
||||
return { action: 'ignored' }
|
||||
return rejectLifecycleMessage(
|
||||
db,
|
||||
msg,
|
||||
'unknown_dispatch',
|
||||
`worker_done references unknown dispatch ${dispatchId}.`,
|
||||
onLog
|
||||
)
|
||||
}
|
||||
if (dispatch.task_id !== taskId) {
|
||||
onLog(
|
||||
`Warning: worker_done dispatch ${dispatchId} belongs to ${dispatch.task_id}, not ${taskId}`
|
||||
return rejectLifecycleMessage(
|
||||
db,
|
||||
msg,
|
||||
'task_dispatch_mismatch',
|
||||
`worker_done dispatch ${dispatchId} belongs to ${dispatch.task_id}, not ${taskId}.`,
|
||||
onLog
|
||||
)
|
||||
return { action: 'ignored' }
|
||||
}
|
||||
if (!hasLifecycleAuthority(dispatch, msg)) {
|
||||
const reason = buildLifecycleAuthorityRejectionReason(dispatchId, dispatch, msg)
|
||||
onLog(`Warning: worker_done rejected: ${reason}`)
|
||||
db.convertLifecycleMessageToRejection(msg.id, reason)
|
||||
db.convertLifecycleMessageToRejection(msg.id, 'sender_not_assignee', reason)
|
||||
return { action: 'rejected', code: 'sender_not_assignee', reason }
|
||||
}
|
||||
// Why: `orchestration.send` can release the DB lock before waking the
|
||||
// coordinator; the later coordinator read still needs to observe completion.
|
||||
if (dispatch.status === 'completed' && task.status === 'completed') {
|
||||
return { action: 'completed', taskId, dispatchId }
|
||||
}
|
||||
if (dispatch.status !== 'dispatched') {
|
||||
onLog(`Warning: worker_done for inactive dispatch ${dispatchId} ignored`)
|
||||
return { action: 'ignored' }
|
||||
}
|
||||
if (db.getDispatchContext(taskId)?.id !== dispatchId || task.status !== 'dispatched') {
|
||||
onLog(`Warning: worker_done for stale dispatch ${dispatchId} ignored`)
|
||||
return { action: 'ignored' }
|
||||
}
|
||||
|
||||
const filesModified =
|
||||
Array.isArray(payload.filesModified) &&
|
||||
payload.filesModified.every((file) => typeof file === 'string')
|
||||
@@ -228,17 +274,48 @@ function reconcileWorkerDoneMessage(
|
||||
: []
|
||||
|
||||
const result = JSON.stringify({
|
||||
provenance: 'worker_report',
|
||||
outcome,
|
||||
messageId: msg.id,
|
||||
reportedBy: msg.from_handle,
|
||||
subject: msg.subject,
|
||||
body: msg.body,
|
||||
completedBy: msg.from_handle,
|
||||
filesModified,
|
||||
reportPath: typeof payload.reportPath === 'string' ? payload.reportPath : null,
|
||||
completedAt: new Date().toISOString()
|
||||
})
|
||||
db.updateTaskStatus(taskId, 'completed', result)
|
||||
const settlement = db.settleWorkerReport({
|
||||
taskId,
|
||||
dispatchId,
|
||||
outcome: outcome as WorkerReportOutcome,
|
||||
result
|
||||
})
|
||||
if (settlement.action === 'rejected') {
|
||||
return rejectLifecycleMessage(db, msg, settlement.code, settlement.reason, onLog)
|
||||
}
|
||||
suppressEarlierHeartbeats(db, msg, dispatchId)
|
||||
|
||||
onLog(`Task ${taskId} completed`)
|
||||
if (outcome === 'failed') {
|
||||
onLog(`Task ${taskId} failed by worker report`)
|
||||
return { action: 'failed', taskId, dispatchId }
|
||||
}
|
||||
onLog(`Task ${taskId} completed by worker report`)
|
||||
return { action: 'completed', taskId, dispatchId }
|
||||
}
|
||||
|
||||
function rejectLifecycleMessage(
|
||||
db: OrchestrationDb,
|
||||
msg: MessageRow,
|
||||
code: LifecycleRejectionCode,
|
||||
reason: string,
|
||||
onLog: LogFn
|
||||
): LifecycleRejectionResult {
|
||||
onLog(`Warning: ${msg.type} rejected: ${reason}`)
|
||||
db.convertLifecycleMessageToRejection(msg.id, code, reason)
|
||||
return { action: 'rejected', code, reason }
|
||||
}
|
||||
|
||||
function buildLifecycleAuthorityRejectionReason(
|
||||
dispatchId: string,
|
||||
dispatch: { assignee_handle: string | null; assignee_pane_key: string | null },
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
import { mkdtempSync, rmSync, statSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { OrchestrationDb } from './db'
|
||||
|
||||
describe.skipIf(process.platform === 'win32')('orchestration database permissions', () => {
|
||||
let directory: string | undefined
|
||||
let db: OrchestrationDb | undefined
|
||||
|
||||
afterEach(() => {
|
||||
db?.close()
|
||||
if (directory) {
|
||||
rmSync(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
it('restricts the database and live SQLite sidecars to the current user', () => {
|
||||
directory = mkdtempSync(join(tmpdir(), 'orca-orchestration-permissions-'))
|
||||
const dbPath = join(directory, 'orchestration.db')
|
||||
db = new OrchestrationDb(dbPath)
|
||||
|
||||
for (const path of [dbPath, `${dbPath}-wal`, `${dbPath}-shm`]) {
|
||||
expect(statSync(path).mode & 0o777).toBe(0o600)
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,11 @@
|
||||
export class OrchestrationError extends Error {
|
||||
readonly code: string
|
||||
readonly data?: unknown
|
||||
|
||||
constructor(code: string, message: string, data?: unknown) {
|
||||
super(message)
|
||||
this.name = 'OrchestrationError'
|
||||
this.code = code
|
||||
this.data = data
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { OrchestrationDb } from './db'
|
||||
|
||||
describe('OrchestrationDb mutation and question state', () => {
|
||||
let db: OrchestrationDb | undefined
|
||||
|
||||
afterEach(() => {
|
||||
db?.close()
|
||||
})
|
||||
|
||||
function createDb(): OrchestrationDb {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
return db
|
||||
}
|
||||
|
||||
describe('durable mutation receipts', () => {
|
||||
it('replays completed input and rejects request ID reuse with changed input', () => {
|
||||
const d = createDb()
|
||||
const started = d.beginMutationReceipt({
|
||||
callerFingerprint: 'caller_a',
|
||||
requestId: 'request_1',
|
||||
method: 'orchestration.send',
|
||||
payloadHash: 'hash_a'
|
||||
})
|
||||
expect(started.disposition).toBe('started')
|
||||
|
||||
d.completeMutationReceipt({
|
||||
callerFingerprint: 'caller_a',
|
||||
requestId: 'request_1',
|
||||
method: 'orchestration.send',
|
||||
payloadHash: 'hash_a',
|
||||
receipt: '{"messageId":"msg_1"}'
|
||||
})
|
||||
expect(
|
||||
d.beginMutationReceipt({
|
||||
callerFingerprint: 'caller_a',
|
||||
requestId: 'request_1',
|
||||
method: 'orchestration.send',
|
||||
payloadHash: 'hash_a'
|
||||
})
|
||||
).toMatchObject({
|
||||
disposition: 'completed',
|
||||
row: { receipt: '{"messageId":"msg_1"}' }
|
||||
})
|
||||
|
||||
expect(() =>
|
||||
d.beginMutationReceipt({
|
||||
callerFingerprint: 'caller_a',
|
||||
requestId: 'request_1',
|
||||
method: 'orchestration.send',
|
||||
payloadHash: 'hash_b'
|
||||
})
|
||||
).toThrow('already used with different input')
|
||||
})
|
||||
|
||||
it('keeps caller namespaces separate and can discard only pending work', () => {
|
||||
const d = createDb()
|
||||
for (const callerFingerprint of ['caller_a', 'caller_b']) {
|
||||
d.beginMutationReceipt({
|
||||
callerFingerprint,
|
||||
requestId: 'same_request',
|
||||
method: 'orchestration.send',
|
||||
payloadHash: 'same_hash'
|
||||
})
|
||||
}
|
||||
expect(d.getMutationReceipt('caller_a', 'same_request')?.state).toBe('pending')
|
||||
expect(d.getMutationReceipt('caller_b', 'same_request')?.state).toBe('pending')
|
||||
|
||||
d.discardPendingMutationReceipt('caller_a', 'same_request')
|
||||
expect(d.getMutationReceipt('caller_a', 'same_request')).toBeUndefined()
|
||||
expect(d.getMutationReceipt('caller_b', 'same_request')?.state).toBe('pending')
|
||||
})
|
||||
})
|
||||
|
||||
describe('question threads', () => {
|
||||
it('accepts a question message in the fresh canonical schema', () => {
|
||||
const d = createDb()
|
||||
const message = d.insertMessage({
|
||||
from: 'worker',
|
||||
to: 'run:run_1',
|
||||
subject: 'Need input',
|
||||
type: 'question'
|
||||
})
|
||||
|
||||
expect(message.type).toBe('question')
|
||||
})
|
||||
|
||||
it('uses the original message ID and records one durable answer', () => {
|
||||
const d = createDb()
|
||||
const run = d.createRun({
|
||||
objective: 'Questions',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
const task = d.createTask({ spec: 'ask', runId: run.id })
|
||||
const dispatch = d.createDispatchContext(task.id, 'term_worker')
|
||||
const created = d.createQuestion({
|
||||
runId: run.id,
|
||||
dispatchId: dispatch.id,
|
||||
askerHandle: 'term_worker',
|
||||
question: 'Which format?',
|
||||
options: ['old', 'new']
|
||||
})
|
||||
|
||||
expect(created.question.message_id).toBe(created.message.id)
|
||||
expect(created.message).toMatchObject({
|
||||
run_id: run.id,
|
||||
to_handle: `run:${run.id}`,
|
||||
type: 'question',
|
||||
thread_id: created.message.id
|
||||
})
|
||||
const answer = d.answerQuestion({
|
||||
messageId: created.message.id,
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
body: 'old'
|
||||
})
|
||||
const replay = d.answerQuestion({
|
||||
messageId: created.message.id,
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
body: 'old'
|
||||
})
|
||||
|
||||
expect(answer.message.to_handle).toBe(`dispatch:${dispatch.id}`)
|
||||
expect(answer.question.status).toBe('answered')
|
||||
expect(replay.message.id).toBe(answer.message.id)
|
||||
expect(replay.duplicate).toBe(true)
|
||||
expect(() =>
|
||||
d.answerQuestion({
|
||||
messageId: created.message.id,
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
body: 'new'
|
||||
})
|
||||
).toThrow(/different answer/)
|
||||
})
|
||||
|
||||
it('closes pending questions with their Dispatch', () => {
|
||||
const d = createDb()
|
||||
const run = d.createRun({
|
||||
objective: 'Close questions',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
const task = d.createTask({ spec: 'ask', runId: run.id })
|
||||
const dispatch = d.createDispatchContext(task.id, 'term_worker')
|
||||
const created = d.createQuestion({
|
||||
runId: run.id,
|
||||
dispatchId: dispatch.id,
|
||||
askerHandle: 'term_worker',
|
||||
question: 'Still active?'
|
||||
})
|
||||
|
||||
expect(d.closeQuestionsForDispatch(dispatch.id)).toEqual([created.message.id])
|
||||
expect(d.getQuestion(created.message.id)?.status).toBe('closed')
|
||||
expect(() =>
|
||||
d.answerQuestion({
|
||||
messageId: created.message.id,
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
body: 'late'
|
||||
})
|
||||
).toThrow(/inactive/)
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,101 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
|
||||
|
||||
describe('OrchestrationDb reset scopes', () => {
|
||||
let db: OrchestrationDb | undefined
|
||||
|
||||
afterEach(() => db?.close())
|
||||
|
||||
function createState() {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const run = db.createRun({
|
||||
objective: 'Reset contract',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
const task = db.createTask({ spec: 'work', runId: run.id })
|
||||
const started = db.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
startOptions: { worktree: 'current' },
|
||||
runtimeEpoch: 'runtime_1',
|
||||
federation: {
|
||||
environmentId: 'environment_1',
|
||||
environmentName: 'Windows',
|
||||
peerFingerprint: 'peer_1',
|
||||
protocolVersion: 1
|
||||
},
|
||||
mutationReceipt: {
|
||||
callerFingerprint: 'caller_1',
|
||||
requestId: 'request_1',
|
||||
method: 'orchestration.workerStart',
|
||||
payloadHash: 'hash_1'
|
||||
}
|
||||
})
|
||||
const message = db.insertMessage({
|
||||
runId: run.id,
|
||||
from: 'worker',
|
||||
to: `run:${run.id}`,
|
||||
subject: 'status'
|
||||
})
|
||||
db.enqueueFederationRelay({
|
||||
dispatchId: started.dispatch.id,
|
||||
direction: 'to_home',
|
||||
kind: 'question',
|
||||
payload: '{}',
|
||||
messageId: 'question_1',
|
||||
remoteQuestion: true
|
||||
})
|
||||
return { run, task, started, message }
|
||||
}
|
||||
|
||||
it('resetAll clears Runs, worker/federation state, and messages', () => {
|
||||
const state = createState()
|
||||
|
||||
db!.resetAll()
|
||||
|
||||
expect(db!.listRuns()).toEqual([expect.objectContaining({ id: LEGACY_RUN_ID, legacy: 1 })])
|
||||
expect(db!.getTask(state.task.id)).toBeUndefined()
|
||||
expect(db!.getWorkerDispatch(state.started.dispatch.id)).toBeUndefined()
|
||||
expect(db!.getFederatedDispatch(state.started.dispatch.id)).toBeUndefined()
|
||||
// The ledger survives so a lost reset response cannot replay as a new mutation.
|
||||
expect(db!.getMutationReceipt('caller_1', 'request_1')).toBeDefined()
|
||||
expect(db!.getInbox()).toEqual([])
|
||||
expect(
|
||||
db!.listFederationRelay({
|
||||
dispatchId: state.started.dispatch.id,
|
||||
direction: 'to_home',
|
||||
afterSequence: 0
|
||||
})
|
||||
).toEqual([])
|
||||
})
|
||||
|
||||
it('resetTasks preserves Runs and messages while clearing every worker attachment', () => {
|
||||
const state = createState()
|
||||
|
||||
db!.resetTasks()
|
||||
|
||||
expect(db!.getRun(state.run.id)).toBeDefined()
|
||||
expect(db!.getMessageById(state.message.id)).toBeDefined()
|
||||
expect(db!.getTask(state.task.id)).toBeUndefined()
|
||||
expect(db!.getWorkerDispatch(state.started.dispatch.id)).toBeUndefined()
|
||||
expect(db!.getFederatedDispatch(state.started.dispatch.id)).toBeUndefined()
|
||||
expect(db!.getRemoteQuestion('question_1')).toBeUndefined()
|
||||
})
|
||||
|
||||
it('resetMessages preserves active relay cursors while clearing the Run inbox', () => {
|
||||
const state = createState()
|
||||
|
||||
db!.resetMessages()
|
||||
|
||||
expect(db!.getTask(state.task.id)).toBeDefined()
|
||||
expect(db!.getInbox()).toEqual([])
|
||||
expect(db!.getRemoteQuestion('question_1')).toBeDefined()
|
||||
expect(
|
||||
db!.listFederationRelay({
|
||||
dispatchId: state.started.dispatch.id,
|
||||
direction: 'to_home',
|
||||
afterSequence: 0
|
||||
})
|
||||
).toHaveLength(1)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,279 @@
|
||||
import { mkdtempSync, rmSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
|
||||
|
||||
describe('OrchestrationDb Run state', () => {
|
||||
let db: OrchestrationDb | undefined
|
||||
|
||||
afterEach(() => {
|
||||
db?.close()
|
||||
})
|
||||
|
||||
function createDb(): OrchestrationDb {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
return db
|
||||
}
|
||||
|
||||
function createBoundRun(d: OrchestrationDb) {
|
||||
return d.createRun({
|
||||
objective: 'Mailbox test',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
}
|
||||
|
||||
describe('Run deliveries', () => {
|
||||
it('returns one bounded FIFO batch and replays it until acknowledgment', () => {
|
||||
const d = createDb()
|
||||
const run = createBoundRun(d)
|
||||
for (let index = 0; index < 55; index++) {
|
||||
d.insertMessage({
|
||||
from: 'worker',
|
||||
to: `run:${run.id}`,
|
||||
subject: `message ${index}`,
|
||||
runId: run.id
|
||||
})
|
||||
}
|
||||
|
||||
const first = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})
|
||||
const replay = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})
|
||||
|
||||
expect(first?.messages).toHaveLength(50)
|
||||
expect(first?.messages[0].subject).toBe('message 0')
|
||||
expect(first?.messages[49].subject).toBe('message 49')
|
||||
expect(replay?.delivery.id).toBe(first?.delivery.id)
|
||||
expect(replay?.replayed).toBe(true)
|
||||
|
||||
d.acknowledgeRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
deliveryId: first!.delivery.id
|
||||
})
|
||||
const next = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})
|
||||
expect(next?.messages.map((message) => message.subject)).toEqual([
|
||||
'message 50',
|
||||
'message 51',
|
||||
'message 52',
|
||||
'message 53',
|
||||
'message 54'
|
||||
])
|
||||
})
|
||||
|
||||
it('acknowledges the whole batch idempotently without consuming newer mail', () => {
|
||||
const d = createDb()
|
||||
const run = createBoundRun(d)
|
||||
d.insertMessage({ from: 'a', to: `run:${run.id}`, subject: 'first', runId: run.id })
|
||||
const delivery = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})!
|
||||
d.insertMessage({ from: 'b', to: `run:${run.id}`, subject: 'newer', runId: run.id })
|
||||
|
||||
const firstAck = d.acknowledgeRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
deliveryId: delivery.delivery.id
|
||||
})
|
||||
const duplicateAck = d.acknowledgeRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
deliveryId: delivery.delivery.id
|
||||
})
|
||||
|
||||
expect(firstAck.duplicate).toBe(false)
|
||||
expect(duplicateAck.duplicate).toBe(true)
|
||||
expect(
|
||||
d
|
||||
.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})
|
||||
?.messages.map((message) => message.subject)
|
||||
).toEqual(['newer'])
|
||||
})
|
||||
|
||||
it('uses type filters only as wake predicates and returns the full oldest batch', () => {
|
||||
const d = createDb()
|
||||
const run = createBoundRun(d)
|
||||
d.insertMessage({ from: 'a', to: `run:${run.id}`, subject: 'status', runId: run.id })
|
||||
expect(
|
||||
d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
wakeTypes: ['worker_done']
|
||||
})
|
||||
).toBeUndefined()
|
||||
d.insertMessage({
|
||||
from: 'b',
|
||||
to: `run:${run.id}`,
|
||||
subject: 'done',
|
||||
type: 'worker_done',
|
||||
runId: run.id
|
||||
})
|
||||
|
||||
const delivery = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
wakeTypes: ['worker_done']
|
||||
})
|
||||
expect(delivery?.messages.map((message) => message.subject)).toEqual(['status', 'done'])
|
||||
})
|
||||
|
||||
it('fences an outstanding batch when the Run consumer changes', () => {
|
||||
const d = createDb()
|
||||
const run = createBoundRun(d)
|
||||
d.insertMessage({ from: 'a', to: `run:${run.id}`, subject: 'one', runId: run.id })
|
||||
const oldDelivery = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})!
|
||||
const rebound = d.bindRun({
|
||||
runId: run.id,
|
||||
coordinatorHandle: 'term_new',
|
||||
coordinatorPaneKey: 'tab_new:22222222-2222-4222-9222-222222222222'
|
||||
})!
|
||||
|
||||
let fencedError: unknown
|
||||
try {
|
||||
d.acknowledgeRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation,
|
||||
deliveryId: oldDelivery.delivery.id
|
||||
})
|
||||
} catch (error) {
|
||||
fencedError = error
|
||||
}
|
||||
expect(fencedError).toMatchObject({ code: 'consumer_fenced' })
|
||||
const replacement = d.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: rebound.consumer_generation
|
||||
})
|
||||
expect(replacement?.delivery.id).not.toBe(oldDelivery.delivery.id)
|
||||
expect(replacement?.messages.map((message) => message.subject)).toEqual(['one'])
|
||||
})
|
||||
|
||||
it('replays an outstanding batch after reopening the database', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'orca-delivery-'))
|
||||
const dbPath = join(dir, 'orchestration.db')
|
||||
try {
|
||||
const firstDb = new OrchestrationDb(dbPath)
|
||||
const run = createBoundRun(firstDb)
|
||||
firstDb.insertMessage({
|
||||
from: 'a',
|
||||
to: `run:${run.id}`,
|
||||
subject: 'survives',
|
||||
runId: run.id
|
||||
})
|
||||
const first = firstDb.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})!
|
||||
firstDb.close()
|
||||
|
||||
const reopened = new OrchestrationDb(dbPath)
|
||||
db = reopened
|
||||
const replay = reopened.getOrCreateRunDelivery({
|
||||
runId: run.id,
|
||||
consumerGeneration: run.consumer_generation
|
||||
})
|
||||
expect(replay?.delivery.id).toBe(first.delivery.id)
|
||||
expect(replay?.messages[0].subject).toBe('survives')
|
||||
} finally {
|
||||
db?.close()
|
||||
db = undefined
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('lightweight Run scope', () => {
|
||||
it('binds creation to one pane and fences that pane when it creates another Run', () => {
|
||||
const d = createDb()
|
||||
const first = d.createRun({
|
||||
objective: 'First objective',
|
||||
coordinatorHandle: 'term_first',
|
||||
coordinatorPaneKey: 'tab_a:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
expect(first).toMatchObject({ consumer_generation: 1, legacy: 0 })
|
||||
expect(d.getCurrentRunForPane('tab_reminted:11111111-1111-4111-8111-111111111111')?.id).toBe(
|
||||
first.id
|
||||
)
|
||||
|
||||
const second = d.createRun({
|
||||
objective: 'Second objective',
|
||||
coordinatorHandle: 'term_second',
|
||||
coordinatorPaneKey: 'tab_b:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
expect(d.getRun(first.id)).toMatchObject({
|
||||
coordinator_handle: null,
|
||||
coordinator_pane_key: null,
|
||||
consumer_generation: 2
|
||||
})
|
||||
expect(d.getCurrentRunForPane('tab_b:11111111-1111-4111-8111-111111111111')?.id).toBe(
|
||||
second.id
|
||||
)
|
||||
})
|
||||
|
||||
it('rebinds a Run by incrementing its consumer generation', () => {
|
||||
const d = createDb()
|
||||
const run = d.createRun({
|
||||
objective: 'Move coordinator',
|
||||
coordinatorHandle: 'term_old',
|
||||
coordinatorPaneKey: 'tab_old:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
|
||||
expect(
|
||||
d.bindRun({
|
||||
runId: run.id,
|
||||
coordinatorHandle: 'term_new',
|
||||
coordinatorPaneKey: 'tab_new:22222222-2222-4222-9222-222222222222'
|
||||
})
|
||||
).toMatchObject({
|
||||
coordinator_handle: 'term_new',
|
||||
consumer_generation: 2
|
||||
})
|
||||
expect(d.getCurrentRunForPane('tab_old:11111111-1111-4111-8111-111111111111')).toBeUndefined()
|
||||
expect(
|
||||
d.bindRun({
|
||||
runId: LEGACY_RUN_ID,
|
||||
coordinatorHandle: 'term_new',
|
||||
coordinatorPaneKey: 'tab_new:22222222-2222-4222-9222-222222222222'
|
||||
})
|
||||
).toBeUndefined()
|
||||
})
|
||||
|
||||
it('associates task, dispatch, message, and gate rows with the selected Run', () => {
|
||||
const d = createDb()
|
||||
const run = d.createRun({
|
||||
objective: 'Scoped work',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
|
||||
})
|
||||
const task = d.createTask({ spec: 'work', runId: run.id })
|
||||
const dispatch = d.createDispatchContext(task.id, 'term_worker')
|
||||
const message = d.insertMessage({
|
||||
runId: run.id,
|
||||
from: 'term_worker',
|
||||
to: 'term_coord',
|
||||
subject: 'status'
|
||||
})
|
||||
const gate = d.createGate({ taskId: task.id, question: 'Continue?' })
|
||||
|
||||
expect(task.run_id).toBe(run.id)
|
||||
expect(dispatch.run_id).toBe(run.id)
|
||||
expect(message.run_id).toBe(run.id)
|
||||
expect(gate.run_id).toBe(run.id)
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,274 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { OrchestrationDb } from './db'
|
||||
|
||||
describe('OrchestrationDb worker Dispatch state', () => {
|
||||
let db: OrchestrationDb | undefined
|
||||
|
||||
afterEach(() => {
|
||||
db?.close()
|
||||
})
|
||||
|
||||
function createDb(): OrchestrationDb {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
return db
|
||||
}
|
||||
|
||||
it('creates and activates a composed worker Dispatch transactionally', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'worker' })
|
||||
const started = d.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
startOptions: { topology: 'current', agent: 'codex' }
|
||||
})
|
||||
expect(started).toMatchObject({
|
||||
dispatch: { status: 'pending' },
|
||||
worker: { state: 'starting', stage: 'accepted' }
|
||||
})
|
||||
expect(d.getTask(task.id)?.status).toBe('dispatched')
|
||||
|
||||
const capability = d.prepareStartingWorkerAuthority({
|
||||
dispatchId: started.dispatch.id,
|
||||
handle: 'term_worker',
|
||||
paneKey: 'tab_worker:leaf_worker',
|
||||
processIncarnation: 'runtime:pty:1',
|
||||
worktreeId: 'repo::worktree',
|
||||
setupState: 'not_applicable',
|
||||
effects: [{ kind: 'terminal', action: 'created', id: 'term_worker' }]
|
||||
})
|
||||
expect(capability).toMatch(/^dcap_/)
|
||||
expect(d.markWorkerDispatchReady(started.dispatch.id)).toMatchObject({
|
||||
state: 'ready',
|
||||
stage: 'input_accepted'
|
||||
})
|
||||
expect(d.getDispatchContextById(started.dispatch.id)).toMatchObject({
|
||||
status: 'dispatched',
|
||||
assignee_handle: 'term_worker'
|
||||
})
|
||||
})
|
||||
|
||||
it('commits worker-start mutation acceptance with the starting Dispatch', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'atomic acceptance' })
|
||||
const mutationReceipt = {
|
||||
callerFingerprint: 'caller_fingerprint',
|
||||
requestId: 'worker_start_request',
|
||||
method: 'orchestration.workerStart',
|
||||
payloadHash: 'payload_hash'
|
||||
}
|
||||
|
||||
const started = d.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
startOptions: { topology: 'current' },
|
||||
mutationReceipt
|
||||
})
|
||||
|
||||
expect(d.getMutationReceipt('caller_fingerprint', 'worker_start_request')).toMatchObject({
|
||||
state: 'pending',
|
||||
method: 'orchestration.workerStart'
|
||||
})
|
||||
expect(d.getWorkerDispatch(started.dispatch.id)).toMatchObject({
|
||||
state: 'starting',
|
||||
stage: 'accepted'
|
||||
})
|
||||
expect(d.getTask(task.id)?.status).toBe('dispatched')
|
||||
})
|
||||
|
||||
it('rolls back worker-start mutation acceptance when the Task cannot start', () => {
|
||||
const d = createDb()
|
||||
|
||||
expect(() =>
|
||||
d.createStartingWorkerDispatch({
|
||||
taskId: 'task_missing',
|
||||
startOptions: {},
|
||||
mutationReceipt: {
|
||||
callerFingerprint: 'caller_fingerprint',
|
||||
requestId: 'invalid_worker_start',
|
||||
method: 'orchestration.workerStart',
|
||||
payloadHash: 'payload_hash'
|
||||
}
|
||||
})
|
||||
).toThrow('was not found')
|
||||
expect(d.getMutationReceipt('caller_fingerprint', 'invalid_worker_start')).toBeUndefined()
|
||||
})
|
||||
|
||||
it('fails a composed start without losing residual resource receipts', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'worker' })
|
||||
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
|
||||
d.recordWorkerStage({
|
||||
dispatchId: started.dispatch.id,
|
||||
stage: 'terminal_created',
|
||||
effects: [{ kind: 'terminal', action: 'created', id: 'term_worker' }],
|
||||
residualResources: [{ kind: 'terminal', id: 'term_worker' }]
|
||||
})
|
||||
|
||||
expect(d.failWorkerStart(started.dispatch.id, 'agent_readiness', 'timed out')).toMatchObject({
|
||||
state: 'failed',
|
||||
stage: 'agent_readiness',
|
||||
last_error: 'timed out',
|
||||
residual_resources: expect.stringContaining('term_worker')
|
||||
})
|
||||
expect(d.getTask(task.id)?.status).toBe('failed')
|
||||
})
|
||||
|
||||
it('allows retry only from the Task current terminal Dispatch', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'retry current' })
|
||||
const first = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
|
||||
d.failWorkerStart(first.dispatch.id, 'agent_readiness', 'first failed')
|
||||
const second = d.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
retryOf: first.dispatch.id,
|
||||
startOptions: {}
|
||||
})
|
||||
d.failWorkerStart(second.dispatch.id, 'agent_readiness', 'second failed')
|
||||
|
||||
expect(() =>
|
||||
d.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
retryOf: first.dispatch.id,
|
||||
startOptions: {}
|
||||
})
|
||||
).toThrow('cannot retry')
|
||||
expect(
|
||||
d.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
retryOf: second.dispatch.id,
|
||||
startOptions: {}
|
||||
}).worker.state
|
||||
).toBe('starting')
|
||||
})
|
||||
|
||||
it('treats abandon of a superseded Dispatch as a no-op', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'stale abandon' })
|
||||
const first = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
|
||||
d.failWorkerStart(first.dispatch.id, 'agent_readiness', 'first failed')
|
||||
const second = d.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
retryOf: first.dispatch.id,
|
||||
startOptions: {}
|
||||
})
|
||||
d.prepareStartingWorkerAuthority({
|
||||
dispatchId: second.dispatch.id,
|
||||
handle: 'term_replacement',
|
||||
paneKey: 'tab_replacement:leaf_replacement',
|
||||
processIncarnation: 'runtime:pty:2',
|
||||
worktreeId: 'repo::worktree',
|
||||
setupState: 'not_applicable',
|
||||
effects: []
|
||||
})
|
||||
d.markWorkerDispatchReady(second.dispatch.id)
|
||||
|
||||
expect(d.abandonWorkerDispatch(first.dispatch.id)).toMatchObject({
|
||||
disposition: 'stale',
|
||||
worker: { state: 'failed' }
|
||||
})
|
||||
expect(d.getTask(task.id)?.status).toBe('dispatched')
|
||||
expect(d.getWorkerDispatch(second.dispatch.id)?.state).toBe('ready')
|
||||
expect(
|
||||
d.settleWorkerReport({
|
||||
taskId: task.id,
|
||||
dispatchId: second.dispatch.id,
|
||||
outcome: 'succeeded',
|
||||
result: '{}'
|
||||
})
|
||||
).toMatchObject({ action: 'settled' })
|
||||
expect(d.getTask(task.id)?.status).toBe('completed')
|
||||
})
|
||||
|
||||
it('lets the stop fence win before a late worker completion', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'race' })
|
||||
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
|
||||
d.prepareStartingWorkerAuthority({
|
||||
dispatchId: started.dispatch.id,
|
||||
handle: 'term_worker',
|
||||
paneKey: 'tab_worker:leaf_worker',
|
||||
processIncarnation: 'runtime:pty:1',
|
||||
worktreeId: 'repo::worktree',
|
||||
setupState: 'not_applicable',
|
||||
effects: []
|
||||
})
|
||||
d.markWorkerDispatchReady(started.dispatch.id)
|
||||
|
||||
expect(d.beginWorkerStop(started.dispatch.id).disposition).toBe('stopping')
|
||||
expect(
|
||||
d.settleWorkerReport({
|
||||
taskId: task.id,
|
||||
dispatchId: started.dispatch.id,
|
||||
outcome: 'succeeded',
|
||||
result: '{}'
|
||||
})
|
||||
).toMatchObject({ action: 'rejected', code: 'inactive_dispatch' })
|
||||
expect(d.settleWorkerStop(started.dispatch.id).state).toBe('stopped')
|
||||
expect(d.getTask(task.id)?.status).toBe('blocked')
|
||||
})
|
||||
|
||||
it('allows explicit stop recovery from uncertain local and remote starts', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'uncertain local start' })
|
||||
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
|
||||
d.markWorkerStartUnknown(started.dispatch.id, 'agent_readiness', 'connection lost')
|
||||
|
||||
expect(d.beginWorkerStop(started.dispatch.id)).toMatchObject({
|
||||
disposition: 'stopping',
|
||||
worker: { state: 'stopping' }
|
||||
})
|
||||
|
||||
d.createRemoteDispatchAttachment({
|
||||
dispatchId: 'ctx_remote_unknown',
|
||||
taskId: 'task_remote_unknown',
|
||||
homePeerFingerprint: 'home_peer',
|
||||
protocolVersion: 1,
|
||||
runtimeEpoch: 'worker_epoch',
|
||||
mutationReceipt: {
|
||||
callerFingerprint: 'home_peer',
|
||||
requestId: 'remote_unknown_start',
|
||||
method: 'orchestration.federationAttachStart',
|
||||
payloadHash: 'remote_unknown_payload'
|
||||
}
|
||||
})
|
||||
d.recordRemoteAttachmentStage({
|
||||
dispatchId: 'ctx_remote_unknown',
|
||||
stage: 'agent_readiness',
|
||||
state: 'start_unknown',
|
||||
terminalHandle: 'term_remote_worker'
|
||||
})
|
||||
|
||||
expect(d.beginRemoteAttachmentStop('ctx_remote_unknown')).toMatchObject({
|
||||
state: 'stopping',
|
||||
stage: 'stop_requested',
|
||||
capability_hash: null
|
||||
})
|
||||
})
|
||||
|
||||
it('returns already-settled when completion wins before stop', () => {
|
||||
const d = createDb()
|
||||
const task = d.createTask({ spec: 'race' })
|
||||
const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} })
|
||||
d.prepareStartingWorkerAuthority({
|
||||
dispatchId: started.dispatch.id,
|
||||
handle: 'term_worker',
|
||||
paneKey: 'tab_worker:leaf_worker',
|
||||
processIncarnation: 'runtime:pty:1',
|
||||
worktreeId: 'repo::worktree',
|
||||
setupState: 'not_applicable',
|
||||
effects: []
|
||||
})
|
||||
d.markWorkerDispatchReady(started.dispatch.id)
|
||||
expect(
|
||||
d.settleWorkerReport({
|
||||
taskId: task.id,
|
||||
dispatchId: started.dispatch.id,
|
||||
outcome: 'succeeded',
|
||||
result: '{}'
|
||||
})
|
||||
).toMatchObject({ action: 'settled' })
|
||||
|
||||
expect(d.beginWorkerStop(started.dispatch.id)).toMatchObject({
|
||||
disposition: 'already_settled',
|
||||
worker: { state: 'succeeded' }
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -45,9 +45,12 @@ describe('buildDispatchPreamble', () => {
|
||||
expect(result).toContain('reportPath')
|
||||
expect(result).toContain('--task-id task_abc123')
|
||||
expect(result).toContain('--dispatch-id ctx_def456')
|
||||
expect(result).toContain('--outcome succeeded')
|
||||
expect(result).toContain('replace it with --outcome failed')
|
||||
expect(result).toContain('--files-modified "path/a,path/b"')
|
||||
expect(result).toContain('--report-path "<optional: path to the full artifact>"')
|
||||
expect(result).toMatch(/orchestration send --to term_coord --from term_worker/)
|
||||
expect(result).toMatch(/orchestration send --from term_worker/)
|
||||
expect(result).not.toContain('orchestration send --to term_coord')
|
||||
})
|
||||
|
||||
it(
|
||||
@@ -85,12 +88,12 @@ describe('buildDispatchPreamble', () => {
|
||||
expect(result).toContain('--task-id task_abc123')
|
||||
expect(result).toContain('--dispatch-id ctx_def456')
|
||||
expect(result).toContain('--phase "<short: investigating|implementing|reviewing|waiting>"')
|
||||
expect(result).toMatch(/orchestration send --to term_coord --from term_worker/)
|
||||
expect(result).toMatch(/orchestration send --from term_worker/)
|
||||
})
|
||||
|
||||
it('includes ask block with BEHAVIOR RULE #1 forbidding AskUserQuestion', () => {
|
||||
const result = buildDispatchPreamble(baseParams())
|
||||
expect(result).toMatch(/orchestration ask --to term_coord --from term_worker/)
|
||||
expect(result).toMatch(/orchestration ask --from term_worker/)
|
||||
expect(result).toContain('--question')
|
||||
expect(result).toContain('--timeout-ms 600000')
|
||||
// Why: the exact phrase is asserted so the rule can't be trimmed away by
|
||||
@@ -101,21 +104,27 @@ describe('buildDispatchPreamble', () => {
|
||||
// else (e.g., not in an example payload or header). Count occurrences
|
||||
// of the exact token as a sanity check.
|
||||
const occurrences = (result.match(/AskUserQuestion/g) ?? []).length
|
||||
// Three mentions: the one-liner ban, the TUI-prompt rationale, and the
|
||||
// "when tempted to reach for AskUserQuestion" closing line.
|
||||
expect(occurrences).toBe(3)
|
||||
expect(occurrences).toBe(2)
|
||||
})
|
||||
|
||||
it('binds every injected worker command to the dispatched terminal', () => {
|
||||
const result = buildDispatchPreamble(baseParams())
|
||||
|
||||
expect(result).toMatch(/orchestration ask --to term_coord --from term_worker/)
|
||||
expect(result).toMatch(
|
||||
/orchestration send --to term_coord --from term_worker \\\n --type escalation/
|
||||
)
|
||||
expect(result).toMatch(/orchestration ask --from term_worker/)
|
||||
expect(result).toMatch(/orchestration send --from term_worker \\\n --type escalation/)
|
||||
expect(result).toContain('orchestration check --terminal term_worker')
|
||||
})
|
||||
|
||||
it('carries the minted Dispatch capability on lifecycle and question commands', () => {
|
||||
const result = buildDispatchPreamble({
|
||||
...baseParams(),
|
||||
dispatchCapability: 'dcap_test_secret'
|
||||
})
|
||||
|
||||
expect(result.match(/--dispatch-capability dcap_test_secret/g)).toHaveLength(4)
|
||||
expect(result).not.toContain('"dispatchCapability"')
|
||||
})
|
||||
|
||||
it('tells prompt-returning workers to idle without post-done polling', () => {
|
||||
const result = buildDispatchPreamble(baseParams())
|
||||
const section = afterWorkerDoneSection(result)
|
||||
|
||||
@@ -8,6 +8,7 @@ export type PreambleParams = {
|
||||
// prevents stale messages from a previously-failed dispatch from completing
|
||||
// or refreshing the retry.
|
||||
dispatchId: string
|
||||
dispatchCapability?: string
|
||||
taskSpec: string
|
||||
coordinatorHandle: string
|
||||
workerHandle: string
|
||||
@@ -52,6 +53,9 @@ export function buildDispatchPreamble(params: PreambleParams): string {
|
||||
cli,
|
||||
workerKind: params.workerKind ?? 'prompt-returning-agent'
|
||||
})
|
||||
const capabilityFlag = params.dispatchCapability
|
||||
? ` --dispatch-capability ${params.dispatchCapability}`
|
||||
: ''
|
||||
|
||||
const header = `You are working inside Orca, a multi-agent IDE. You are a dispatched worker.
|
||||
Your coordinator's terminal handle is: ${params.coordinatorHandle}
|
||||
@@ -62,7 +66,7 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
|
||||
=== CLI COMMANDS ===
|
||||
|
||||
# Report task completion (REQUIRED when done — even on failure).
|
||||
# Report the terminal task outcome (REQUIRED exactly once).
|
||||
#
|
||||
# RULE: --body must be a 3-sentence executive summary (what you did,
|
||||
# what you found, what's left). Never send an empty body; the coordinator
|
||||
@@ -70,14 +74,15 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
# If you produced a long-form artifact, include its path as
|
||||
# payload.reportPath so the coordinator can find it without a file search.
|
||||
#
|
||||
# RULE: send worker_done exactly once. Failure is still a worker_done
|
||||
# with subject like "Failed: <reason>" — never silently exit.
|
||||
# RULE: send worker_done exactly once. Use --outcome succeeded when the
|
||||
# requested work is done, or replace it with --outcome failed when it is not.
|
||||
# Never encode failure only in prose and never silently exit.
|
||||
# Include BOTH taskId and dispatchId in the payload so a late completion
|
||||
# from a failed retry cannot complete the current dispatch.
|
||||
${cli} orchestration send --to ${params.coordinatorHandle} --from ${params.workerHandle} \\
|
||||
${cli} orchestration send --from ${params.workerHandle}${capabilityFlag} \\
|
||||
--type worker_done --subject "<short status>" \\
|
||||
--body "<3-sentence summary: what you did, what you found, what's left>" \\
|
||||
--task-id ${params.taskId} --dispatch-id ${params.dispatchId} \\
|
||||
--task-id ${params.taskId} --dispatch-id ${params.dispatchId} --outcome succeeded \\
|
||||
--files-modified "path/a,path/b" \\
|
||||
--report-path "<optional: path to the full artifact>"
|
||||
|
||||
@@ -91,7 +96,7 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
# attributes the heartbeat to the specific dispatch context, not just
|
||||
# the task, so a straggler heartbeat from a previously-failed dispatch
|
||||
# cannot mask a hung retry.
|
||||
${cli} orchestration send --to ${params.coordinatorHandle} --from ${params.workerHandle} \\
|
||||
${cli} orchestration send --from ${params.workerHandle}${capabilityFlag} \\
|
||||
--type heartbeat --subject "alive" \\
|
||||
--task-id ${params.taskId} --dispatch-id ${params.dispatchId} \\
|
||||
--phase "<short: investigating|implementing|reviewing|waiting>"
|
||||
@@ -104,18 +109,18 @@ Slack, GitHub comments, or any other channel to reach a human during the run.
|
||||
# coordinator cannot see and cannot answer — your session will hang forever
|
||||
# waiting on a human. Every interactive question goes through \`ask\` below.
|
||||
#
|
||||
# The \`ask\` verb is a thin wrapper: it sends a decision_gate message and
|
||||
# blocks on \`check --wait\` until the coordinator replies, then prints the
|
||||
# reply body. Use it anywhere you would otherwise have reached for
|
||||
# AskUserQuestion.
|
||||
${cli} orchestration ask --to ${params.coordinatorHandle} --from ${params.workerHandle} \\
|
||||
# The \`ask\` verb durably records a question in this Dispatch's Run and
|
||||
# blocks until the coordinator replies, then prints the reply body. If the
|
||||
# call times out or disconnects, resume with the returned message ID instead
|
||||
# of creating a duplicate question.
|
||||
${cli} orchestration ask --from ${params.workerHandle}${capabilityFlag} \\
|
||||
--question "<your question>" \\
|
||||
--options "<optional,comma,separated>" \\
|
||||
--timeout-ms 600000
|
||||
|
||||
# Escalate a blocker or failure (pre-completion, when you need the
|
||||
# coordinator to do something before you can continue):
|
||||
${cli} orchestration send --to ${params.coordinatorHandle} --from ${params.workerHandle} \\
|
||||
${cli} orchestration send --from ${params.workerHandle}${capabilityFlag} \\
|
||||
--type escalation --subject "Blocked: <reason>" \\
|
||||
--body "<details>" \\
|
||||
--task-id ${params.taskId}
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { buildObservedSetupCommand, createSetupCompletionScanner } from './setup-completion-signal'
|
||||
|
||||
describe('orchestration setup completion signal', () => {
|
||||
it('preserves a POSIX setup exit code in a visible completion signal', () => {
|
||||
const { command } = buildObservedSetupCommand(
|
||||
'/repo/.git/orca/setup-runner.sh',
|
||||
'posix',
|
||||
'token-posix'
|
||||
)
|
||||
|
||||
expect(command).toContain('bash /repo/.git/orca/setup-runner.sh')
|
||||
expect(command).toContain('__ORCA_SETUP_COMPLETE__:token-posix:%s\\n')
|
||||
expect(command).toContain('"$status"')
|
||||
expect(command).toContain('exit "$status"')
|
||||
})
|
||||
|
||||
it('preserves a native Windows setup path and exit code without shell interpolation', () => {
|
||||
const runnerPath = 'C:\\repo %name%!^&\\.git\\orca\\setup-runner.cmd'
|
||||
const observed = buildObservedSetupCommand(runnerPath, 'windows', 'token-windows')
|
||||
const encodedCommand = observed.command.split(' ').at(-1)
|
||||
const script = Buffer.from(encodedCommand ?? '', 'base64').toString('utf16le')
|
||||
|
||||
expect(observed.command).toContain('powershell.exe -NoLogo -NoProfile -NonInteractive')
|
||||
expect(observed.env).toEqual({ ORCA_SETUP_RUNNER_PATH: runnerPath })
|
||||
expect(script).toContain('& $runner')
|
||||
expect(script).toContain('__ORCA_SETUP_COMPLETE__:token-windows:')
|
||||
expect(script).toContain('exit $status')
|
||||
expect(script).not.toContain(runnerPath)
|
||||
})
|
||||
|
||||
it('keeps a WSL runner on the POSIX completion path', () => {
|
||||
const { command } = buildObservedSetupCommand(
|
||||
'\\\\wsl.localhost\\Ubuntu\\repo\\.git\\orca\\setup-runner.sh',
|
||||
'windows',
|
||||
'token-wsl'
|
||||
)
|
||||
|
||||
expect(command).toContain('bash /repo/.git/orca/setup-runner.sh')
|
||||
expect(command).toContain('__ORCA_SETUP_COMPLETE__:token-wsl:%s\\n')
|
||||
expect(command).toContain('exit "$status"')
|
||||
})
|
||||
|
||||
it('recognizes one completion signal across output chunk boundaries', () => {
|
||||
const onComplete = vi.fn()
|
||||
const scanner = createSetupCompletionScanner('token-chunks', onComplete)
|
||||
|
||||
scanner.scan('installing...\r\n__ORCA_SETUP_COMPLETE__:wrong:0\r\n__ORCA_SETUP_COMP')
|
||||
scanner.scan('LETE__:token-chunks:1')
|
||||
expect(onComplete).not.toHaveBeenCalled()
|
||||
scanner.scan('7\r')
|
||||
expect(onComplete).not.toHaveBeenCalled()
|
||||
scanner.scan('\nPS C:\\repo>')
|
||||
scanner.scan('__ORCA_SETUP_COMPLETE__:token-chunks:0\r\n')
|
||||
|
||||
expect(onComplete).toHaveBeenCalledOnce()
|
||||
expect(onComplete).toHaveBeenCalledWith(17)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,80 @@
|
||||
import {
|
||||
resolveSetupRunnerCommand,
|
||||
type SetupRunnerCommandPlatform
|
||||
} from '../../../shared/setup-runner-command'
|
||||
|
||||
const SETUP_COMPLETION_PREFIX = '__ORCA_SETUP_COMPLETE__:'
|
||||
const SETUP_COMPLETION_CARRY_LENGTH = SETUP_COMPLETION_PREFIX.length + 96
|
||||
const WINDOWS_SETUP_RUNNER_ENV = 'ORCA_SETUP_RUNNER_PATH'
|
||||
|
||||
export function buildObservedSetupCommand(
|
||||
runnerScriptPath: string,
|
||||
platform: SetupRunnerCommandPlatform,
|
||||
completionToken: string
|
||||
): { command: string; env?: Record<string, string> } {
|
||||
const resolution = resolveSetupRunnerCommand(runnerScriptPath, platform)
|
||||
if (resolution.shell === 'windows') {
|
||||
const script = [
|
||||
`$runner = $env:${WINDOWS_SETUP_RUNNER_ENV}`,
|
||||
'& $runner',
|
||||
'$succeeded = $?',
|
||||
'$status = $LASTEXITCODE',
|
||||
'if ($null -eq $status) { $status = if ($succeeded) { 0 } else { 1 } }',
|
||||
`Write-Output ('${completionPrefix(completionToken)}' + $status)`,
|
||||
'exit $status'
|
||||
].join('; ')
|
||||
return {
|
||||
command: `powershell.exe -NoLogo -NoProfile -NonInteractive -EncodedCommand ${Buffer.from(
|
||||
script,
|
||||
'utf16le'
|
||||
).toString('base64')}`,
|
||||
env: { [WINDOWS_SETUP_RUNNER_ENV]: resolution.runnerScriptPathForShell }
|
||||
}
|
||||
}
|
||||
|
||||
const script = [
|
||||
`( ${resolution.command} )`,
|
||||
'status=$?',
|
||||
`printf '\\n${completionPrefix(completionToken)}%s\\n' "$status"`,
|
||||
'exit "$status"'
|
||||
].join('; ')
|
||||
return { command: `bash -lc ${quotePosixArg(script)}` }
|
||||
}
|
||||
|
||||
export function createSetupCompletionScanner(
|
||||
completionToken: string,
|
||||
onComplete: (exitCode: number) => void
|
||||
): {
|
||||
scan: (data: string) => void
|
||||
} {
|
||||
const expectedPrefix = completionPrefix(completionToken)
|
||||
let carry = ''
|
||||
let completed = false
|
||||
return {
|
||||
scan(data: string): void {
|
||||
if (completed || data.length === 0) {
|
||||
return
|
||||
}
|
||||
const combined = `${carry}${data}`
|
||||
const markerIndex = combined.lastIndexOf(expectedPrefix)
|
||||
if (markerIndex >= 0) {
|
||||
const suffix = combined.slice(markerIndex + expectedPrefix.length)
|
||||
const match = suffix.match(/^(-?\d+)\r?\n/)
|
||||
if (match) {
|
||||
completed = true
|
||||
onComplete(Number.parseInt(match[1], 10))
|
||||
return
|
||||
}
|
||||
}
|
||||
carry = combined.slice(-SETUP_COMPLETION_CARRY_LENGTH)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function completionPrefix(completionToken: string): string {
|
||||
return `${SETUP_COMPLETION_PREFIX}${completionToken}:`
|
||||
}
|
||||
|
||||
function quotePosixArg(value: string): string {
|
||||
return `'${value.replace(/'/g, `'\\''`)}'`
|
||||
}
|
||||
@@ -1,12 +1,16 @@
|
||||
export type MessageType =
|
||||
| 'status'
|
||||
| 'dispatch'
|
||||
| 'worker_done'
|
||||
| 'merge_ready'
|
||||
| 'escalation'
|
||||
| 'handoff'
|
||||
| 'decision_gate'
|
||||
| 'heartbeat'
|
||||
export const MESSAGE_TYPES = [
|
||||
'status',
|
||||
'dispatch',
|
||||
'worker_done',
|
||||
'merge_ready',
|
||||
'escalation',
|
||||
'handoff',
|
||||
'decision_gate',
|
||||
'question',
|
||||
'heartbeat'
|
||||
] as const
|
||||
|
||||
export type MessageType = (typeof MESSAGE_TYPES)[number]
|
||||
|
||||
export type MessagePriority = 'normal' | 'high' | 'urgent'
|
||||
|
||||
@@ -14,12 +18,158 @@ export type TaskStatus = 'pending' | 'ready' | 'dispatched' | 'completed' | 'fai
|
||||
|
||||
export type DispatchStatus = 'pending' | 'dispatched' | 'completed' | 'failed' | 'circuit_broken'
|
||||
|
||||
export type WorkerReportOutcome = 'succeeded' | 'failed'
|
||||
|
||||
export type WorkerReportSettlement =
|
||||
| { action: 'settled'; outcome: WorkerReportOutcome; duplicate: boolean }
|
||||
| {
|
||||
action: 'rejected'
|
||||
code:
|
||||
| 'unknown_task'
|
||||
| 'unknown_dispatch'
|
||||
| 'task_dispatch_mismatch'
|
||||
| 'inactive_dispatch'
|
||||
| 'stale_dispatch'
|
||||
reason: string
|
||||
}
|
||||
|
||||
export type GateStatus = 'pending' | 'resolved' | 'timeout'
|
||||
|
||||
export type CoordinatorStatus = 'idle' | 'running' | 'completed' | 'failed'
|
||||
|
||||
export type RunRow = {
|
||||
id: string
|
||||
objective: string
|
||||
home_database: string
|
||||
coordinator_handle: string | null
|
||||
coordinator_pane_key: string | null
|
||||
consumer_generation: number
|
||||
legacy: number
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export type DeliveryStatus = 'outstanding' | 'acknowledged' | 'fenced'
|
||||
|
||||
export type DeliveryRow = {
|
||||
id: string
|
||||
run_id: string
|
||||
consumer_generation: number
|
||||
message_ids: string
|
||||
status: DeliveryStatus
|
||||
created_at: string
|
||||
acknowledged_at: string | null
|
||||
}
|
||||
|
||||
export type QuestionStatus = 'pending' | 'answered' | 'closed'
|
||||
|
||||
export type QuestionRow = {
|
||||
message_id: string
|
||||
run_id: string
|
||||
dispatch_id: string
|
||||
asker_handle: string
|
||||
status: QuestionStatus
|
||||
answer_message_id: string | null
|
||||
answer_body: string | null
|
||||
answered_by_generation: number | null
|
||||
created_at: string
|
||||
answered_at: string | null
|
||||
closed_at: string | null
|
||||
}
|
||||
|
||||
export type MutationState = 'pending' | 'completed'
|
||||
|
||||
export type MutationReceiptRow = {
|
||||
caller_fingerprint: string
|
||||
request_id: string
|
||||
method: string
|
||||
payload_hash: string
|
||||
state: MutationState
|
||||
receipt: string | null
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export type WorkerDispatchState =
|
||||
| 'starting'
|
||||
| 'ready'
|
||||
| 'start_unknown'
|
||||
| 'failed'
|
||||
| 'succeeded'
|
||||
| 'stopping'
|
||||
| 'stop_unknown'
|
||||
| 'stopped'
|
||||
| 'abandoned'
|
||||
|
||||
export type WorkerDispatchRow = {
|
||||
dispatch_id: string
|
||||
runtime_epoch: string | null
|
||||
state: WorkerDispatchState
|
||||
stage: string
|
||||
worktree_id: string | null
|
||||
agent_terminal_handle: string | null
|
||||
setup_state: string
|
||||
effects: string
|
||||
residual_resources: string
|
||||
start_options: string
|
||||
last_error: string | null
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export type FederatedDispatchRow = {
|
||||
dispatch_id: string
|
||||
environment_id: string
|
||||
environment_name: string
|
||||
peer_fingerprint: string
|
||||
remote_runtime_epoch: string | null
|
||||
protocol_version: number
|
||||
remote_worktree_id: string | null
|
||||
remote_terminal_handle: string | null
|
||||
to_home_imported_sequence: number
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export type RemoteDispatchAttachmentRow = {
|
||||
dispatch_id: string
|
||||
task_id: string
|
||||
home_peer_fingerprint: string
|
||||
protocol_version: number
|
||||
runtime_epoch: string
|
||||
capability_hash: string | null
|
||||
pane_key: string | null
|
||||
process_incarnation: string | null
|
||||
state: WorkerDispatchState
|
||||
stage: string
|
||||
worktree_id: string | null
|
||||
terminal_handle: string | null
|
||||
setup_state: string
|
||||
effects: string
|
||||
residual_resources: string
|
||||
to_worker_imported_sequence: number
|
||||
last_error: string | null
|
||||
created_at: string
|
||||
updated_at: string
|
||||
}
|
||||
|
||||
export type FederationRelayDirection = 'to_home' | 'to_worker'
|
||||
|
||||
export type FederationRelayItemRow = {
|
||||
dispatch_id: string
|
||||
direction: FederationRelayDirection
|
||||
sequence: number
|
||||
message_id: string
|
||||
kind: string
|
||||
payload: string
|
||||
byte_count: number
|
||||
acked_at: string | null
|
||||
created_at: string
|
||||
}
|
||||
|
||||
export type MessageRow = {
|
||||
id: string
|
||||
run_id: string
|
||||
from_handle: string
|
||||
to_handle: string
|
||||
subject: string
|
||||
@@ -37,6 +187,7 @@ export type MessageRow = {
|
||||
|
||||
export type TaskRow = {
|
||||
id: string
|
||||
run_id: string
|
||||
parent_id: string | null
|
||||
created_by_terminal_handle: string | null
|
||||
task_title: string | null
|
||||
@@ -51,9 +202,13 @@ export type TaskRow = {
|
||||
|
||||
export type DispatchContextRow = {
|
||||
id: string
|
||||
run_id: string
|
||||
task_id: string
|
||||
assignee_handle: string | null
|
||||
assignee_pane_key: string | null
|
||||
capability_hash: string | null
|
||||
process_incarnation: string | null
|
||||
capability_revoked_at: string | null
|
||||
status: DispatchStatus
|
||||
failure_count: number
|
||||
last_failure: string | null
|
||||
@@ -65,6 +220,7 @@ export type DispatchContextRow = {
|
||||
|
||||
export type DecisionGateRow = {
|
||||
id: string
|
||||
run_id: string
|
||||
task_id: string
|
||||
question: string
|
||||
options: string
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { decodeWorkerOutputCursor, encodeWorkerOutputCursor } from './worker-output-cursor'
|
||||
|
||||
describe('worker output cursors', () => {
|
||||
it('round-trips a source-pinned cursor without exposing source details', () => {
|
||||
const cursor = encodeWorkerOutputCursor('dispatch_1', 'transcript', 'source_digest', 42)
|
||||
|
||||
expect(cursor).toMatch(/^owr1_/)
|
||||
expect(cursor).not.toContain('source_digest')
|
||||
expect(decodeWorkerOutputCursor(cursor, 'dispatch_1')).toEqual({
|
||||
source: 'transcript',
|
||||
sourceIdentity: 'source_digest',
|
||||
position: 42,
|
||||
legacy: false
|
||||
})
|
||||
})
|
||||
|
||||
it('accepts legacy numeric terminal cursors', () => {
|
||||
expect(decodeWorkerOutputCursor(0, 'dispatch_1')).toEqual({
|
||||
source: 'terminal',
|
||||
sourceIdentity: null,
|
||||
position: 0,
|
||||
legacy: true
|
||||
})
|
||||
expect(decodeWorkerOutputCursor('17', 'dispatch_1')).toMatchObject({
|
||||
source: 'terminal',
|
||||
position: 17,
|
||||
legacy: true
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects another Dispatch and malformed cursor data', () => {
|
||||
const cursor = encodeWorkerOutputCursor('dispatch_1', 'terminal', 'terminal_digest', 1)
|
||||
|
||||
expect(() => decodeWorkerOutputCursor(cursor, 'dispatch_2')).toThrow(
|
||||
expect.objectContaining({ code: 'cursor_dispatch_mismatch' })
|
||||
)
|
||||
expect(() => decodeWorkerOutputCursor('owr1_not-json', 'dispatch_1')).toThrow(
|
||||
expect.objectContaining({ code: 'cursor_invalid' })
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,122 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { OrchestrationError } from './orchestration-error'
|
||||
|
||||
const WORKER_OUTPUT_CURSOR_PREFIX = 'owr1_'
|
||||
const WORKER_OUTPUT_CURSOR_MAX_LENGTH = 2_048
|
||||
|
||||
type WorkerOutputCursorPayload = {
|
||||
v: 1
|
||||
d: string
|
||||
s: 'terminal' | 'transcript'
|
||||
i: string
|
||||
p: number
|
||||
}
|
||||
|
||||
export type DecodedWorkerOutputCursor =
|
||||
| {
|
||||
source: 'terminal'
|
||||
sourceIdentity: string | null
|
||||
position: number
|
||||
legacy: boolean
|
||||
}
|
||||
| {
|
||||
source: 'transcript'
|
||||
sourceIdentity: string
|
||||
position: number
|
||||
legacy: false
|
||||
}
|
||||
|
||||
export function createWorkerOutputSourceIdentity(fields: readonly string[]): string {
|
||||
return createHash('sha256').update(JSON.stringify(fields)).digest('base64url').slice(0, 32)
|
||||
}
|
||||
|
||||
export function encodeWorkerOutputCursor(
|
||||
dispatchId: string,
|
||||
source: WorkerOutputCursorPayload['s'],
|
||||
sourceIdentity: string,
|
||||
position: number
|
||||
): string {
|
||||
const payload: WorkerOutputCursorPayload = {
|
||||
v: 1,
|
||||
d: dispatchId,
|
||||
s: source,
|
||||
i: sourceIdentity,
|
||||
p: position
|
||||
}
|
||||
return `${WORKER_OUTPUT_CURSOR_PREFIX}${Buffer.from(JSON.stringify(payload)).toString('base64url')}`
|
||||
}
|
||||
|
||||
export function decodeWorkerOutputCursor(
|
||||
cursor: string | number | undefined,
|
||||
dispatchId: string
|
||||
): DecodedWorkerOutputCursor | null {
|
||||
if (cursor === undefined) {
|
||||
return null
|
||||
}
|
||||
if (typeof cursor === 'number') {
|
||||
return decodeLegacyTerminalCursor(cursor)
|
||||
}
|
||||
if (/^\d+$/.test(cursor)) {
|
||||
return decodeLegacyTerminalCursor(Number.parseInt(cursor, 10))
|
||||
}
|
||||
if (
|
||||
cursor.length > WORKER_OUTPUT_CURSOR_MAX_LENGTH ||
|
||||
!cursor.startsWith(WORKER_OUTPUT_CURSOR_PREFIX)
|
||||
) {
|
||||
throw invalidCursor()
|
||||
}
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(
|
||||
Buffer.from(cursor.slice(WORKER_OUTPUT_CURSOR_PREFIX.length), 'base64url').toString('utf8')
|
||||
)
|
||||
} catch {
|
||||
throw invalidCursor()
|
||||
}
|
||||
if (!isWorkerOutputCursorPayload(parsed)) {
|
||||
throw invalidCursor()
|
||||
}
|
||||
if (parsed.d !== dispatchId) {
|
||||
throw new OrchestrationError(
|
||||
'cursor_dispatch_mismatch',
|
||||
'The worker-read cursor belongs to a different Dispatch.'
|
||||
)
|
||||
}
|
||||
return {
|
||||
source: parsed.s,
|
||||
sourceIdentity: parsed.i,
|
||||
position: parsed.p,
|
||||
legacy: false
|
||||
}
|
||||
}
|
||||
|
||||
function decodeLegacyTerminalCursor(position: number): DecodedWorkerOutputCursor {
|
||||
if (!Number.isSafeInteger(position) || position < 0) {
|
||||
throw invalidCursor()
|
||||
}
|
||||
return { source: 'terminal', sourceIdentity: null, position, legacy: true }
|
||||
}
|
||||
|
||||
function isWorkerOutputCursorPayload(value: unknown): value is WorkerOutputCursorPayload {
|
||||
if (!value || typeof value !== 'object' || Array.isArray(value)) {
|
||||
return false
|
||||
}
|
||||
const payload = value as Record<string, unknown>
|
||||
return (
|
||||
payload.v === 1 &&
|
||||
typeof payload.d === 'string' &&
|
||||
payload.d.length > 0 &&
|
||||
payload.d.length <= 512 &&
|
||||
(payload.s === 'terminal' || payload.s === 'transcript') &&
|
||||
typeof payload.i === 'string' &&
|
||||
payload.i.length > 0 &&
|
||||
payload.i.length <= 128 &&
|
||||
typeof payload.p === 'number' &&
|
||||
Number.isSafeInteger(payload.p) &&
|
||||
payload.p >= 0
|
||||
)
|
||||
}
|
||||
|
||||
function invalidCursor(): OrchestrationError {
|
||||
return new OrchestrationError('cursor_invalid', 'The worker-read cursor is invalid.')
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import type { AgentStatusIpcPayload } from '../../../shared/agent-status-types'
|
||||
import { selectExactWorkerProviderSession } from './worker-provider-session'
|
||||
|
||||
function status(
|
||||
paneKey: string,
|
||||
sessionId: string,
|
||||
overrides: Partial<AgentStatusIpcPayload> = {}
|
||||
): AgentStatusIpcPayload {
|
||||
return {
|
||||
paneKey,
|
||||
connectionId: null,
|
||||
receivedAt: 200,
|
||||
stateStartedAt: 190,
|
||||
state: 'working',
|
||||
prompt: '',
|
||||
agentType: 'codex',
|
||||
providerSession: { key: 'session_id', id: sessionId },
|
||||
...overrides
|
||||
}
|
||||
}
|
||||
|
||||
describe('exact worker provider session selection', () => {
|
||||
it('selects only the current pane, connection, and observation window', () => {
|
||||
const selected = selectExactWorkerProviderSession({
|
||||
paneKey: 'tab:worker',
|
||||
processIncarnation: 'pty:incarnation',
|
||||
connectionId: 'ssh-windows',
|
||||
launchToken: undefined,
|
||||
observedAfter: 150,
|
||||
statuses: [
|
||||
status('tab:sibling', 'sibling', { connectionId: 'ssh-windows', receivedAt: 300 }),
|
||||
status('tab:worker', 'old', { connectionId: 'ssh-windows', receivedAt: 100 }),
|
||||
status('tab:worker', 'wrong-host', { connectionId: 'ssh-mac', receivedAt: 400 }),
|
||||
status('tab:worker', 'exact', { connectionId: 'ssh-windows', receivedAt: 250 })
|
||||
]
|
||||
})
|
||||
|
||||
expect(selected).toEqual({
|
||||
paneKey: 'tab:worker',
|
||||
processIncarnation: 'pty:incarnation',
|
||||
agent: 'codex',
|
||||
providerSession: { key: 'session_id', id: 'exact' },
|
||||
observedAt: 250
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects stale and provider-session-only rows', () => {
|
||||
expect(
|
||||
selectExactWorkerProviderSession({
|
||||
paneKey: 'tab:worker',
|
||||
processIncarnation: 'pty:incarnation',
|
||||
connectionId: null,
|
||||
launchToken: undefined,
|
||||
observedAfter: 300,
|
||||
statuses: [
|
||||
status('tab:worker', 'stale', { receivedAt: 200 }),
|
||||
status('tab:worker', 'identity-only', {
|
||||
receivedAt: 400,
|
||||
providerSessionOnly: true
|
||||
})
|
||||
]
|
||||
})
|
||||
).toBeNull()
|
||||
})
|
||||
|
||||
it('rejects a prior process snapshot when the launch token changed', () => {
|
||||
expect(
|
||||
selectExactWorkerProviderSession({
|
||||
paneKey: 'tab:worker',
|
||||
processIncarnation: 'pty:new-incarnation',
|
||||
connectionId: null,
|
||||
launchToken: 'launch-new',
|
||||
observedAfter: 0,
|
||||
statuses: [status('tab:worker', 'prior', { launchToken: 'launch-old' })]
|
||||
})
|
||||
).toBeNull()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,34 @@
|
||||
import type { AgentStatusIpcPayload } from '../../../shared/agent-status-types'
|
||||
import type { ExactWorkerProviderSession } from '../../../shared/orchestration-worker-output'
|
||||
|
||||
export function selectExactWorkerProviderSession(args: {
|
||||
paneKey: string
|
||||
processIncarnation: string
|
||||
connectionId: string | null | undefined
|
||||
launchToken: string | null | undefined
|
||||
observedAfter: number
|
||||
statuses: readonly AgentStatusIpcPayload[]
|
||||
}): ExactWorkerProviderSession | null {
|
||||
const status = args.statuses
|
||||
.filter(
|
||||
(entry) =>
|
||||
entry.paneKey === args.paneKey &&
|
||||
(args.connectionId === undefined || entry.connectionId === args.connectionId) &&
|
||||
(!args.launchToken || entry.launchToken === args.launchToken) &&
|
||||
entry.providerSessionOnly !== true &&
|
||||
entry.providerSession !== undefined &&
|
||||
entry.agentType !== undefined &&
|
||||
entry.receivedAt >= args.observedAfter
|
||||
)
|
||||
.sort((left, right) => right.receivedAt - left.receivedAt)[0]
|
||||
if (!status?.providerSession || !status.agentType) {
|
||||
return null
|
||||
}
|
||||
return {
|
||||
paneKey: args.paneKey,
|
||||
processIncarnation: args.processIncarnation,
|
||||
agent: status.agentType,
|
||||
providerSession: { ...status.providerSession },
|
||||
observedAt: status.receivedAt
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
boundWorkerTranscriptMessages,
|
||||
redactWorkerTerminalLines
|
||||
} from './worker-transcript-payload'
|
||||
|
||||
describe('worker transcript wire bounds', () => {
|
||||
it('clips oversized blocks and omits local image paths', () => {
|
||||
const result = boundWorkerTranscriptMessages([
|
||||
{
|
||||
id: 'message-1',
|
||||
role: 'assistant',
|
||||
timestamp: null,
|
||||
source: 'transcript',
|
||||
blocks: [
|
||||
{ type: 'text', text: 'x'.repeat(5_000) },
|
||||
{ type: 'image-ref', path: 'C:\\Users\\worker\\secret.png', alt: 'screenshot' }
|
||||
]
|
||||
}
|
||||
])
|
||||
|
||||
expect(result.messages[0]?.blocks[0]).toMatchObject({
|
||||
type: 'text',
|
||||
text: expect.stringContaining('… (truncated)')
|
||||
})
|
||||
expect(result.messages[0]?.blocks[1]).toEqual({
|
||||
type: 'image-ref',
|
||||
alt: 'screenshot'
|
||||
})
|
||||
expect(JSON.stringify(result)).not.toContain('C:\\\\Users')
|
||||
expect(result.warnings).toContain('Local image paths were omitted from transcript output.')
|
||||
})
|
||||
|
||||
it('keeps fallback identifiers stable without exposing the transcript path', () => {
|
||||
const transcriptPath = 'C:\\Users\\worker\\.codex\\session.jsonl'
|
||||
const message = {
|
||||
id: `${transcriptPath}:0000000000000042`,
|
||||
turnId: `${transcriptPath}:0000000000000001`,
|
||||
role: 'assistant' as const,
|
||||
timestamp: null,
|
||||
source: 'transcript' as const,
|
||||
blocks: [{ type: 'image-ref' as const, url: `file:///${transcriptPath}` }]
|
||||
}
|
||||
|
||||
const first = boundWorkerTranscriptMessages([message], transcriptPath)
|
||||
const second = boundWorkerTranscriptMessages([message], transcriptPath)
|
||||
|
||||
expect(first.messages).toEqual(second.messages)
|
||||
expect(first.messages[0]?.id).toMatch(/^worker-message-/)
|
||||
expect(first.messages[0]?.turnId).toMatch(/^worker-message-/)
|
||||
expect(first.messages[0]?.blocks[0]).toEqual({ type: 'image-ref' })
|
||||
expect(JSON.stringify(first)).not.toContain('Users')
|
||||
expect(first.warnings).toEqual(
|
||||
expect.arrayContaining([
|
||||
'Transcript-backed message identifiers were made opaque.',
|
||||
'Local image paths were omitted from transcript output.'
|
||||
])
|
||||
)
|
||||
})
|
||||
|
||||
it('redacts dispatch capabilities from prose and tool payloads', () => {
|
||||
const capability = `dcap_${'A'.repeat(43)}`
|
||||
const result = boundWorkerTranscriptMessages([
|
||||
{
|
||||
id: 'message-secret',
|
||||
role: 'assistant',
|
||||
timestamp: null,
|
||||
source: 'transcript',
|
||||
blocks: [
|
||||
{ type: 'text', text: `Use --dispatch-capability ${capability}` },
|
||||
{
|
||||
type: 'tool-call',
|
||||
name: 'exec_command',
|
||||
input: {
|
||||
cmd: `orca orchestration send --dispatch-capability ${capability}`,
|
||||
[capability]: 'secret key'
|
||||
}
|
||||
},
|
||||
{ type: 'tool-result', output: `echoed ${capability}` }
|
||||
]
|
||||
}
|
||||
])
|
||||
|
||||
expect(JSON.stringify(result)).not.toContain(capability)
|
||||
expect(JSON.stringify(result.messages)).toContain('[dispatch capability redacted]')
|
||||
expect(result.warnings).toContain(
|
||||
'Dispatch capability tokens were redacted from transcript output.'
|
||||
)
|
||||
})
|
||||
|
||||
it('redacts dispatch capabilities from terminal fallback lines', () => {
|
||||
const capability = `dcap_${'A'.repeat(43)}`
|
||||
|
||||
expect(redactWorkerTerminalLines([`send --dispatch-capability ${capability}`, 'safe'])).toEqual(
|
||||
{
|
||||
lines: ['send --dispatch-capability [dispatch capability redacted]', 'safe'],
|
||||
warnings: ['Dispatch capability tokens were redacted from terminal output.']
|
||||
}
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,226 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import type { NativeChatBlock, NativeChatMessage } from '../../../shared/native-chat-types'
|
||||
|
||||
export const DEFAULT_WORKER_TRANSCRIPT_MESSAGE_LIMIT = 40
|
||||
export const MAX_WORKER_TRANSCRIPT_MESSAGE_LIMIT = 50
|
||||
const MAX_WORKER_TRANSCRIPT_BLOCKS = 6
|
||||
const MAX_WORKER_TRANSCRIPT_BLOCK_CHARS = 1_200
|
||||
const MAX_WORKER_TRANSCRIPT_INPUT_ITEMS = 20
|
||||
const MAX_WORKER_TRANSCRIPT_INPUT_NODES = 100
|
||||
const MAX_WORKER_TRANSCRIPT_RESPONSE_BYTES = 512 * 1024
|
||||
const TRUNCATION_MARKER = '\n… (truncated)'
|
||||
const DISPATCH_CAPABILITY_PATTERN = /\bdcap_[A-Za-z0-9_-]{20,}\b/g
|
||||
const DISPATCH_CAPABILITY_REDACTION = '[dispatch capability redacted]'
|
||||
|
||||
export function clampWorkerTranscriptLimit(limit: number | undefined): number {
|
||||
if (!Number.isFinite(limit) || (limit ?? 0) <= 0) {
|
||||
return DEFAULT_WORKER_TRANSCRIPT_MESSAGE_LIMIT
|
||||
}
|
||||
return Math.min(Math.floor(limit!), MAX_WORKER_TRANSCRIPT_MESSAGE_LIMIT)
|
||||
}
|
||||
|
||||
export function redactWorkerTerminalLines(lines: readonly string[]): {
|
||||
lines: string[]
|
||||
warnings: string[]
|
||||
} {
|
||||
let redacted = false
|
||||
const bounded = lines.map((line) => {
|
||||
const result = replaceDispatchCapabilities(line)
|
||||
redacted ||= result.redacted
|
||||
return result.value
|
||||
})
|
||||
return {
|
||||
lines: bounded,
|
||||
warnings: redacted ? ['Dispatch capability tokens were redacted from terminal output.'] : []
|
||||
}
|
||||
}
|
||||
|
||||
export function boundWorkerTranscriptMessages(
|
||||
messages: readonly NativeChatMessage[],
|
||||
transcriptPath?: string
|
||||
): {
|
||||
messages: NativeChatMessage[]
|
||||
limited: boolean
|
||||
warnings: string[]
|
||||
} {
|
||||
const warnings = new Set<string>()
|
||||
const bounded: NativeChatMessage[] = []
|
||||
let bytes = 2
|
||||
for (const message of messages) {
|
||||
const next = boundMessage(message, transcriptPath, warnings)
|
||||
const serializedBytes = Buffer.byteLength(JSON.stringify(next), 'utf8') + 1
|
||||
if (bounded.length > 0 && bytes + serializedBytes > MAX_WORKER_TRANSCRIPT_RESPONSE_BYTES) {
|
||||
warnings.add('Transcript response was clipped to the wire-size limit.')
|
||||
return { messages: bounded, limited: true, warnings: [...warnings] }
|
||||
}
|
||||
bounded.push(next)
|
||||
bytes += serializedBytes
|
||||
}
|
||||
return { messages: bounded, limited: false, warnings: [...warnings] }
|
||||
}
|
||||
|
||||
function boundMessage(
|
||||
message: NativeChatMessage,
|
||||
transcriptPath: string | undefined,
|
||||
warnings: Set<string>
|
||||
): NativeChatMessage {
|
||||
const blocks = message.blocks.slice(0, MAX_WORKER_TRANSCRIPT_BLOCKS)
|
||||
if (blocks.length < message.blocks.length) {
|
||||
warnings.add('Some transcript blocks were omitted from oversized messages.')
|
||||
}
|
||||
return {
|
||||
...message,
|
||||
id: boundIdentifier(message.id, transcriptPath, warnings),
|
||||
...(message.turnId
|
||||
? { turnId: boundIdentifier(message.turnId, transcriptPath, warnings) }
|
||||
: {}),
|
||||
blocks: blocks.map((block) => boundBlock(block, warnings))
|
||||
}
|
||||
}
|
||||
|
||||
function boundBlock(block: NativeChatBlock, warnings: Set<string>): NativeChatBlock {
|
||||
if (block.type === 'text') {
|
||||
return { ...block, text: clipText(block.text, warnings) }
|
||||
}
|
||||
if (block.type === 'tool-result') {
|
||||
return { ...block, output: clipText(block.output, warnings) }
|
||||
}
|
||||
if (block.type === 'tool-call') {
|
||||
const budget = {
|
||||
remaining: MAX_WORKER_TRANSCRIPT_BLOCK_CHARS,
|
||||
nodes: MAX_WORKER_TRANSCRIPT_INPUT_NODES
|
||||
}
|
||||
return {
|
||||
...block,
|
||||
name: clipMetadata(block.name, warnings),
|
||||
input: boundToolInput(block.input, budget, 0, warnings)
|
||||
}
|
||||
}
|
||||
if (block.path || (block.url && isLocalFileLocator(block.url))) {
|
||||
warnings.add('Local image paths were omitted from transcript output.')
|
||||
return {
|
||||
type: 'image-ref',
|
||||
...(block.alt ? { alt: clipText(block.alt, warnings) } : {})
|
||||
}
|
||||
}
|
||||
return {
|
||||
...block,
|
||||
...(block.url ? { url: clipMetadata(block.url, warnings) } : {}),
|
||||
...(block.alt ? { alt: clipText(block.alt, warnings) } : {})
|
||||
}
|
||||
}
|
||||
|
||||
function boundIdentifier(
|
||||
value: string,
|
||||
transcriptPath: string | undefined,
|
||||
warnings: Set<string>
|
||||
): string {
|
||||
if (transcriptPath && value.includes(transcriptPath)) {
|
||||
warnings.add('Transcript-backed message identifiers were made opaque.')
|
||||
return `worker-message-${createHash('sha256').update(value).digest('base64url').slice(0, 32)}`
|
||||
}
|
||||
return clipMetadata(value, warnings)
|
||||
}
|
||||
|
||||
function isLocalFileLocator(value: string): boolean {
|
||||
return (
|
||||
/^file:/i.test(value) ||
|
||||
/^[a-z]:[\\/]/i.test(value) ||
|
||||
value.startsWith('/') ||
|
||||
value.startsWith('\\\\')
|
||||
)
|
||||
}
|
||||
|
||||
function clipMetadata(value: string, warnings: Set<string>): string {
|
||||
const redacted = redactSensitiveText(value, warnings)
|
||||
if (redacted.length <= 512) {
|
||||
return redacted
|
||||
}
|
||||
warnings.add('Oversized transcript metadata was clipped.')
|
||||
return redacted.slice(0, 512)
|
||||
}
|
||||
|
||||
function clipText(value: string, warnings: Set<string>): string {
|
||||
const redacted = redactSensitiveText(value, warnings)
|
||||
if (redacted.length <= MAX_WORKER_TRANSCRIPT_BLOCK_CHARS) {
|
||||
return redacted
|
||||
}
|
||||
warnings.add('Oversized transcript text was clipped.')
|
||||
return `${redacted.slice(0, MAX_WORKER_TRANSCRIPT_BLOCK_CHARS)}${TRUNCATION_MARKER}`
|
||||
}
|
||||
|
||||
function boundToolInput(
|
||||
value: unknown,
|
||||
budget: { remaining: number; nodes: number },
|
||||
depth: number,
|
||||
warnings: Set<string>
|
||||
): unknown {
|
||||
budget.nodes--
|
||||
if (budget.nodes < 0 || budget.remaining <= 0) {
|
||||
warnings.add('Oversized tool input was clipped.')
|
||||
return '… (truncated)'
|
||||
}
|
||||
if (typeof value === 'string') {
|
||||
const redacted = redactSensitiveText(value, warnings)
|
||||
const length = Math.min(redacted.length, budget.remaining)
|
||||
budget.remaining -= length
|
||||
if (length < redacted.length) {
|
||||
warnings.add('Oversized tool input was clipped.')
|
||||
return `${redacted.slice(0, length)}… (truncated)`
|
||||
}
|
||||
return redacted
|
||||
}
|
||||
if (!value || typeof value !== 'object') {
|
||||
return value
|
||||
}
|
||||
if (depth >= 5) {
|
||||
warnings.add('Deep tool input was clipped.')
|
||||
return '… (truncated)'
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
const result = value
|
||||
.slice(0, MAX_WORKER_TRANSCRIPT_INPUT_ITEMS)
|
||||
.map((item) => boundToolInput(item, budget, depth + 1, warnings))
|
||||
if (value.length > MAX_WORKER_TRANSCRIPT_INPUT_ITEMS) {
|
||||
warnings.add('Oversized tool input was clipped.')
|
||||
result.push('… (truncated)')
|
||||
}
|
||||
return result
|
||||
}
|
||||
const result: Record<string, unknown> = Object.create(null)
|
||||
let count = 0
|
||||
for (const [rawKey, entry] of Object.entries(value)) {
|
||||
if (count >= MAX_WORKER_TRANSCRIPT_INPUT_ITEMS || budget.remaining <= 0) {
|
||||
warnings.add('Oversized tool input was clipped.')
|
||||
result['…'] = 'truncated'
|
||||
break
|
||||
}
|
||||
const redactedKey = redactSensitiveText(rawKey, warnings)
|
||||
const key = redactedKey.slice(0, Math.min(redactedKey.length, budget.remaining, 128))
|
||||
budget.remaining -= key.length
|
||||
result[key] = boundToolInput(entry, budget, depth + 1, warnings)
|
||||
count++
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
function redactSensitiveText(value: string, warnings: Set<string>): string {
|
||||
const result = replaceDispatchCapabilities(value)
|
||||
if (!result.redacted) {
|
||||
return result.value
|
||||
}
|
||||
warnings.add('Dispatch capability tokens were redacted from transcript output.')
|
||||
return result.value
|
||||
}
|
||||
|
||||
function replaceDispatchCapabilities(value: string): { value: string; redacted: boolean } {
|
||||
DISPATCH_CAPABILITY_PATTERN.lastIndex = 0
|
||||
const redacted = DISPATCH_CAPABILITY_PATTERN.test(value)
|
||||
DISPATCH_CAPABILITY_PATTERN.lastIndex = 0
|
||||
return {
|
||||
value: redacted
|
||||
? value.replace(DISPATCH_CAPABILITY_PATTERN, DISPATCH_CAPABILITY_REDACTION)
|
||||
: value,
|
||||
redacted
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,188 @@
|
||||
import { appendFile, mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { readWorkerTranscript } from './worker-transcript-read'
|
||||
|
||||
function codexMessage(id: string, text: string): string {
|
||||
return JSON.stringify({
|
||||
timestamp: '2026-07-24T12:00:00.000Z',
|
||||
type: 'event_msg',
|
||||
payload: { id, type: 'agent_message', message: text }
|
||||
})
|
||||
}
|
||||
|
||||
function grokMessage(id: string, text: string): string {
|
||||
return JSON.stringify({
|
||||
id,
|
||||
timestamp: '2026-07-24T12:00:00.000Z',
|
||||
type: 'assistant',
|
||||
content: text
|
||||
})
|
||||
}
|
||||
|
||||
describe('worker transcript reads', () => {
|
||||
let directory: string
|
||||
let transcriptPath: string
|
||||
|
||||
beforeEach(async () => {
|
||||
directory = await mkdtemp(join(tmpdir(), 'orca-worker-transcript-'))
|
||||
transcriptPath = join(directory, 'rollout-session.jsonl')
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(directory, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
it('returns a bounded tail followed by new messages from the exact file', async () => {
|
||||
await writeFile(
|
||||
transcriptPath,
|
||||
[codexMessage('one', 'first'), codexMessage('two', 'second'), codexMessage('three', 'third')]
|
||||
.join('\n')
|
||||
.concat('\n')
|
||||
)
|
||||
|
||||
const initial = await readWorkerTranscript({
|
||||
agent: 'codex',
|
||||
sessionId: 'session-exact',
|
||||
transcriptPath,
|
||||
limit: 2
|
||||
})
|
||||
expect(initial).toMatchObject({
|
||||
ok: true,
|
||||
messages: [
|
||||
{ id: 'two', blocks: [{ type: 'text', text: 'second' }] },
|
||||
{ id: 'three', blocks: [{ type: 'text', text: 'third' }] }
|
||||
],
|
||||
limited: true
|
||||
})
|
||||
if (!initial.ok) {
|
||||
throw new Error('Expected the initial transcript page')
|
||||
}
|
||||
|
||||
await appendFile(transcriptPath, `{malformed}\n${codexMessage('four', 'fourth')}\n`)
|
||||
const appended = await readWorkerTranscript({
|
||||
agent: 'codex',
|
||||
sessionId: 'session-exact',
|
||||
transcriptPath,
|
||||
offset: initial.nextOffset,
|
||||
limit: 2
|
||||
})
|
||||
|
||||
expect(appended).toMatchObject({
|
||||
ok: true,
|
||||
messages: [{ id: 'four', blocks: [{ type: 'text', text: 'fourth' }] }],
|
||||
limited: false,
|
||||
warnings: ['1 malformed transcript record(s) were skipped.']
|
||||
})
|
||||
})
|
||||
|
||||
it('reports source changes and unsupported providers without guessing', async () => {
|
||||
await writeFile(transcriptPath, `${codexMessage('one', 'first')}\n`)
|
||||
|
||||
await expect(
|
||||
readWorkerTranscript({
|
||||
agent: 'codex',
|
||||
sessionId: 'session-exact',
|
||||
transcriptPath,
|
||||
offset: 10_000,
|
||||
limit: 2
|
||||
})
|
||||
).resolves.toMatchObject({ ok: false, reason: 'source_changed' })
|
||||
|
||||
await expect(
|
||||
readWorkerTranscript({
|
||||
agent: 'gemini',
|
||||
sessionId: 'session-other',
|
||||
transcriptPath,
|
||||
limit: 2
|
||||
})
|
||||
).resolves.toEqual({ ok: false, reason: 'provider_unsupported', warnings: [] })
|
||||
})
|
||||
|
||||
it('reuses the Native Chat Grok decoder', async () => {
|
||||
await writeFile(transcriptPath, `${grokMessage('grok-one', 'Grok structured output')}\n`)
|
||||
|
||||
await expect(
|
||||
readWorkerTranscript({
|
||||
agent: 'grok',
|
||||
sessionId: 'session-grok',
|
||||
transcriptPath,
|
||||
limit: 2
|
||||
})
|
||||
).resolves.toMatchObject({
|
||||
ok: true,
|
||||
messages: [
|
||||
{
|
||||
role: 'assistant',
|
||||
blocks: [{ type: 'text', text: 'Grok structured output' }]
|
||||
}
|
||||
]
|
||||
})
|
||||
})
|
||||
|
||||
it('makes file-position fallback IDs opaque', async () => {
|
||||
await writeFile(
|
||||
transcriptPath,
|
||||
`${JSON.stringify({
|
||||
timestamp: '2026-07-24T12:00:00.000Z',
|
||||
type: 'event_msg',
|
||||
payload: { type: 'agent_message', message: 'no provider id' }
|
||||
})}\n`
|
||||
)
|
||||
|
||||
const result = await readWorkerTranscript({
|
||||
agent: 'codex',
|
||||
sessionId: 'session-exact',
|
||||
transcriptPath,
|
||||
limit: 2
|
||||
})
|
||||
|
||||
expect(result).toMatchObject({
|
||||
ok: true,
|
||||
messages: [{ id: expect.stringMatching(/^worker-message-/) }],
|
||||
warnings: ['Transcript-backed message identifiers were made opaque.']
|
||||
})
|
||||
expect(result.ok && JSON.stringify(result.messages)).not.toContain(transcriptPath)
|
||||
})
|
||||
|
||||
it('advances past a record larger than the forward scan window', async () => {
|
||||
await writeFile(transcriptPath, 'x'.repeat(8 * 1024 * 1024 + 10))
|
||||
|
||||
const oversized = await readWorkerTranscript({
|
||||
agent: 'codex',
|
||||
sessionId: 'session-exact',
|
||||
transcriptPath,
|
||||
offset: 0,
|
||||
limit: 2
|
||||
})
|
||||
expect(oversized).toMatchObject({
|
||||
ok: true,
|
||||
messages: [],
|
||||
limited: true,
|
||||
warnings: expect.arrayContaining([
|
||||
'1 oversized transcript record(s) were skipped.',
|
||||
'Transcript scanning stopped at the bounded byte limit; continue with the cursor.'
|
||||
])
|
||||
})
|
||||
if (!oversized.ok) {
|
||||
throw new Error('Expected the oversized transcript page')
|
||||
}
|
||||
expect(oversized.nextOffset).toBe(8 * 1024 * 1024)
|
||||
|
||||
await appendFile(transcriptPath, `\n${codexMessage('after', 'after oversized')}\n`)
|
||||
const continued = await readWorkerTranscript({
|
||||
agent: 'codex',
|
||||
sessionId: 'session-exact',
|
||||
transcriptPath,
|
||||
offset: oversized.nextOffset,
|
||||
limit: 2
|
||||
})
|
||||
|
||||
expect(continued).toMatchObject({
|
||||
ok: true,
|
||||
messages: [{ id: 'after', blocks: [{ type: 'text', text: 'after oversized' }] }],
|
||||
limited: false
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,263 @@
|
||||
import { open, stat } from 'node:fs/promises'
|
||||
import type { AgentType, NativeChatMessage } from '../../../shared/native-chat-types'
|
||||
import { resolveNativeChatTranscriptAgent } from '../../../shared/native-chat-agent-support'
|
||||
import type { OrchestrationWorkerReadFallbackReason } from '../../../shared/orchestration-worker-output'
|
||||
import { resolveSessionFilePath } from '../../native-chat/session-file-resolver'
|
||||
import {
|
||||
MAX_NATIVE_CHAT_TRANSCRIPT_RECORD_BYTES,
|
||||
nativeChatLineDecoderForAgent,
|
||||
readNativeChatTranscriptTailFile,
|
||||
type NativeChatLineDecoder
|
||||
} from '../../native-chat/transcript-tail-reader'
|
||||
import { transcriptFallbackId } from '../../native-chat/transcript-fallback-id'
|
||||
import {
|
||||
boundWorkerTranscriptMessages,
|
||||
clampWorkerTranscriptLimit
|
||||
} from './worker-transcript-payload'
|
||||
|
||||
const MAX_FORWARD_TRANSCRIPT_SCAN_BYTES = 8 * 1024 * 1024
|
||||
|
||||
type WorkerTranscriptReadFailure = {
|
||||
ok: false
|
||||
reason: OrchestrationWorkerReadFallbackReason | 'source_changed'
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
type WorkerTranscriptReadSuccess = {
|
||||
ok: true
|
||||
filePath: string
|
||||
messages: NativeChatMessage[]
|
||||
nextOffset: number
|
||||
limited: boolean
|
||||
warnings: string[]
|
||||
}
|
||||
|
||||
export type WorkerTranscriptReadResult = WorkerTranscriptReadFailure | WorkerTranscriptReadSuccess
|
||||
|
||||
export async function readWorkerTranscript(args: {
|
||||
agent: AgentType
|
||||
sessionId: string
|
||||
transcriptPath?: string
|
||||
offset?: number
|
||||
limit?: number
|
||||
}): Promise<WorkerTranscriptReadResult> {
|
||||
const transcriptAgent = resolveNativeChatTranscriptAgent(args.agent)
|
||||
if (!transcriptAgent) {
|
||||
return { ok: false, reason: 'provider_unsupported', warnings: [] }
|
||||
}
|
||||
const decode = nativeChatLineDecoderForAgent(args.agent)
|
||||
if (!decode) {
|
||||
return { ok: false, reason: 'provider_unsupported', warnings: [] }
|
||||
}
|
||||
let filePath: string | null
|
||||
try {
|
||||
filePath = await resolveSessionFilePath(args.agent, args.sessionId, {
|
||||
transcriptPath: args.transcriptPath
|
||||
})
|
||||
} catch {
|
||||
return { ok: false, reason: 'transcript_unreadable', warnings: [] }
|
||||
}
|
||||
if (!filePath) {
|
||||
return { ok: false, reason: 'transcript_missing', warnings: [] }
|
||||
}
|
||||
const limit = clampWorkerTranscriptLimit(args.limit)
|
||||
try {
|
||||
const page =
|
||||
args.offset === undefined
|
||||
? await readInitialPage(filePath, limit, decode)
|
||||
: await readForwardPage(filePath, args.offset, limit, decode)
|
||||
if (!page.ok) {
|
||||
return page
|
||||
}
|
||||
const bounded = boundWorkerTranscriptMessages(page.messages, filePath)
|
||||
return {
|
||||
ok: true,
|
||||
filePath,
|
||||
messages: bounded.messages,
|
||||
nextOffset: page.nextOffset,
|
||||
limited: page.limited || bounded.limited,
|
||||
warnings: [...page.warnings, ...bounded.warnings]
|
||||
}
|
||||
} catch (error) {
|
||||
const code = (error as NodeJS.ErrnoException | null)?.code
|
||||
return {
|
||||
ok: false,
|
||||
reason:
|
||||
code === 'ENOENT'
|
||||
? 'transcript_missing'
|
||||
: code === 'EACCES' || code === 'EPERM'
|
||||
? 'transcript_unreadable'
|
||||
: 'transcript_parse_failed',
|
||||
warnings: []
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function readInitialPage(
|
||||
filePath: string,
|
||||
limit: number,
|
||||
decode: NativeChatLineDecoder
|
||||
): Promise<WorkerTranscriptReadSuccess> {
|
||||
const page = await readNativeChatTranscriptTailFile(filePath, limit, decode, false)
|
||||
return {
|
||||
ok: true,
|
||||
filePath,
|
||||
messages: page.messages,
|
||||
nextOffset: page.consumedTo,
|
||||
limited: page.hasMore,
|
||||
warnings: recordWarnings(page.malformedRecordCount, page.oversizedRecordCount)
|
||||
}
|
||||
}
|
||||
|
||||
async function readForwardPage(
|
||||
filePath: string,
|
||||
startOffset: number,
|
||||
limit: number,
|
||||
decode: NativeChatLineDecoder
|
||||
): Promise<WorkerTranscriptReadResult> {
|
||||
const fileSize = (await stat(filePath)).size
|
||||
if (startOffset > fileSize) {
|
||||
return { ok: false, reason: 'source_changed', warnings: [] }
|
||||
}
|
||||
if (startOffset === fileSize) {
|
||||
return {
|
||||
ok: true,
|
||||
filePath,
|
||||
messages: [],
|
||||
nextOffset: startOffset,
|
||||
limited: false,
|
||||
warnings: []
|
||||
}
|
||||
}
|
||||
const scanEnd = Math.min(fileSize, startOffset + MAX_FORWARD_TRANSCRIPT_SCAN_BYTES)
|
||||
const handle = await open(filePath, 'r')
|
||||
const messages: NativeChatMessage[] = []
|
||||
let pendingChunks: Buffer[] = []
|
||||
let pendingBytes = 0
|
||||
let pendingStart = startOffset
|
||||
let droppingOversizedRecord = await startsInsideRecord(handle, startOffset)
|
||||
let malformedRecordCount = 0
|
||||
let oversizedRecordCount = 0
|
||||
let nextOffset = startOffset
|
||||
try {
|
||||
const stream = handle.createReadStream({
|
||||
start: startOffset,
|
||||
end: scanEnd - 1,
|
||||
autoClose: false
|
||||
})
|
||||
let absoluteOffset = startOffset
|
||||
for await (const rawChunk of stream) {
|
||||
const chunk = Buffer.isBuffer(rawChunk) ? rawChunk : Buffer.from(rawChunk)
|
||||
let segmentStart = 0
|
||||
let newline = chunk.indexOf(0x0a)
|
||||
while (newline >= 0) {
|
||||
retainPart(chunk.subarray(segmentStart, newline))
|
||||
const lineEnd = absoluteOffset + newline + 1
|
||||
if (!droppingOversizedRecord) {
|
||||
decodeLine()
|
||||
}
|
||||
resetLine(lineEnd)
|
||||
nextOffset = lineEnd
|
||||
if (messages.length >= limit) {
|
||||
return successfulPage(lineEnd < fileSize)
|
||||
}
|
||||
segmentStart = newline + 1
|
||||
newline = chunk.indexOf(0x0a, segmentStart)
|
||||
}
|
||||
if (segmentStart < chunk.length) {
|
||||
retainPart(chunk.subarray(segmentStart))
|
||||
}
|
||||
absoluteOffset += chunk.length
|
||||
}
|
||||
if (droppingOversizedRecord) {
|
||||
nextOffset = scanEnd
|
||||
}
|
||||
return successfulPage(scanEnd < fileSize, scanEnd < fileSize)
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
|
||||
function retainPart(part: Buffer): void {
|
||||
if (droppingOversizedRecord) {
|
||||
return
|
||||
}
|
||||
pendingBytes += part.length
|
||||
if (pendingBytes > MAX_NATIVE_CHAT_TRANSCRIPT_RECORD_BYTES) {
|
||||
pendingChunks = []
|
||||
droppingOversizedRecord = true
|
||||
oversizedRecordCount++
|
||||
return
|
||||
}
|
||||
pendingChunks.push(part)
|
||||
}
|
||||
|
||||
function resetLine(nextStart: number): void {
|
||||
pendingChunks = []
|
||||
pendingBytes = 0
|
||||
droppingOversizedRecord = false
|
||||
pendingStart = nextStart
|
||||
}
|
||||
|
||||
function decodeLine(): void {
|
||||
let line = Buffer.concat(pendingChunks).toString('utf8')
|
||||
if (line.endsWith('\r')) {
|
||||
line = line.slice(0, -1)
|
||||
}
|
||||
if (!line) {
|
||||
return
|
||||
}
|
||||
try {
|
||||
JSON.parse(line)
|
||||
} catch {
|
||||
malformedRecordCount++
|
||||
return
|
||||
}
|
||||
const message = decode(line, transcriptFallbackId(filePath, pendingStart))
|
||||
if (message) {
|
||||
messages.push(message)
|
||||
}
|
||||
}
|
||||
|
||||
function successfulPage(limited: boolean, scanLimited = false): WorkerTranscriptReadSuccess {
|
||||
return {
|
||||
ok: true,
|
||||
filePath,
|
||||
messages,
|
||||
nextOffset,
|
||||
limited,
|
||||
warnings: recordWarnings(malformedRecordCount, oversizedRecordCount, scanLimited)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function startsInsideRecord(
|
||||
handle: Awaited<ReturnType<typeof open>>,
|
||||
offset: number
|
||||
): Promise<boolean> {
|
||||
if (offset === 0) {
|
||||
return false
|
||||
}
|
||||
const previousByte = Buffer.allocUnsafe(1)
|
||||
const { bytesRead } = await handle.read(previousByte, 0, 1, offset - 1)
|
||||
return bytesRead === 1 && previousByte[0] !== 0x0a
|
||||
}
|
||||
|
||||
function recordWarnings(
|
||||
malformedRecordCount = 0,
|
||||
oversizedRecordCount = 0,
|
||||
scanLimited = false
|
||||
): string[] {
|
||||
const warnings: string[] = []
|
||||
if (malformedRecordCount > 0) {
|
||||
warnings.push(`${malformedRecordCount} malformed transcript record(s) were skipped.`)
|
||||
}
|
||||
if (oversizedRecordCount > 0) {
|
||||
warnings.push(`${oversizedRecordCount} oversized transcript record(s) were skipped.`)
|
||||
}
|
||||
if (scanLimited) {
|
||||
warnings.push(
|
||||
'Transcript scanning stopped at the bounded byte limit; continue with the cursor.'
|
||||
)
|
||||
}
|
||||
return warnings
|
||||
}
|
||||
@@ -44,6 +44,9 @@ export type RpcRequest = {
|
||||
authToken: string
|
||||
method: string
|
||||
params?: unknown
|
||||
orchestrationCapability?: string
|
||||
orchestrationContractVersion?: number
|
||||
orchestrationRequestId?: string
|
||||
}
|
||||
|
||||
export type RpcContext = {
|
||||
@@ -60,6 +63,19 @@ export type RpcContext = {
|
||||
pairedDeviceId?: string
|
||||
// Why: lets handlers gate mobile payload truncation to phones only; undefined for in-process callers → treat as full-class (no clip).
|
||||
clientKind?: 'mobile' | 'runtime'
|
||||
// Why: Dispatch authority rides in the authenticated RPC envelope, never in user payload fields.
|
||||
orchestrationCapability?: string
|
||||
// Why: long-lived mutations such as ask can durably expose acceptance before their waiter settles.
|
||||
recordMutationReceipt?: (receipt: unknown) => void
|
||||
// Why: worker-start commits this identity with its starting Dispatch so crash recovery always has an inspectable operation.
|
||||
orchestrationMutation?: {
|
||||
callerFingerprint: string
|
||||
requestId: string
|
||||
method: string
|
||||
payloadHash: string
|
||||
}
|
||||
// Why: federation pins the authenticated saved-environment caller without exposing its token to handlers or storage.
|
||||
authenticatedCallerFingerprint?: string
|
||||
pairing?: PairingRpcContext
|
||||
// Why: mobile terminal traffic bypasses JSON streaming; undefined on Unix/socket and non-E2EE WebSocket paths.
|
||||
sendBinary?: (bytes: Uint8Array<ArrayBufferLike>) => boolean | void
|
||||
|
||||
@@ -2,6 +2,7 @@ import { describe, expect, it, vi } from 'vitest'
|
||||
import { z } from 'zod'
|
||||
import type { PersistedUIState } from '../../../shared/types'
|
||||
import { getDefaultUIState } from '../../../shared/constants'
|
||||
import { ORCHESTRATION_CONTRACT_VERSION } from '../../../shared/protocol-version'
|
||||
import {
|
||||
ORCA_RUNTIME_RPC_BROWSER_UI_SOURCE,
|
||||
ORCA_RUNTIME_RPC_FEATURE_INTERACTION_SOURCE_KEY
|
||||
@@ -11,7 +12,15 @@ import { defineMethod, defineStreamingMethod, type RpcRequest } from './core'
|
||||
import type { OrcaRuntimeService } from '../orca-runtime'
|
||||
|
||||
function makeRequest(method: string, params: unknown = {}): RpcRequest {
|
||||
return { id: 'req-1', authToken: 'tok', method, params }
|
||||
return {
|
||||
id: 'req-1',
|
||||
authToken: 'tok',
|
||||
method,
|
||||
params,
|
||||
...(method.startsWith('orchestration.')
|
||||
? { orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION }
|
||||
: {})
|
||||
}
|
||||
}
|
||||
|
||||
function makeRuntime(ui: PersistedUIState = getDefaultUIState()): OrcaRuntimeService {
|
||||
|
||||
@@ -15,9 +15,9 @@ import {
|
||||
type RpcRequest,
|
||||
type RpcResponse
|
||||
} from './core'
|
||||
|
||||
import type { TerminalStreamFrame } from '../../../shared/terminal-stream-protocol'
|
||||
import type { FeatureInteractionId } from '../../../shared/feature-interactions'
|
||||
import { isBrowserPaneUiRuntimeRpcParams } from '../../../shared/runtime-rpc-feature-interaction-source'
|
||||
import {
|
||||
computerErrorData,
|
||||
errorResponse,
|
||||
@@ -29,6 +29,13 @@ import {
|
||||
import { ALL_RPC_METHODS } from './methods'
|
||||
import { emulatorProbe, emulatorProbeError } from '../../emulator/emulator-probe'
|
||||
import type { OrcaRuntimeService } from '../orca-runtime'
|
||||
import {
|
||||
OrchestrationMutationExecutor,
|
||||
authenticatedCallerFingerprint,
|
||||
type DurableMutationInvocation
|
||||
} from './orchestration-mutation-executor'
|
||||
import { orchestrationMigrationFence } from './orchestration-contract-fence'
|
||||
import { getRuntimeFeatureInteractionId } from './runtime-feature-interaction'
|
||||
|
||||
export type DispatcherOptions = {
|
||||
runtime: OrcaRuntimeService
|
||||
@@ -38,10 +45,12 @@ export type DispatcherOptions = {
|
||||
export class RpcDispatcher {
|
||||
private readonly runtime: OrcaRuntimeService
|
||||
private readonly registry: RpcRegistry
|
||||
private readonly orchestrationMutations: OrchestrationMutationExecutor
|
||||
|
||||
constructor({ runtime, methods = ALL_RPC_METHODS }: DispatcherOptions) {
|
||||
this.runtime = runtime
|
||||
this.registry = buildRegistry(methods)
|
||||
this.orchestrationMutations = new OrchestrationMutationExecutor(runtime)
|
||||
}
|
||||
|
||||
async dispatch(request: RpcRequest, options?: { signal?: AbortSignal }): Promise<RpcResponse> {
|
||||
@@ -56,6 +65,11 @@ export class RpcDispatcher {
|
||||
)
|
||||
}
|
||||
|
||||
const migrationFence = orchestrationMigrationFence(request, meta)
|
||||
if (migrationFence) {
|
||||
return migrationFence
|
||||
}
|
||||
|
||||
const parsedParams = this.parseParams(request, method, meta)
|
||||
if (parsedParams.error) {
|
||||
return parsedParams.error
|
||||
@@ -78,10 +92,17 @@ export class RpcDispatcher {
|
||||
emulatorProbe(`rpc ${request.method}`, request.params)
|
||||
}
|
||||
try {
|
||||
const result = await method.handler(parsedParams.value, {
|
||||
runtime: this.runtime,
|
||||
signal: options?.signal
|
||||
})
|
||||
const invoke = (mutation?: DurableMutationInvocation) =>
|
||||
method.handler(parsedParams.value, {
|
||||
runtime: this.runtime,
|
||||
signal: options?.signal,
|
||||
requestId: request.id,
|
||||
orchestrationCapability: request.orchestrationCapability,
|
||||
authenticatedCallerFingerprint: authenticatedCallerFingerprint(request),
|
||||
recordMutationReceipt: mutation?.recordReceipt,
|
||||
orchestrationMutation: mutation?.identity
|
||||
})
|
||||
const result = await this.orchestrationMutations.run(request, parsedParams.value, invoke)
|
||||
this.recordRuntimeFeatureInteraction(request.method, result, undefined, request.params)
|
||||
return successResponse(request.id, meta, result)
|
||||
} catch (error) {
|
||||
@@ -123,6 +144,12 @@ export class RpcDispatcher {
|
||||
return
|
||||
}
|
||||
|
||||
const migrationFence = orchestrationMigrationFence(request, meta)
|
||||
if (migrationFence) {
|
||||
reply(JSON.stringify(migrationFence))
|
||||
return
|
||||
}
|
||||
|
||||
const parsedParams = this.parseParams(request, method, meta)
|
||||
if (parsedParams.error) {
|
||||
reply(JSON.stringify(parsedParams.error))
|
||||
@@ -131,18 +158,24 @@ export class RpcDispatcher {
|
||||
|
||||
if (!isStreamingMethod(method)) {
|
||||
try {
|
||||
const result = await method.handler(parsedParams.value, {
|
||||
runtime: this.runtime,
|
||||
signal: options?.signal,
|
||||
requestId: request.id,
|
||||
connectionId: options?.connectionId,
|
||||
clientId: options?.clientId,
|
||||
pairedDeviceId: options?.pairedDeviceId,
|
||||
clientKind: options?.clientKind,
|
||||
pairing: options?.pairing,
|
||||
sendBinary: options?.sendBinary,
|
||||
registerBinaryStreamHandler: options?.registerBinaryStreamHandler
|
||||
})
|
||||
const invoke = (mutation?: DurableMutationInvocation) =>
|
||||
method.handler(parsedParams.value, {
|
||||
runtime: this.runtime,
|
||||
signal: options?.signal,
|
||||
requestId: request.id,
|
||||
connectionId: options?.connectionId,
|
||||
clientId: options?.clientId,
|
||||
pairedDeviceId: options?.pairedDeviceId,
|
||||
clientKind: options?.clientKind,
|
||||
orchestrationCapability: request.orchestrationCapability,
|
||||
authenticatedCallerFingerprint: authenticatedCallerFingerprint(request),
|
||||
recordMutationReceipt: mutation?.recordReceipt,
|
||||
orchestrationMutation: mutation?.identity,
|
||||
pairing: options?.pairing,
|
||||
sendBinary: options?.sendBinary,
|
||||
registerBinaryStreamHandler: options?.registerBinaryStreamHandler
|
||||
})
|
||||
const result = await this.orchestrationMutations.run(request, parsedParams.value, invoke)
|
||||
this.recordRuntimeFeatureInteraction(request.method, result, undefined, request.params)
|
||||
reply(JSON.stringify(successResponse(request.id, meta, result)))
|
||||
} catch (error) {
|
||||
@@ -270,50 +303,3 @@ export class RpcDispatcher {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function getRuntimeFeatureInteractionId(
|
||||
method: string,
|
||||
result: unknown,
|
||||
rawParams?: unknown
|
||||
): FeatureInteractionId | null {
|
||||
if (method === 'browser.profileImportFromBrowser') {
|
||||
return hasBooleanResult(result, 'ok') ? 'cookie-import' : null
|
||||
}
|
||||
if (method === 'browser.profileClearDefaultCookies') {
|
||||
return hasBooleanResult(result, 'cleared') ? 'cookie-import' : null
|
||||
}
|
||||
if (method === 'browser.screencast.unsubscribe') {
|
||||
return null
|
||||
}
|
||||
if (method.startsWith('browser.') && isBrowserPaneUiRuntimeRpcParams(rawParams)) {
|
||||
return null
|
||||
}
|
||||
if (method.startsWith('browser.') && !method.startsWith('browser.profile')) {
|
||||
return 'agent-browser-use'
|
||||
}
|
||||
if (method.startsWith('emulator.')) {
|
||||
// Emulator commands are allowed from terminal/CLI (workspace-scoped, like other automation).
|
||||
// Return null to indicate no special feature-interaction restriction (or add 'emulator-use' later).
|
||||
return null
|
||||
}
|
||||
if (method === 'computer.permissions') {
|
||||
return 'computer-use-setup'
|
||||
}
|
||||
if (
|
||||
method.startsWith('computer.') &&
|
||||
method !== 'computer.capabilities' &&
|
||||
method !== 'computer.permissionsStatus'
|
||||
) {
|
||||
return 'computer-use'
|
||||
}
|
||||
if (method.startsWith('orchestration.')) {
|
||||
return 'agent-orchestration'
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
function hasBooleanResult(value: unknown, key: string): boolean {
|
||||
return (
|
||||
value !== null && typeof value === 'object' && (value as Record<string, unknown>)[key] === true
|
||||
)
|
||||
}
|
||||
|
||||
@@ -30,6 +30,18 @@ describe('mapRuntimeError', () => {
|
||||
})
|
||||
})
|
||||
|
||||
it.each(['remote_runtime_unavailable', 'runtime_timeout', 'invalid_runtime_response'])(
|
||||
'preserves structured remote transport failure %s',
|
||||
(code) => {
|
||||
const error = Object.assign(new Error(`Remote transport failed: ${code}`), { code })
|
||||
|
||||
expect(mapRuntimeError('req_1', { runtimeId: 'runtime-1' }, error)).toMatchObject({
|
||||
ok: false,
|
||||
error: { code, message: `Remote transport failed: ${code}` }
|
||||
})
|
||||
}
|
||||
)
|
||||
|
||||
it.each([
|
||||
['window_not_focused', 'keyboard input requires focus', 'restore-window'],
|
||||
['permission_denied', 'missing DBUS_SESSION_BUS_ADDRESS', 'permissions'],
|
||||
|
||||
@@ -60,7 +60,41 @@ const RUNTIME_PASSTHROUGH_CODES: ReadonlySet<string> = new Set([
|
||||
const COMPUTER_PASSTHROUGH_CODES: ReadonlySet<string> = new Set(Object.values(COMPUTER_ERROR_CODES))
|
||||
const LINEAR_PASSTHROUGH_CODES: ReadonlySet<string> = new Set(LINEAR_ERROR_CODES)
|
||||
const STRUCTURED_RUNTIME_PASSTHROUGH_CODES: ReadonlySet<string> = new Set([
|
||||
'worktree_id_requires_full_path'
|
||||
'worktree_id_requires_full_path',
|
||||
'run_not_found',
|
||||
'run_required',
|
||||
'stable_pane_required',
|
||||
'consumer_fenced',
|
||||
'task_not_found',
|
||||
'task_not_startable',
|
||||
'dispatch_not_found',
|
||||
'dispatch_run_mismatch',
|
||||
'dispatch_inactive',
|
||||
'worker_identity_changed',
|
||||
'cursor_invalid',
|
||||
'cursor_dispatch_mismatch',
|
||||
'source_changed',
|
||||
'transcript_required',
|
||||
'server_required',
|
||||
'worktree_not_found_on_server',
|
||||
'resource_server_mismatch',
|
||||
'peer_changed',
|
||||
'remote_runtime_unavailable',
|
||||
'runtime_timeout',
|
||||
'invalid_runtime_response',
|
||||
'capability_unsupported',
|
||||
'relay_quota_exceeded',
|
||||
'dispatch_capability_invalid',
|
||||
'agent_unconfigured',
|
||||
'terminal_worktree_mismatch',
|
||||
'request_mismatch',
|
||||
'orchestration_migration_required',
|
||||
'operation_unknown',
|
||||
'question_not_found',
|
||||
'answer_conflict',
|
||||
'stale_delivery',
|
||||
'waiter_exists',
|
||||
'invalid_argument'
|
||||
])
|
||||
|
||||
export function mapRuntimeError(id: string, meta: RpcEnvelopeMeta, error: unknown): RpcFailure {
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { ORCHESTRATION_CONTRACT_VERSION } from '../../../../shared/protocol-version'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import type { RpcRequest } from '../core'
|
||||
import { RpcDispatcher } from '../dispatcher'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
|
||||
describe('orchestration federated message targeting', () => {
|
||||
let db: OrchestrationDb | undefined
|
||||
let runtime: OrcaRuntimeService | undefined
|
||||
|
||||
afterEach(() => {
|
||||
runtime?.stopOrchestrationFederationRelay()
|
||||
db?.close()
|
||||
})
|
||||
|
||||
it('rejects explicit send and ask targets without enqueueing a relay', async () => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
runtime = new OrcaRuntimeService()
|
||||
runtime.setOrchestrationDb(db)
|
||||
const paneKey = 'tab_worker:leaf_worker'
|
||||
const processIncarnation = 'worker_epoch:pty:1'
|
||||
const dispatchId = 'ctx_remote_targeting'
|
||||
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue(paneKey)
|
||||
vi.spyOn(runtime, 'getTerminalProcessIncarnation').mockReturnValue(processIncarnation)
|
||||
db.createRemoteDispatchAttachment({
|
||||
dispatchId,
|
||||
taskId: 'task_remote_targeting',
|
||||
homePeerFingerprint: 'home_peer',
|
||||
protocolVersion: 1,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
mutationReceipt: {
|
||||
callerFingerprint: 'home_peer',
|
||||
requestId: 'attach_request',
|
||||
method: 'orchestration.federationAttachStart',
|
||||
payloadHash: 'attach_payload'
|
||||
}
|
||||
})
|
||||
const capability = db.prepareRemoteAttachmentAuthority({
|
||||
dispatchId,
|
||||
paneKey,
|
||||
processIncarnation,
|
||||
worktreeId: 'repo::remote-worktree',
|
||||
terminalHandle: 'term_remote_worker',
|
||||
setupState: 'not_applicable',
|
||||
effects: []
|
||||
})
|
||||
db.markRemoteAttachmentReady(dispatchId)
|
||||
const dispatcher = new RpcDispatcher({ runtime, methods: ORCHESTRATION_METHODS })
|
||||
const requests: RpcRequest[] = [
|
||||
request('send_to', capability, 'orchestration.send', {
|
||||
from: 'term_remote_worker',
|
||||
to: 'run:explicit',
|
||||
subject: 'Wrong explicit target'
|
||||
}),
|
||||
request('send_run', capability, 'orchestration.send', {
|
||||
from: 'term_remote_worker',
|
||||
run: 'run_explicit',
|
||||
subject: 'Wrong explicit Run'
|
||||
}),
|
||||
request('ask_to', capability, 'orchestration.ask', {
|
||||
from: 'term_remote_worker',
|
||||
to: 'run:explicit',
|
||||
question: 'Wrong explicit target?'
|
||||
}),
|
||||
request('ask_run', capability, 'orchestration.ask', {
|
||||
from: 'term_remote_worker',
|
||||
run: 'run_explicit',
|
||||
question: 'Wrong explicit Run?'
|
||||
})
|
||||
]
|
||||
|
||||
for (const item of requests) {
|
||||
await expect(dispatcher.dispatch(item)).resolves.toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
code: 'invalid_argument',
|
||||
message: 'Federated Dispatch messages route to their Run home; omit --to and --run.'
|
||||
}
|
||||
})
|
||||
}
|
||||
expect(
|
||||
db.listFederationRelay({ dispatchId, direction: 'to_home', afterSequence: 0 })
|
||||
).toHaveLength(0)
|
||||
})
|
||||
})
|
||||
|
||||
function request(
|
||||
id: string,
|
||||
capability: string,
|
||||
method: 'orchestration.send' | 'orchestration.ask',
|
||||
params: Record<string, unknown>
|
||||
): RpcRequest {
|
||||
return {
|
||||
id: `rpc_${id}`,
|
||||
authToken: 'worker-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: `request_${id}`,
|
||||
orchestrationCapability: capability,
|
||||
method,
|
||||
params
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
import { isTuiAgent } from '../../../../shared/tui-agent-config'
|
||||
import type { RuntimeStatus } from '../../../../shared/runtime-types'
|
||||
import {
|
||||
ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY,
|
||||
ORCHESTRATION_FEDERATION_CONTROL_MAIL_PROTOCOL_VERSION,
|
||||
ORCHESTRATION_FEDERATION_CONTROL_MAIL_RUNTIME_CAPABILITY,
|
||||
ORCHESTRATION_FEDERATION_RUNTIME_CAPABILITY
|
||||
} from '../../../../shared/protocol-version'
|
||||
import { orchestrationMigrationData } from '../../../../shared/orchestration-rpc-contract'
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import type { OrchestrationDb } from '../../orchestration/db'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import type { WorkerStartInput } from './orchestration-worker-start-schema'
|
||||
|
||||
export async function startFederatedWorker(args: {
|
||||
params: WorkerStartInput
|
||||
runtime: OrcaRuntimeService
|
||||
db: OrchestrationDb
|
||||
runId: string
|
||||
task: { id: string; spec: string; status: string }
|
||||
orchestrationMutation?: {
|
||||
callerFingerprint: string
|
||||
requestId: string
|
||||
method: string
|
||||
payloadHash: string
|
||||
}
|
||||
}): Promise<unknown> {
|
||||
const { params, runtime, db, task, runId, orchestrationMutation } = args
|
||||
if (!orchestrationMutation) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'Remote worker-start requires a durable retry request.'
|
||||
)
|
||||
}
|
||||
const worktree = params.worktree ?? 'current'
|
||||
if (worktree === 'current' || worktree === 'new-child') {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'--on requires an exact remote worktree selector or new-top-level.'
|
||||
)
|
||||
}
|
||||
const createsWorktree = worktree === 'new-top-level'
|
||||
validateRemoteWorkerStart(params, createsWorktree)
|
||||
|
||||
const server = runtime.resolveOrchestrationWorkerServer(params.on as string)
|
||||
const status = (await runtime.callOrchestrationWorkerServer(
|
||||
server.environmentId,
|
||||
'status.get',
|
||||
undefined,
|
||||
params.timeoutMs
|
||||
)) as RuntimeStatus
|
||||
if (!status.capabilities?.includes(ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY)) {
|
||||
throw new OrchestrationError(
|
||||
'orchestration_migration_required',
|
||||
`Connected server ${server.name} does not support the current orchestration contract. No effects were applied.`,
|
||||
orchestrationMigrationData('runtime_capability_missing')
|
||||
)
|
||||
}
|
||||
if (!status.capabilities?.includes(ORCHESTRATION_FEDERATION_RUNTIME_CAPABILITY)) {
|
||||
throw new OrchestrationError(
|
||||
'capability_unsupported',
|
||||
`Connected server ${server.name} does not support orchestration federation.`
|
||||
)
|
||||
}
|
||||
const federationProtocolVersion = status.capabilities?.includes(
|
||||
ORCHESTRATION_FEDERATION_CONTROL_MAIL_RUNTIME_CAPABILITY
|
||||
)
|
||||
? ORCHESTRATION_FEDERATION_CONTROL_MAIL_PROTOCOL_VERSION
|
||||
: 1
|
||||
|
||||
const setupDecision = createsWorktree ? (params.setup ?? 'run') : 'not_applicable'
|
||||
const started = db.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
retryOf: params.retryOf,
|
||||
startOptions: {
|
||||
on: server.environmentId,
|
||||
serverName: server.name,
|
||||
worktree,
|
||||
name: params.name ?? null,
|
||||
repo: params.repo ?? null,
|
||||
baseBranch: params.baseBranch ?? null,
|
||||
terminal: params.terminal ?? null,
|
||||
agent: params.agent ?? null,
|
||||
timeoutMs: params.timeoutMs ?? 60_000,
|
||||
setup: setupDecision,
|
||||
setupSource: createsWorktree
|
||||
? params.setup
|
||||
? 'explicit_request'
|
||||
: 'orchestration_default'
|
||||
: 'existing_worktree'
|
||||
},
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
mutationReceipt: orchestrationMutation,
|
||||
federation: {
|
||||
environmentId: server.environmentId,
|
||||
environmentName: server.name,
|
||||
peerFingerprint: server.peerFingerprint,
|
||||
protocolVersion: federationProtocolVersion
|
||||
}
|
||||
})
|
||||
db.recordWorkerStage({ dispatchId: started.dispatch.id, stage: 'remote_attach_requested' })
|
||||
try {
|
||||
const remote = (await runtime.callOrchestrationWorkerServer(
|
||||
server.environmentId,
|
||||
'orchestration.federationAttachStart',
|
||||
{
|
||||
dispatchId: started.dispatch.id,
|
||||
taskId: task.id,
|
||||
taskSpec: task.spec,
|
||||
protocolVersion: federationProtocolVersion,
|
||||
worktree,
|
||||
name: params.name,
|
||||
repo: params.repo,
|
||||
baseBranch: params.baseBranch,
|
||||
displayName: params.displayName,
|
||||
comment: params.comment,
|
||||
setup: createsWorktree ? (params.setup ?? 'run') : undefined,
|
||||
setupSource: createsWorktree
|
||||
? params.setup
|
||||
? 'explicit_request'
|
||||
: 'orchestration_default'
|
||||
: undefined,
|
||||
terminal: params.terminal,
|
||||
agent: params.agent,
|
||||
timeoutMs: params.timeoutMs,
|
||||
devMode: params.devMode
|
||||
},
|
||||
(params.timeoutMs ?? 60_000) + 15_000,
|
||||
{ orchestrationRequestId: orchestrationMutation.requestId }
|
||||
)) as RemoteStartReceipt
|
||||
if (remote.dispatchId !== started.dispatch.id) {
|
||||
throw new OrchestrationError(
|
||||
'resource_server_mismatch',
|
||||
'The worker server returned a different Dispatch attachment.'
|
||||
)
|
||||
}
|
||||
if (remote.state === 'ready' && remote.worktreeId && remote.terminalHandle) {
|
||||
db.updateFederatedDispatchResources({
|
||||
dispatchId: started.dispatch.id,
|
||||
remoteRuntimeEpoch: remote.runtimeEpoch,
|
||||
worktreeId: remote.worktreeId,
|
||||
terminalHandle: remote.terminalHandle
|
||||
})
|
||||
db.recordWorkerStage({
|
||||
dispatchId: started.dispatch.id,
|
||||
stage: 'remote_input_accepted',
|
||||
worktreeId: remote.worktreeId,
|
||||
terminalHandle: remote.terminalHandle,
|
||||
setupState: remote.setup?.state,
|
||||
effects: remote.effects,
|
||||
residualResources: remote.residualResources
|
||||
})
|
||||
const readyWorker = db.markWorkerDispatchReady(started.dispatch.id)
|
||||
runtime.ensureOrchestrationFederationRelay(runId)
|
||||
return {
|
||||
runId,
|
||||
taskId: task.id,
|
||||
dispatchId: started.dispatch.id,
|
||||
state: 'ready',
|
||||
stage: readyWorker.stage,
|
||||
server: { environmentId: server.environmentId, name: server.name },
|
||||
setup: remote.setup,
|
||||
timeoutMs: params.timeoutMs ?? 60_000,
|
||||
effects: remote.effects ?? [],
|
||||
residualResources: remote.residualResources ?? []
|
||||
}
|
||||
}
|
||||
if (remote.state === 'outcome_unknown') {
|
||||
const worker = db.markWorkerStartUnknown(
|
||||
started.dispatch.id,
|
||||
remote.failedStage ?? 'remote_attach',
|
||||
remote.lastError ?? 'The worker server reported an unknown start outcome.'
|
||||
)
|
||||
return federatedUnknownReceipt(worker, task.id, server.name)
|
||||
}
|
||||
const worker = db.failWorkerStart(
|
||||
started.dispatch.id,
|
||||
remote.failedStage ?? 'remote_attach',
|
||||
remote.lastError ?? `The worker server returned ${remote.state}.`
|
||||
)
|
||||
return {
|
||||
runId,
|
||||
taskId: task.id,
|
||||
dispatchId: started.dispatch.id,
|
||||
state: worker.state,
|
||||
stage: worker.stage,
|
||||
server: { environmentId: server.environmentId, name: server.name },
|
||||
failedStage: worker.stage,
|
||||
lastError: worker.last_error,
|
||||
setup: remote.setup,
|
||||
effects: remote.effects ?? [],
|
||||
residualResources: remote.residualResources ?? []
|
||||
}
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error)
|
||||
if (error instanceof OrchestrationError && isKnownRemoteStartFailure(error.code)) {
|
||||
const worker = db.failWorkerStart(started.dispatch.id, 'remote_attach', reason)
|
||||
return {
|
||||
runId,
|
||||
taskId: task.id,
|
||||
dispatchId: started.dispatch.id,
|
||||
state: worker.state,
|
||||
stage: worker.stage,
|
||||
server: { environmentId: server.environmentId, name: server.name },
|
||||
failedStage: worker.stage,
|
||||
lastError: worker.last_error,
|
||||
effects: [],
|
||||
residualResources: []
|
||||
}
|
||||
}
|
||||
const worker = db.markWorkerStartUnknown(started.dispatch.id, 'remote_attach', reason)
|
||||
return federatedUnknownReceipt(worker, task.id, server.name)
|
||||
}
|
||||
}
|
||||
|
||||
type RemoteStartReceipt = {
|
||||
dispatchId: string
|
||||
state: string
|
||||
runtimeEpoch: string
|
||||
worktreeId?: string
|
||||
terminalHandle?: string
|
||||
setup?: { state: string }
|
||||
effects?: unknown[]
|
||||
residualResources?: unknown[]
|
||||
failedStage?: string
|
||||
lastError?: string
|
||||
}
|
||||
|
||||
function validateRemoteWorkerStart(params: WorkerStartInput, createsWorktree: boolean): void {
|
||||
if (createsWorktree && (!params.name || !params.repo)) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'Remote new-top-level requires --name and an explicit --repo from remote discovery.'
|
||||
)
|
||||
}
|
||||
if (createsWorktree && params.terminal) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'--terminal cannot combine with remote new-worktree creation.'
|
||||
)
|
||||
}
|
||||
if (!createsWorktree && (params.name || params.repo || params.baseBranch || params.setup)) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'Creation and setup options apply only to remote new-top-level worktrees.'
|
||||
)
|
||||
}
|
||||
if (params.terminal && params.agent) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'--terminal reuses an existing agent and cannot combine with --agent.'
|
||||
)
|
||||
}
|
||||
if (!params.terminal && (!params.agent || !isTuiAgent(params.agent))) {
|
||||
throw new OrchestrationError(
|
||||
'agent_unconfigured',
|
||||
'A configured --agent is required when remote worker-start creates a terminal.'
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function isKnownRemoteStartFailure(code: string): boolean {
|
||||
return [
|
||||
'invalid_argument',
|
||||
'agent_unconfigured',
|
||||
'worktree_not_found_on_server',
|
||||
'terminal_worktree_mismatch',
|
||||
'capability_unsupported'
|
||||
].includes(code)
|
||||
}
|
||||
|
||||
function federatedUnknownReceipt(
|
||||
worker: { dispatch_id: string; state: string; stage: string; last_error: string | null },
|
||||
taskId: string,
|
||||
serverName: string
|
||||
): unknown {
|
||||
return {
|
||||
taskId,
|
||||
dispatchId: worker.dispatch_id,
|
||||
state: 'outcome_unknown',
|
||||
stage: worker.stage,
|
||||
server: { name: serverName },
|
||||
failedStage: worker.stage,
|
||||
lastError: worker.last_error,
|
||||
effects: [],
|
||||
residualResources: [],
|
||||
nextCommands: [
|
||||
`orca orchestration worker-show --dispatch ${worker.dispatch_id} --json`,
|
||||
`orca orchestration worker-abandon --dispatch ${worker.dispatch_id} --json`
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,345 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { ORCHESTRATION_CONTRACT_VERSION } from '../../../../shared/protocol-version'
|
||||
import type { RuntimeRpcResponse } from '../../../../shared/runtime-rpc-envelope'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import type { OrchestrationEnvironmentTransport } from '../../orchestration/environment-transport'
|
||||
import type { RpcRequest } from '../core'
|
||||
import { RpcDispatcher } from '../dispatcher'
|
||||
import { authenticatedCallerFingerprint } from '../orchestration-mutation-executor'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
|
||||
describe('orchestration federation control mail', () => {
|
||||
const homeToken = 'run-home-device-token'
|
||||
const workerToken = 'worker-local-token'
|
||||
const workerPeerFingerprint = 'worker-peer'
|
||||
const coordinatorPaneKey = 'tab_coord:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
|
||||
const workerPaneKey = 'tab_worker:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
|
||||
const processIncarnation = 'worker-runtime:pty:1'
|
||||
let homeDb: OrchestrationDb
|
||||
let workerDb: OrchestrationDb
|
||||
let homeRuntime: OrcaRuntimeService
|
||||
let workerRuntime: OrcaRuntimeService
|
||||
let homeDispatcher: RpcDispatcher
|
||||
let workerDispatcher: RpcDispatcher
|
||||
let dispatchId: string
|
||||
let runId: string
|
||||
|
||||
beforeEach(() => {
|
||||
workerDb = new OrchestrationDb(':memory:')
|
||||
workerRuntime = new OrcaRuntimeService()
|
||||
workerRuntime.setOrchestrationDb(workerDb)
|
||||
vi.spyOn(workerRuntime, 'getTerminalPaneKey').mockImplementation((handle) =>
|
||||
handle === 'term_worker' ? workerPaneKey : null
|
||||
)
|
||||
vi.spyOn(workerRuntime, 'getTerminalProcessIncarnation').mockImplementation((handle) =>
|
||||
handle === 'term_worker' ? processIncarnation : null
|
||||
)
|
||||
workerDispatcher = new RpcDispatcher({
|
||||
runtime: workerRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
|
||||
const transport: OrchestrationEnvironmentTransport = {
|
||||
resolve: () => ({
|
||||
environmentId: 'environment_worker',
|
||||
name: 'worker',
|
||||
peerFingerprint: workerPeerFingerprint
|
||||
}),
|
||||
call: async (_selector, method, params, _timeoutMs, envelope) => {
|
||||
if (method === 'status.get') {
|
||||
return {
|
||||
id: 'status',
|
||||
ok: true,
|
||||
result: workerRuntime.getStatus(),
|
||||
_meta: { runtimeId: workerRuntime.getRuntimeId() }
|
||||
}
|
||||
}
|
||||
const response = (await workerDispatcher.dispatch({
|
||||
id: `remote_${method}`,
|
||||
authToken: homeToken,
|
||||
method,
|
||||
params,
|
||||
orchestrationContractVersion: envelope?.orchestrationContractVersion,
|
||||
orchestrationRequestId: envelope?.orchestrationRequestId
|
||||
})) as RuntimeRpcResponse<unknown>
|
||||
return response
|
||||
}
|
||||
}
|
||||
homeDb = new OrchestrationDb(':memory:')
|
||||
homeRuntime = new OrcaRuntimeService(null, undefined, {
|
||||
orchestrationEnvironmentTransport: transport
|
||||
})
|
||||
homeRuntime.setOrchestrationDb(homeDb)
|
||||
vi.spyOn(homeRuntime, 'getTerminalPaneKey').mockImplementation((handle) =>
|
||||
handle === 'term_coord' ? coordinatorPaneKey : null
|
||||
)
|
||||
homeDispatcher = new RpcDispatcher({
|
||||
runtime: homeRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
|
||||
const run = homeDb.createRun({
|
||||
objective: 'Federated control mail',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey
|
||||
})
|
||||
runId = run.id
|
||||
const task = homeDb.createTask({ spec: 'Wait for coordinator guidance', runId })
|
||||
const started = homeDb.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
startOptions: {},
|
||||
federation: {
|
||||
environmentId: 'environment_worker',
|
||||
environmentName: 'worker',
|
||||
peerFingerprint: workerPeerFingerprint,
|
||||
protocolVersion: 2
|
||||
}
|
||||
})
|
||||
dispatchId = started.dispatch.id
|
||||
homeDb.markWorkerDispatchReady(dispatchId)
|
||||
|
||||
const homeFingerprint = authenticatedCallerFingerprint({
|
||||
id: 'home',
|
||||
authToken: homeToken,
|
||||
method: 'orchestration.federationImport'
|
||||
})
|
||||
workerDb.createRemoteDispatchAttachment({
|
||||
dispatchId,
|
||||
taskId: task.id,
|
||||
homePeerFingerprint: homeFingerprint,
|
||||
protocolVersion: 2,
|
||||
runtimeEpoch: workerRuntime.getRuntimeId(),
|
||||
mutationReceipt: {
|
||||
callerFingerprint: homeFingerprint,
|
||||
requestId: 'attach-worker',
|
||||
method: 'orchestration.federationAttachStart',
|
||||
payloadHash: 'attach-worker-payload'
|
||||
}
|
||||
})
|
||||
workerDb.prepareRemoteAttachmentAuthority({
|
||||
dispatchId,
|
||||
paneKey: workerPaneKey,
|
||||
processIncarnation,
|
||||
worktreeId: 'repo::worker',
|
||||
terminalHandle: 'term_worker',
|
||||
setupState: 'not_applicable',
|
||||
effects: []
|
||||
})
|
||||
workerDb.markRemoteAttachmentReady(dispatchId)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
homeRuntime.stopOrchestrationFederationRelay()
|
||||
homeDb.close()
|
||||
workerDb.close()
|
||||
})
|
||||
|
||||
it('routes an exact Dispatch message through the durable relay', async () => {
|
||||
vi.spyOn(homeRuntime, 'ensureOrchestrationFederationRelay').mockImplementation(() => {})
|
||||
const sent = await homeDispatcher.dispatch({
|
||||
id: 'send-control',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'send-control-request',
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_coord',
|
||||
to: `dispatch:${dispatchId}`,
|
||||
subject: 'Continue',
|
||||
body: 'Run the focused follow-up.',
|
||||
type: 'status'
|
||||
}
|
||||
})
|
||||
|
||||
expect(sent).toMatchObject({
|
||||
ok: true,
|
||||
result: { relay: { dispatchId, accepted: true } }
|
||||
})
|
||||
expect(homeDb.listPendingFederationRelay(dispatchId, 'to_worker')).toHaveLength(1)
|
||||
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
const checked = await workerDispatcher.dispatch(checkRequest('check-imported'))
|
||||
|
||||
expect(checked).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
dispatchId,
|
||||
count: 1,
|
||||
messages: [
|
||||
{
|
||||
to_handle: `dispatch:${dispatchId}`,
|
||||
subject: 'Continue',
|
||||
body: 'Run the focused follow-up.'
|
||||
}
|
||||
]
|
||||
}
|
||||
})
|
||||
expect(homeDb.listPendingFederationRelay(dispatchId, 'to_worker')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('wakes a remote worker waiter when control mail imports', async () => {
|
||||
const waiting = workerDispatcher.dispatch(checkRequest('wait-for-control', true))
|
||||
await Promise.resolve()
|
||||
|
||||
const imported = await workerDispatcher.dispatch(
|
||||
importRequest('import-control', 1, 'relay-control')
|
||||
)
|
||||
|
||||
expect(imported).toMatchObject({
|
||||
ok: true,
|
||||
result: { acknowledgedThrough: 1, imported: 1 }
|
||||
})
|
||||
await expect(waiting).resolves.toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
dispatchId,
|
||||
count: 1,
|
||||
messages: [{ id: 'relay-control', subject: 'Continue' }]
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
it('accepts a repeated import after a lost acknowledgment without duplicating mail', async () => {
|
||||
const first = await workerDispatcher.dispatch(importRequest('first-import', 1, 'relay-control'))
|
||||
const repeated = await workerDispatcher.dispatch(
|
||||
importRequest('repeated-import', 1, 'different-message-id')
|
||||
)
|
||||
|
||||
expect(first).toMatchObject({ ok: true, result: { imported: 1 } })
|
||||
expect(repeated).toMatchObject({ ok: true, result: { imported: 0 } })
|
||||
expect(workerDb.getUnreadMessages(`dispatch:${dispatchId}`)).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('does not deliver pending control mail after worker completion', async () => {
|
||||
vi.spyOn(homeRuntime, 'ensureOrchestrationFederationRelay').mockImplementation(() => {})
|
||||
await homeDispatcher.dispatch({
|
||||
id: 'send-stale-control',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'send-stale-control-request',
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_coord',
|
||||
to: `dispatch:${dispatchId}`,
|
||||
subject: 'Stale follow-up',
|
||||
body: 'This must not arrive after completion.',
|
||||
type: 'status'
|
||||
}
|
||||
})
|
||||
const waiting = workerDispatcher.dispatch(checkRequest('wait-before-completion', true, 30))
|
||||
await Promise.resolve()
|
||||
|
||||
const taskId = homeDb.getDispatchContextById(dispatchId)!.task_id
|
||||
workerDb.enqueueFederationRelay({
|
||||
dispatchId,
|
||||
direction: 'to_home',
|
||||
kind: 'worker_done',
|
||||
payload: JSON.stringify({
|
||||
from: `dispatch:${dispatchId}`,
|
||||
subject: 'Done',
|
||||
body: 'Completed before the follow-up arrived.',
|
||||
type: 'worker_done',
|
||||
priority: 'normal',
|
||||
threadId: null,
|
||||
payload: JSON.stringify({
|
||||
taskId,
|
||||
dispatchId,
|
||||
outcome: 'succeeded',
|
||||
filesModified: []
|
||||
})
|
||||
}),
|
||||
settleRemoteOutcome: 'succeeded'
|
||||
})
|
||||
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
|
||||
expect(homeDb.getWorkerDispatch(dispatchId)?.state).toBe('succeeded')
|
||||
expect(workerDb.getUnreadMessages(`dispatch:${dispatchId}`)).toHaveLength(0)
|
||||
expect(homeDb.listPendingFederationRelay(dispatchId, 'to_worker')).toHaveLength(1)
|
||||
await expect(
|
||||
workerDispatcher.dispatch(importRequest('late-direct-import', 1, 'late-control'))
|
||||
).resolves.toMatchObject({
|
||||
ok: false,
|
||||
error: { code: 'dispatch_inactive' }
|
||||
})
|
||||
await expect(waiting).resolves.toMatchObject({
|
||||
ok: true,
|
||||
result: { count: 0, timedOut: true }
|
||||
})
|
||||
})
|
||||
|
||||
it('wakes only waiters whose filter matches an imported control message', async () => {
|
||||
const escalationWaiter = workerDispatcher.dispatch(
|
||||
checkRequest('wait-escalation', true, 1_000, 'escalation')
|
||||
)
|
||||
const statusWaiter = workerDispatcher.dispatch(checkRequest('wait-status', true, 30, 'status'))
|
||||
await Promise.resolve()
|
||||
|
||||
await workerDispatcher.dispatch(
|
||||
importRequest('import-escalation', 1, 'relay-escalation', 'escalation')
|
||||
)
|
||||
|
||||
await expect(escalationWaiter).resolves.toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
count: 1,
|
||||
messages: [{ id: 'relay-escalation', type: 'escalation' }]
|
||||
}
|
||||
})
|
||||
await expect(statusWaiter).resolves.toMatchObject({
|
||||
ok: true,
|
||||
result: { count: 0, timedOut: true }
|
||||
})
|
||||
})
|
||||
|
||||
function checkRequest(id: string, wait = false, timeoutMs = 5_000, types?: string): RpcRequest {
|
||||
return {
|
||||
id,
|
||||
authToken: workerToken,
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
method: 'orchestration.check',
|
||||
params: {
|
||||
terminal: 'term_worker',
|
||||
wait,
|
||||
timeoutMs,
|
||||
types
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function importRequest(
|
||||
id: string,
|
||||
sequence: number,
|
||||
messageId: string,
|
||||
type = 'status'
|
||||
): RpcRequest {
|
||||
return {
|
||||
id,
|
||||
authToken: homeToken,
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
method: 'orchestration.federationImport',
|
||||
params: {
|
||||
dispatchId,
|
||||
items: [
|
||||
{
|
||||
dispatch_id: dispatchId,
|
||||
direction: 'to_worker',
|
||||
sequence,
|
||||
message_id: messageId,
|
||||
kind: 'control_message',
|
||||
payload: JSON.stringify({
|
||||
from: `run:${runId}`,
|
||||
subject: 'Continue',
|
||||
body: 'Run the focused follow-up.',
|
||||
type,
|
||||
priority: 'normal',
|
||||
threadId: null,
|
||||
payload: null
|
||||
})
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
@@ -0,0 +1,207 @@
|
||||
import { z } from 'zod'
|
||||
import { ORCHESTRATION_WORKER_READ_SOURCES } from '../../../../shared/orchestration-worker-output'
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import type { RemoteDispatchAttachmentRow } from '../../orchestration/types'
|
||||
import { defineMethod, type RpcMethod } from '../core'
|
||||
import { OptionalFiniteNumber, requiredString } from '../schemas'
|
||||
import { readExactWorkerOutput } from './orchestration-worker-output'
|
||||
|
||||
const FederationDispatchParams = z.object({
|
||||
dispatchId: requiredString('Missing Dispatch ID')
|
||||
})
|
||||
const FederationReadParams = FederationDispatchParams.extend({
|
||||
cursor: OptionalFiniteNumber,
|
||||
limit: OptionalFiniteNumber
|
||||
})
|
||||
const FederationOutputReadParams = FederationDispatchParams.extend({
|
||||
cursor: z.union([z.number().int().nonnegative(), z.string().min(1).max(2_048)]).optional(),
|
||||
limit: OptionalFiniteNumber,
|
||||
source: z.enum(ORCHESTRATION_WORKER_READ_SOURCES).optional()
|
||||
})
|
||||
|
||||
export const ORCHESTRATION_FEDERATION_CONTROL_METHODS: RpcMethod[] = [
|
||||
defineMethod({
|
||||
name: 'orchestration.federationShow',
|
||||
params: FederationDispatchParams,
|
||||
handler: async (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
const attachment = requireHomeAttachment(
|
||||
runtime,
|
||||
params.dispatchId,
|
||||
authenticatedCallerFingerprint
|
||||
)
|
||||
const observation = await inspectRemoteAttachment(runtime, params.dispatchId)
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
attachment: exposeRemoteAttachment(attachment),
|
||||
terminal: observation.exact ? observation.terminal : null,
|
||||
observation: { status: observation.status, exactWorker: observation.exact }
|
||||
}
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.federationRead',
|
||||
params: FederationReadParams,
|
||||
handler: async (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
requireHomeAttachment(runtime, params.dispatchId, authenticatedCallerFingerprint)
|
||||
const observation = await inspectRemoteAttachment(runtime, params.dispatchId)
|
||||
if (!observation.exact || !observation.terminal || observation.status !== 'running') {
|
||||
throw new OrchestrationError(
|
||||
'worker_identity_changed',
|
||||
`Remote Dispatch ${params.dispatchId} no longer resolves to its exact process.`
|
||||
)
|
||||
}
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
terminal: await runtime.readTerminal(observation.terminal.handle, {
|
||||
cursor: params.cursor,
|
||||
limit: params.limit
|
||||
})
|
||||
}
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.federationReadOutput',
|
||||
params: FederationOutputReadParams,
|
||||
handler: async (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
const attachment = requireHomeAttachment(
|
||||
runtime,
|
||||
params.dispatchId,
|
||||
authenticatedCallerFingerprint
|
||||
)
|
||||
const observation = await inspectRemoteAttachment(runtime, params.dispatchId)
|
||||
if (!observation.exact || !observation.terminal) {
|
||||
throw new OrchestrationError(
|
||||
'worker_identity_changed',
|
||||
`Remote Dispatch ${params.dispatchId} no longer resolves to its exact process.`
|
||||
)
|
||||
}
|
||||
const output = await readExactWorkerOutput({
|
||||
runtime,
|
||||
dispatchId: params.dispatchId,
|
||||
terminalHandle: observation.terminal.handle,
|
||||
workerState: attachment.state,
|
||||
terminalStatus: observation.status === 'exited' ? 'exited' : 'running',
|
||||
attachedAt: attachment.created_at,
|
||||
source: params.source,
|
||||
cursor: params.cursor,
|
||||
limit: params.limit
|
||||
})
|
||||
const afterRead = await inspectRemoteAttachment(runtime, params.dispatchId)
|
||||
if (!afterRead.exact) {
|
||||
throw new OrchestrationError(
|
||||
'worker_identity_changed',
|
||||
`Remote Dispatch ${params.dispatchId} changed process while output was read.`
|
||||
)
|
||||
}
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
output
|
||||
}
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.federationStop',
|
||||
params: FederationDispatchParams,
|
||||
handler: async (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
requireHomeAttachment(runtime, params.dispatchId, authenticatedCallerFingerprint)
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const begun = db.beginRemoteAttachmentStop(params.dispatchId)
|
||||
if (['succeeded', 'failed', 'stopped', 'abandoned'].includes(begun.state)) {
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
state: begun.state,
|
||||
alreadySettled: true,
|
||||
processAction: 'none'
|
||||
}
|
||||
}
|
||||
const observation = await inspectRemoteAttachment(runtime, params.dispatchId)
|
||||
if (!observation.exact || !observation.terminal) {
|
||||
const attachment = db.markRemoteAttachmentStopUnknown(
|
||||
params.dispatchId,
|
||||
`The recorded worker process is ${observation.status}; no terminal was closed.`
|
||||
)
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
state: attachment.state,
|
||||
alreadySettled: false,
|
||||
processAction: 'none',
|
||||
lastError: attachment.last_error
|
||||
}
|
||||
}
|
||||
try {
|
||||
const close = await runtime.closeTerminal(observation.terminal.handle)
|
||||
const attachment = db.settleRemoteAttachmentStop(params.dispatchId)
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
state: attachment.state,
|
||||
alreadySettled: false,
|
||||
processAction: 'closed_agent_terminal',
|
||||
close
|
||||
}
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error)
|
||||
const attachment = db.markRemoteAttachmentStopUnknown(params.dispatchId, reason)
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
state: attachment.state,
|
||||
alreadySettled: false,
|
||||
processAction: 'unknown',
|
||||
lastError: reason
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
]
|
||||
|
||||
function requireHomeAttachment(
|
||||
runtime: OrcaRuntimeService,
|
||||
dispatchId: string,
|
||||
callerFingerprint: string | undefined
|
||||
): RemoteDispatchAttachmentRow {
|
||||
const attachment = runtime.getOrchestrationDb().getRemoteDispatchAttachment(dispatchId)
|
||||
if (!attachment || attachment.home_peer_fingerprint !== callerFingerprint) {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_not_found',
|
||||
`Remote Dispatch ${dispatchId} was not found for this Run home.`
|
||||
)
|
||||
}
|
||||
return attachment
|
||||
}
|
||||
|
||||
async function inspectRemoteAttachment(runtime: OrcaRuntimeService, dispatchId: string) {
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const attachment = db.getRemoteDispatchAttachment(dispatchId)
|
||||
if (!attachment?.terminal_handle) {
|
||||
return { terminal: null, exact: false, status: 'unattached' as const }
|
||||
}
|
||||
const terminal = await runtime.showTerminal(attachment.terminal_handle).catch(() => null)
|
||||
if (!terminal) {
|
||||
return { terminal: null, exact: false, status: 'missing' as const }
|
||||
}
|
||||
const exact = db.isRemoteAttachmentProcessCurrent({
|
||||
dispatchId,
|
||||
paneKey: runtime.getTerminalPaneKey(attachment.terminal_handle),
|
||||
processIncarnation: runtime.getTerminalProcessIncarnation(attachment.terminal_handle)
|
||||
})
|
||||
return {
|
||||
terminal,
|
||||
exact,
|
||||
status: exact
|
||||
? terminal.connected === false
|
||||
? ('exited' as const)
|
||||
: ('running' as const)
|
||||
: ('identity_changed' as const)
|
||||
}
|
||||
}
|
||||
|
||||
function exposeRemoteAttachment(attachment: RemoteDispatchAttachmentRow) {
|
||||
return {
|
||||
...attachment,
|
||||
effects: JSON.parse(attachment.effects) as unknown[],
|
||||
residualResources: JSON.parse(attachment.residual_resources) as unknown[]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
appendFederationSetupEffect,
|
||||
appendFederationTerminalEffects,
|
||||
type FederationEffect
|
||||
} from './orchestration-federation-effects'
|
||||
|
||||
describe('orchestration federation effects', () => {
|
||||
it('uses exact terminal handles instead of display titles for setup identity', () => {
|
||||
const effects: FederationEffect[] = []
|
||||
|
||||
appendFederationTerminalEffects(
|
||||
effects,
|
||||
[
|
||||
{ handle: 'term_agent', title: 'Codex' },
|
||||
{ handle: 'term_configured', title: 'Setup' },
|
||||
{ handle: 'term_setup', title: 'PowerShell' }
|
||||
],
|
||||
'term_agent',
|
||||
'term_setup'
|
||||
)
|
||||
appendFederationSetupEffect(effects, {
|
||||
requested: 'run',
|
||||
effective: 'run',
|
||||
source: 'orchestration_default',
|
||||
hookFound: true,
|
||||
startupPolicy: 'start-immediately',
|
||||
state: 'running'
|
||||
})
|
||||
|
||||
expect(effects).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ id: 'term_configured', role: 'configured_tab' }),
|
||||
expect.objectContaining({ id: 'term_setup', role: 'setup' }),
|
||||
expect.objectContaining({ kind: 'setup', terminalId: 'term_setup' })
|
||||
])
|
||||
)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,79 @@
|
||||
export type FederationEffect = {
|
||||
kind: 'worktree' | 'terminal' | 'setup' | 'dispatch_input'
|
||||
action?: string
|
||||
role?: string
|
||||
id?: string
|
||||
state?: string
|
||||
tabId?: string
|
||||
leafId?: string
|
||||
requested?: string
|
||||
effective?: string
|
||||
source?: string
|
||||
hookFound?: boolean
|
||||
startupPolicy?: string
|
||||
terminalId?: string
|
||||
}
|
||||
|
||||
export function appendFederationTerminalEffects(
|
||||
effects: FederationEffect[],
|
||||
terminals: { handle: string; title: string | null; tabId?: string; leafId?: string }[],
|
||||
agentHandle: string,
|
||||
setupHandle?: string
|
||||
): void {
|
||||
for (const terminal of terminals) {
|
||||
effects.push({
|
||||
kind: 'terminal',
|
||||
role:
|
||||
terminal.handle === agentHandle
|
||||
? 'agent'
|
||||
: terminal.handle === setupHandle
|
||||
? 'setup'
|
||||
: 'configured_tab',
|
||||
action: terminal.handle === agentHandle ? 'reused_agent_terminal' : 'created',
|
||||
id: terminal.handle,
|
||||
tabId: terminal.tabId,
|
||||
leafId: terminal.leafId
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
export function appendFederationSetupEffect(
|
||||
effects: FederationEffect[],
|
||||
setup: {
|
||||
requested: string
|
||||
effective: string
|
||||
source: string
|
||||
hookFound: boolean
|
||||
startupPolicy: string
|
||||
state: string
|
||||
}
|
||||
): void {
|
||||
const setupTerminal = effects.find(
|
||||
(effect) => effect.kind === 'terminal' && effect.role === 'setup'
|
||||
)
|
||||
effects.push({
|
||||
kind: 'setup',
|
||||
action: setup.requested,
|
||||
...setup,
|
||||
terminalId: setupTerminal?.id
|
||||
})
|
||||
}
|
||||
|
||||
export function isFederationResidualEffect(effect: FederationEffect): boolean {
|
||||
return Boolean(effect.action?.startsWith('created') || effect.action === 'reused_agent_terminal')
|
||||
}
|
||||
|
||||
export function isFederationEffectUnknown(error: unknown, stage: string): boolean {
|
||||
const code =
|
||||
error && typeof error === 'object' && typeof (error as { code?: unknown }).code === 'string'
|
||||
? (error as { code: string }).code
|
||||
: ''
|
||||
if (code === 'operation_unknown') {
|
||||
return true
|
||||
}
|
||||
if (!['worktree_create', 'terminal_create', 'dispatch_input'].includes(stage)) {
|
||||
return false
|
||||
}
|
||||
const message = error instanceof Error ? error.message : String(error)
|
||||
return /connection|disconnect|timed?\s*out|runtime changed|outcome unknown/i.test(message)
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
|
||||
describe('orchestration federated folder placement', () => {
|
||||
let db: OrchestrationDb | undefined
|
||||
|
||||
afterEach(() => db?.close())
|
||||
|
||||
it('rejects a new folder workspace before accepting the remote attachment', async () => {
|
||||
db = new OrchestrationDb(':memory:')
|
||||
const runtime = new OrcaRuntimeService()
|
||||
runtime.setOrchestrationDb(db)
|
||||
vi.spyOn(runtime, 'validateOrchestrationAgentLauncher').mockImplementation(() => {})
|
||||
vi.spyOn(runtime, 'showRepo').mockResolvedValue({
|
||||
id: 'folder-repo',
|
||||
kind: 'folder'
|
||||
} as never)
|
||||
const method = ORCHESTRATION_METHODS.find(
|
||||
(candidate) => candidate.name === 'orchestration.federationAttachStart'
|
||||
)
|
||||
if (!method) {
|
||||
throw new Error('federationAttachStart method is not registered')
|
||||
}
|
||||
|
||||
await expect(
|
||||
method.handler(
|
||||
method.params!.parse({
|
||||
dispatchId: 'ctx_folder',
|
||||
taskId: 'task_folder',
|
||||
taskSpec: 'work in folder',
|
||||
protocolVersion: 1,
|
||||
worktree: 'new-top-level',
|
||||
repo: 'folder-repo',
|
||||
name: 'folder-worker',
|
||||
agent: 'codex'
|
||||
}),
|
||||
{
|
||||
runtime,
|
||||
orchestrationMutation: {
|
||||
callerFingerprint: 'home_peer',
|
||||
requestId: 'request_folder',
|
||||
method: 'orchestration.federationAttachStart',
|
||||
payloadHash: 'folder_payload'
|
||||
}
|
||||
}
|
||||
)
|
||||
).rejects.toMatchObject({
|
||||
code: 'invalid_argument',
|
||||
message:
|
||||
'Folder projects cannot create orchestration worktrees; use an exact existing folder workspace.'
|
||||
})
|
||||
expect(db.getRemoteDispatchAttachment('ctx_folder')).toBeUndefined()
|
||||
expect(db.getMutationReceipt('home_peer', 'request_folder')).toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,10 @@
|
||||
import type { RpcMethod } from '../core'
|
||||
import { ORCHESTRATION_FEDERATION_CONTROL_METHODS } from './orchestration-federation-control'
|
||||
import { ORCHESTRATION_FEDERATION_RELAY_METHODS } from './orchestration-federation-relay'
|
||||
import { ORCHESTRATION_FEDERATION_ATTACH_METHODS } from './orchestration-federation'
|
||||
|
||||
export const ORCHESTRATION_FEDERATION_METHODS: RpcMethod[] = [
|
||||
...ORCHESTRATION_FEDERATION_ATTACH_METHODS,
|
||||
...ORCHESTRATION_FEDERATION_RELAY_METHODS,
|
||||
...ORCHESTRATION_FEDERATION_CONTROL_METHODS
|
||||
]
|
||||
@@ -0,0 +1,312 @@
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { RuntimeRpcResponse } from '../../../../shared/runtime-rpc-envelope'
|
||||
import { ORCHESTRATION_CONTRACT_VERSION } from '../../../../shared/protocol-version'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import type { OrchestrationEnvironmentTransport } from '../../orchestration/environment-transport'
|
||||
import type { RpcRequest } from '../core'
|
||||
import { RpcDispatcher } from '../dispatcher'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
|
||||
describe('orchestration federated worker output', () => {
|
||||
const databases: OrchestrationDb[] = []
|
||||
let homeDb: OrchestrationDb
|
||||
let workerDb: OrchestrationDb
|
||||
let homeRuntime: OrcaRuntimeService
|
||||
let workerRuntime: OrcaRuntimeService
|
||||
let homeDispatcher: RpcDispatcher
|
||||
let workerDispatcher: RpcDispatcher
|
||||
let workerSupportsStructuredRead: boolean
|
||||
|
||||
beforeEach(() => {
|
||||
homeDb = new OrchestrationDb(':memory:')
|
||||
workerDb = new OrchestrationDb(':memory:')
|
||||
databases.push(homeDb, workerDb)
|
||||
workerRuntime = new OrcaRuntimeService()
|
||||
workerRuntime.setOrchestrationDb(workerDb)
|
||||
workerDispatcher = new RpcDispatcher({
|
||||
runtime: workerRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
workerSupportsStructuredRead = true
|
||||
const transport: OrchestrationEnvironmentTransport = {
|
||||
resolve: () => ({
|
||||
environmentId: 'environment_windows',
|
||||
name: 'windows',
|
||||
peerFingerprint: 'windows_peer_fingerprint'
|
||||
}),
|
||||
call: async (_selector, method, params, _timeoutMs, envelope) => {
|
||||
if (method === 'status.get') {
|
||||
return {
|
||||
id: 'status',
|
||||
ok: true,
|
||||
result: workerRuntime.getStatus(),
|
||||
_meta: { runtimeId: workerRuntime.getRuntimeId() }
|
||||
}
|
||||
}
|
||||
if (method === 'orchestration.federationReadOutput' && !workerSupportsStructuredRead) {
|
||||
return {
|
||||
id: `remote_${method}`,
|
||||
ok: false,
|
||||
error: { code: 'method_not_found', message: `Unknown method: ${method}` }
|
||||
}
|
||||
}
|
||||
return (await workerDispatcher.dispatch({
|
||||
id: `remote_${method}`,
|
||||
authToken: 'run-home-device-token',
|
||||
method,
|
||||
params,
|
||||
orchestrationContractVersion: envelope?.orchestrationContractVersion,
|
||||
orchestrationRequestId: envelope?.orchestrationRequestId,
|
||||
orchestrationCapability: envelope?.orchestrationCapability
|
||||
})) as RuntimeRpcResponse<unknown>
|
||||
}
|
||||
}
|
||||
homeRuntime = new OrcaRuntimeService(null, undefined, {
|
||||
orchestrationEnvironmentTransport: transport
|
||||
})
|
||||
homeRuntime.setOrchestrationDb(homeDb)
|
||||
homeDispatcher = new RpcDispatcher({
|
||||
runtime: homeRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
vi.spyOn(homeRuntime, 'getTerminalPaneKey').mockImplementation((handle) =>
|
||||
handle === 'term_coord' ? 'tab_coord:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa' : null
|
||||
)
|
||||
configureWorkerRuntime(workerRuntime)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
homeRuntime.stopOrchestrationFederationRelay()
|
||||
for (const db of databases.splice(0)) {
|
||||
db.close()
|
||||
}
|
||||
})
|
||||
|
||||
function createHomeTask() {
|
||||
const run = homeDb.createRun({
|
||||
objective: 'Mac to Windows output',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
|
||||
})
|
||||
return homeDb.createTask({ spec: 'Read Windows worker output', runId: run.id })
|
||||
}
|
||||
|
||||
function startRequest(taskId: string): RpcRequest {
|
||||
return {
|
||||
id: 'rpc_worker_start',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'request_windows_worker',
|
||||
method: 'orchestration.workerStart',
|
||||
params: {
|
||||
task: taskId,
|
||||
from: 'term_coord',
|
||||
on: 'windows',
|
||||
worktree: 'new-top-level',
|
||||
repo: 'id:windows-repo',
|
||||
name: 'windows-output',
|
||||
agent: 'codex'
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function configureWorkerRuntime(runtime: OrcaRuntimeService): void {
|
||||
vi.spyOn(runtime, 'validateOrchestrationAgentLauncher').mockImplementation(() => {})
|
||||
vi.spyOn(runtime, 'showRepo').mockResolvedValue({
|
||||
id: 'windows-repo',
|
||||
kind: 'git'
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'createManagedWorktree').mockResolvedValue({
|
||||
worktree: { id: 'repo::windows-worktree', repoId: 'repo' },
|
||||
startupTerminal: { spawned: true, handle: 'term_windows_worker' },
|
||||
setupReceipt: {
|
||||
requested: 'run',
|
||||
hookFound: false,
|
||||
startupPolicy: 'start-immediately',
|
||||
state: 'not_configured'
|
||||
}
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'listTerminals').mockResolvedValue({
|
||||
terminals: [{ handle: 'term_windows_worker', title: 'Codex' }],
|
||||
totalCount: 1,
|
||||
truncated: false
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'waitForTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
condition: 'tui-idle',
|
||||
satisfied: true,
|
||||
status: 'running',
|
||||
exitCode: null
|
||||
})
|
||||
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue(
|
||||
'tab_worker:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
|
||||
)
|
||||
vi.spyOn(runtime, 'getTerminalProcessIncarnation').mockReturnValue('windows_runtime:pty:1')
|
||||
vi.spyOn(runtime, 'getTerminalOrchestrationCliCommand').mockReturnValue('orca')
|
||||
vi.spyOn(runtime, 'sendTerminalAgentPrompt').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
accepted: true,
|
||||
bytesWritten: 1
|
||||
})
|
||||
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
worktreeId: 'repo::windows-worktree',
|
||||
status: 'running'
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'readTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
status: 'running',
|
||||
tail: ['remote output'],
|
||||
truncated: false,
|
||||
nextCursor: '1'
|
||||
})
|
||||
}
|
||||
|
||||
async function startRemoteWorker(): Promise<string> {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
return homeDb.getDispatchContext(task.id)!.id
|
||||
}
|
||||
|
||||
it('routes show and read by Dispatch without repeating the worker server', async () => {
|
||||
const dispatchId = await startRemoteWorker()
|
||||
|
||||
const shown = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_show',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerShow',
|
||||
params: { dispatch: dispatchId }
|
||||
})
|
||||
const read = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_read',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerRead',
|
||||
params: { dispatch: dispatchId, limit: 20 }
|
||||
})
|
||||
|
||||
expect(shown).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
server: { environmentId: 'environment_windows', name: 'windows' },
|
||||
observation: { status: 'running', exactWorker: true },
|
||||
terminal: { handle: 'term_windows_worker' }
|
||||
}
|
||||
})
|
||||
expect(read).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
source: 'terminal',
|
||||
fallbackReason: 'session_not_reported',
|
||||
server: { environmentId: 'environment_windows', name: 'windows' },
|
||||
terminal: { tail: ['remote output'] }
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps an opaque terminal cursor across mixed server versions', async () => {
|
||||
const dispatchId = await startRemoteWorker()
|
||||
workerSupportsStructuredRead = false
|
||||
|
||||
const automatic = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_legacy_read',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerRead',
|
||||
params: { dispatch: dispatchId }
|
||||
})
|
||||
const cursor = (automatic as { result: { cursor: string } }).result.cursor
|
||||
const continued = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_legacy_continue',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerRead',
|
||||
params: { dispatch: dispatchId, cursor }
|
||||
})
|
||||
const required = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_legacy_transcript',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerRead',
|
||||
params: { dispatch: dispatchId, source: 'transcript' }
|
||||
})
|
||||
|
||||
expect(automatic).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
source: 'terminal',
|
||||
fallbackReason: 'remote_capability_unavailable',
|
||||
terminal: { tail: ['remote output'] }
|
||||
}
|
||||
})
|
||||
expect(cursor).toMatch(/^owr1_/)
|
||||
expect(continued).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
source: 'terminal',
|
||||
fallbackReason: 'remote_capability_unavailable'
|
||||
}
|
||||
})
|
||||
expect((continued as { result: { cursor: string } }).result.cursor).toMatch(/^owr1_/)
|
||||
expect(required).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
code: 'transcript_required',
|
||||
data: { reason: 'remote_capability_unavailable' }
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
it('reads the exact transcript on the worker server without leaking its path home', async () => {
|
||||
const dispatchId = await startRemoteWorker()
|
||||
const directory = await mkdtemp(join(tmpdir(), 'orca-federated-worker-output-'))
|
||||
const transcriptPath = join(directory, 'windows-session.jsonl')
|
||||
await writeFile(
|
||||
transcriptPath,
|
||||
`${JSON.stringify({
|
||||
type: 'event_msg',
|
||||
payload: { id: 'remote-message', type: 'agent_message', message: 'Windows result' }
|
||||
})}\n`
|
||||
)
|
||||
vi.spyOn(workerRuntime, 'getExactWorkerProviderSession').mockReturnValue({
|
||||
paneKey: 'tab_worker:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb',
|
||||
processIncarnation: 'windows_runtime:pty:1',
|
||||
agent: 'codex',
|
||||
providerSession: {
|
||||
key: 'session_id',
|
||||
id: 'windows-session',
|
||||
transcriptPath
|
||||
},
|
||||
observedAt: Date.now()
|
||||
})
|
||||
|
||||
try {
|
||||
const response = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_transcript_read',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerRead',
|
||||
params: { dispatch: dispatchId }
|
||||
})
|
||||
|
||||
expect(response).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
source: 'transcript',
|
||||
provider: 'codex',
|
||||
server: { environmentId: 'environment_windows' },
|
||||
transcript: {
|
||||
messages: [
|
||||
{
|
||||
id: 'remote-message',
|
||||
blocks: [{ type: 'text', text: 'Windows result' }]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
})
|
||||
expect(JSON.stringify(response)).not.toContain(transcriptPath)
|
||||
} finally {
|
||||
await rm(directory, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,179 @@
|
||||
import { z } from 'zod'
|
||||
import { ORCHESTRATION_FEDERATION_CONTROL_MAIL_PROTOCOL_VERSION } from '../../../../shared/protocol-version'
|
||||
import { importFederatedControlMessage } from '../../orchestration/federation-control-message'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import { defineMethod, type RpcMethod } from '../core'
|
||||
import { OptionalFiniteNumber, requiredString } from '../schemas'
|
||||
|
||||
const FederationPullParams = z.object({
|
||||
dispatchId: requiredString('Missing Dispatch ID'),
|
||||
afterSequence: OptionalFiniteNumber,
|
||||
limit: OptionalFiniteNumber
|
||||
})
|
||||
|
||||
const FederationAckParams = z.object({
|
||||
dispatchId: requiredString('Missing Dispatch ID'),
|
||||
throughSequence: z.number().int().nonnegative()
|
||||
})
|
||||
|
||||
const FederationImportParams = z.object({
|
||||
dispatchId: requiredString('Missing Dispatch ID'),
|
||||
items: z.array(
|
||||
z.object({
|
||||
dispatch_id: requiredString('Missing item Dispatch ID'),
|
||||
direction: z.literal('to_worker'),
|
||||
sequence: z.number().int().positive(),
|
||||
message_id: requiredString('Missing relay message ID'),
|
||||
kind: requiredString('Missing relay kind'),
|
||||
payload: requiredString('Missing relay payload')
|
||||
})
|
||||
)
|
||||
})
|
||||
|
||||
export const ORCHESTRATION_FEDERATION_RELAY_METHODS: RpcMethod[] = [
|
||||
defineMethod({
|
||||
name: 'orchestration.federationPull',
|
||||
params: FederationPullParams,
|
||||
handler: (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
requireHomeAttachment(runtime, params.dispatchId, authenticatedCallerFingerprint)
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
items: runtime.getOrchestrationDb().listFederationRelay({
|
||||
dispatchId: params.dispatchId,
|
||||
direction: 'to_home',
|
||||
afterSequence: params.afterSequence ?? 0,
|
||||
limit: params.limit
|
||||
})
|
||||
}
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.federationAck',
|
||||
params: FederationAckParams,
|
||||
handler: (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
requireHomeAttachment(runtime, params.dispatchId, authenticatedCallerFingerprint)
|
||||
runtime.getOrchestrationDb().acknowledgeFederationRelay({
|
||||
dispatchId: params.dispatchId,
|
||||
direction: 'to_home',
|
||||
throughSequence: params.throughSequence
|
||||
})
|
||||
return { dispatchId: params.dispatchId, acknowledgedThrough: params.throughSequence }
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.federationImport',
|
||||
params: FederationImportParams,
|
||||
handler: (params, { runtime, authenticatedCallerFingerprint }) => {
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const attachment = requireHomeAttachment(
|
||||
runtime,
|
||||
params.dispatchId,
|
||||
authenticatedCallerFingerprint
|
||||
)
|
||||
let cursor = attachment.to_worker_imported_sequence
|
||||
let imported = 0
|
||||
for (const item of params.items) {
|
||||
if (item.dispatch_id !== params.dispatchId || item.sequence > cursor + 1) {
|
||||
throw new OrchestrationError(
|
||||
'operation_unknown',
|
||||
`Home relay for ${params.dispatchId} is not contiguous after sequence ${cursor}.`
|
||||
)
|
||||
}
|
||||
if (item.sequence <= cursor) {
|
||||
continue
|
||||
}
|
||||
const currentAttachment = requireHomeAttachment(
|
||||
runtime,
|
||||
params.dispatchId,
|
||||
authenticatedCallerFingerprint
|
||||
)
|
||||
if (currentAttachment.state !== 'ready') {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_inactive',
|
||||
`Remote Dispatch ${params.dispatchId} is not active.`
|
||||
)
|
||||
}
|
||||
if (item.kind === 'reply') {
|
||||
const reply = parseFederatedReply(item.payload)
|
||||
db.answerRemoteQuestion({
|
||||
messageId: reply.questionId,
|
||||
dispatchId: params.dispatchId,
|
||||
answerMessageId: reply.answerMessageId,
|
||||
body: reply.body
|
||||
})
|
||||
runtime.notifyMessageArrived(`dispatch:${params.dispatchId}`, 'status')
|
||||
} else if (item.kind === 'control_message') {
|
||||
if (
|
||||
currentAttachment.protocol_version <
|
||||
ORCHESTRATION_FEDERATION_CONTROL_MAIL_PROTOCOL_VERSION
|
||||
) {
|
||||
throw new OrchestrationError(
|
||||
'capability_unsupported',
|
||||
`Remote Dispatch ${params.dispatchId} does not support coordinator control mail.`
|
||||
)
|
||||
}
|
||||
const controlMessage = importFederatedControlMessage(db, {
|
||||
dispatchId: params.dispatchId,
|
||||
messageId: item.message_id,
|
||||
payload: item.payload
|
||||
})
|
||||
imported += controlMessage.imported ? 1 : 0
|
||||
if (controlMessage.imported) {
|
||||
runtime.notifyMessageArrived(`dispatch:${params.dispatchId}`, controlMessage.type)
|
||||
}
|
||||
} else {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
`Federated worker relay kind ${item.kind} is not supported.`
|
||||
)
|
||||
}
|
||||
cursor = item.sequence
|
||||
db.setRemoteWorkerImportSequence(params.dispatchId, cursor)
|
||||
}
|
||||
return { dispatchId: params.dispatchId, acknowledgedThrough: cursor, imported }
|
||||
}
|
||||
})
|
||||
]
|
||||
|
||||
function requireHomeAttachment(
|
||||
runtime: Parameters<RpcMethod['handler']>[1]['runtime'],
|
||||
dispatchId: string,
|
||||
callerFingerprint: string | undefined
|
||||
) {
|
||||
const attachment = runtime.getOrchestrationDb().getRemoteDispatchAttachment(dispatchId)
|
||||
if (!attachment || attachment.home_peer_fingerprint !== callerFingerprint) {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_not_found',
|
||||
`Remote Dispatch ${dispatchId} was not found for this Run home.`
|
||||
)
|
||||
}
|
||||
return attachment
|
||||
}
|
||||
|
||||
function parseFederatedReply(payload: string): {
|
||||
questionId: string
|
||||
answerMessageId: string
|
||||
body: string
|
||||
} {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(payload)
|
||||
} catch {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated reply payload is invalid JSON.')
|
||||
}
|
||||
const reply = parsed as Record<string, unknown> | null
|
||||
if (
|
||||
!reply ||
|
||||
typeof reply.questionId !== 'string' ||
|
||||
typeof reply.answerMessageId !== 'string' ||
|
||||
typeof reply.body !== 'string'
|
||||
) {
|
||||
throw new OrchestrationError('invalid_argument', 'Federated reply payload is incomplete.')
|
||||
}
|
||||
return {
|
||||
questionId: reply.questionId,
|
||||
answerMessageId: reply.answerMessageId,
|
||||
body: reply.body
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
import { monitorFederatedSetup } from './orchestration-federation-setup'
|
||||
|
||||
describe('orchestration federated setup evidence', () => {
|
||||
const databases: OrchestrationDb[] = []
|
||||
const runtimes: OrcaRuntimeService[] = []
|
||||
|
||||
afterEach(() => {
|
||||
for (const runtime of runtimes.splice(0)) {
|
||||
runtime.stopOrchestrationFederationRelay()
|
||||
}
|
||||
for (const db of databases.splice(0)) {
|
||||
db.close()
|
||||
}
|
||||
})
|
||||
|
||||
function createRuntime(): { db: OrchestrationDb; runtime: OrcaRuntimeService } {
|
||||
const db = new OrchestrationDb(':memory:')
|
||||
const runtime = new OrcaRuntimeService()
|
||||
runtime.setOrchestrationDb(db)
|
||||
databases.push(db)
|
||||
runtimes.push(runtime)
|
||||
return { db, runtime }
|
||||
}
|
||||
|
||||
it('records remote setup evidence once without changing attachment lifecycle', async () => {
|
||||
const { db, runtime } = createRuntime()
|
||||
const dispatchId = 'ctx_remote_setup'
|
||||
const effects = [
|
||||
{
|
||||
kind: 'terminal' as const,
|
||||
role: 'setup',
|
||||
action: 'created',
|
||||
id: 'term_remote_setup'
|
||||
},
|
||||
{
|
||||
kind: 'setup' as const,
|
||||
action: 'run',
|
||||
state: 'running'
|
||||
},
|
||||
{
|
||||
kind: 'dispatch_input' as const,
|
||||
role: 'agent',
|
||||
id: 'term_remote_worker',
|
||||
state: 'accepted'
|
||||
}
|
||||
]
|
||||
db.createRemoteDispatchAttachment({
|
||||
dispatchId,
|
||||
taskId: 'task_remote_setup',
|
||||
homePeerFingerprint: 'home_peer',
|
||||
protocolVersion: 1,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
mutationReceipt: {
|
||||
callerFingerprint: 'home_peer',
|
||||
requestId: 'request_remote_setup',
|
||||
method: 'orchestration.federationAttachStart',
|
||||
payloadHash: 'remote_setup_payload'
|
||||
}
|
||||
})
|
||||
db.prepareRemoteAttachmentAuthority({
|
||||
dispatchId,
|
||||
paneKey: 'tab_worker:leaf_worker',
|
||||
processIncarnation: 'worker_epoch:pty:1',
|
||||
worktreeId: 'repo::remote-worktree',
|
||||
terminalHandle: 'term_remote_worker',
|
||||
setupState: 'running',
|
||||
effects
|
||||
})
|
||||
db.markRemoteAttachmentReady(dispatchId)
|
||||
vi.spyOn(runtime, 'waitForSetupTerminalCompletion').mockResolvedValue({ exitCode: 1 })
|
||||
const monitorArgs = {
|
||||
runtime,
|
||||
db,
|
||||
dispatchId,
|
||||
worktreeId: 'repo::remote-worktree',
|
||||
terminalHandle: 'term_remote_worker',
|
||||
setup: {
|
||||
requested: 'run' as const,
|
||||
effective: 'run' as const,
|
||||
source: 'orchestration_default',
|
||||
hookFound: true,
|
||||
startupPolicy: 'start-immediately' as const,
|
||||
state: 'running' as const
|
||||
},
|
||||
effects
|
||||
}
|
||||
|
||||
monitorFederatedSetup(monitorArgs)
|
||||
monitorFederatedSetup(monitorArgs)
|
||||
|
||||
await vi.waitFor(() =>
|
||||
expect(db.getRemoteDispatchAttachment(dispatchId)).toMatchObject({
|
||||
state: 'ready',
|
||||
stage: 'input_accepted',
|
||||
setup_state: 'failed'
|
||||
})
|
||||
)
|
||||
expect(
|
||||
db.listFederationRelay({ dispatchId, direction: 'to_home', afterSequence: 0 })
|
||||
).toHaveLength(1)
|
||||
expect(JSON.parse(db.getRemoteDispatchAttachment(dispatchId)?.effects ?? '[]')).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ kind: 'dispatch_input', state: 'accepted' })
|
||||
])
|
||||
)
|
||||
})
|
||||
|
||||
it('refreshes home setup evidence without changing a ready Dispatch lifecycle', async () => {
|
||||
const { db, runtime } = createRuntime()
|
||||
const run = db.createRun({
|
||||
objective: 'Observe remote setup',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:leaf_coord'
|
||||
})
|
||||
const task = db.createTask({ spec: 'remote setup', runId: run.id })
|
||||
const started = db.createStartingWorkerDispatch({
|
||||
taskId: task.id,
|
||||
startOptions: {},
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
federation: {
|
||||
environmentId: 'environment_windows',
|
||||
environmentName: 'windows',
|
||||
peerFingerprint: 'windows_peer',
|
||||
protocolVersion: 1
|
||||
}
|
||||
})
|
||||
db.recordWorkerStage({
|
||||
dispatchId: started.dispatch.id,
|
||||
stage: 'terminal_readying',
|
||||
setupState: 'running',
|
||||
effects: [
|
||||
{ kind: 'setup', action: 'run', state: 'running' },
|
||||
{
|
||||
kind: 'dispatch_input',
|
||||
role: 'agent',
|
||||
id: 'term_remote_worker',
|
||||
state: 'accepted'
|
||||
}
|
||||
]
|
||||
})
|
||||
db.markWorkerDispatchReady(started.dispatch.id)
|
||||
vi.spyOn(runtime, 'resolveOrchestrationWorkerServer').mockReturnValue({
|
||||
environmentId: 'environment_windows',
|
||||
name: 'windows',
|
||||
peerFingerprint: 'windows_peer'
|
||||
})
|
||||
vi.spyOn(runtime, 'callOrchestrationWorkerServer').mockResolvedValue({
|
||||
runtimeEpoch: 'windows_epoch',
|
||||
attachment: {
|
||||
state: 'ready',
|
||||
stage: 'input_accepted',
|
||||
last_error: null,
|
||||
worktree_id: 'repo::remote-worktree',
|
||||
terminal_handle: 'term_remote_worker',
|
||||
setup_state: 'failed',
|
||||
effects: [
|
||||
{ kind: 'setup', action: 'run', state: 'failed' },
|
||||
{
|
||||
kind: 'dispatch_input',
|
||||
role: 'agent',
|
||||
id: 'term_remote_worker',
|
||||
state: 'accepted'
|
||||
}
|
||||
],
|
||||
residualResources: []
|
||||
},
|
||||
terminal: { handle: 'term_remote_worker', connected: true },
|
||||
observation: { status: 'running', exactWorker: true }
|
||||
})
|
||||
const workerShow = ORCHESTRATION_METHODS.find(
|
||||
(method) => method.name === 'orchestration.workerShow'
|
||||
)
|
||||
if (!workerShow) {
|
||||
throw new Error('workerShow method is not registered')
|
||||
}
|
||||
|
||||
await expect(
|
||||
workerShow.handler(workerShow.params!.parse({ dispatch: started.dispatch.id }), { runtime })
|
||||
).resolves.toMatchObject({
|
||||
worker: {
|
||||
state: 'ready',
|
||||
stage: 'input_accepted',
|
||||
setup_state: 'failed',
|
||||
effects: expect.arrayContaining([
|
||||
expect.objectContaining({ kind: 'setup', state: 'failed' }),
|
||||
expect.objectContaining({ kind: 'dispatch_input', state: 'accepted' })
|
||||
])
|
||||
}
|
||||
})
|
||||
expect(db.getTask(task.id)?.status).toBe('dispatched')
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,99 @@
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import type { OrchestrationDb } from '../../orchestration/db'
|
||||
import { applyWaitForSetupOutcome, type WorkerSetupReceipt } from './orchestration-worker-topology'
|
||||
import {
|
||||
isFederationResidualEffect,
|
||||
type FederationEffect
|
||||
} from './orchestration-federation-effects'
|
||||
|
||||
type FederationSetupStageArgs = {
|
||||
db: OrchestrationDb
|
||||
dispatchId: string
|
||||
worktreeId: string
|
||||
terminalHandle: string
|
||||
setup: WorkerSetupReceipt
|
||||
effects: FederationEffect[]
|
||||
}
|
||||
|
||||
function recordStage(args: FederationSetupStageArgs, stage: string): void {
|
||||
args.db.recordRemoteAttachmentStage({
|
||||
dispatchId: args.dispatchId,
|
||||
stage,
|
||||
worktreeId: args.worktreeId,
|
||||
terminalHandle: args.terminalHandle,
|
||||
setupState: args.setup.state,
|
||||
effects: args.effects,
|
||||
residualResources: args.effects.filter(isFederationResidualEffect)
|
||||
})
|
||||
}
|
||||
|
||||
export function persistFederatedReadinessStage(args: FederationSetupStageArgs): void {
|
||||
recordStage(args, 'terminal_readying')
|
||||
}
|
||||
|
||||
export function persistFederatedSetupSpawnFailure(args: FederationSetupStageArgs): boolean {
|
||||
if (args.setup.startupPolicy !== 'wait-for-setup' || args.setup.state !== 'spawn_failed') {
|
||||
return false
|
||||
}
|
||||
recordStage(args, 'setup_start')
|
||||
return true
|
||||
}
|
||||
|
||||
export function persistFederatedSetupWaitOutcome(
|
||||
args: FederationSetupStageArgs & { wait: { satisfied: boolean; status: string } }
|
||||
): void {
|
||||
applyWaitForSetupOutcome(args.setup, args.effects, args.wait)
|
||||
if (args.setup.startupPolicy === 'wait-for-setup') {
|
||||
recordStage(args, args.setup.state === 'failed' ? 'setup_failed' : 'setup_settled')
|
||||
}
|
||||
}
|
||||
|
||||
export function monitorFederatedSetup(
|
||||
args: FederationSetupStageArgs & { runtime: OrcaRuntimeService }
|
||||
): void {
|
||||
const setupTerminal = args.effects.find(
|
||||
(effect) => effect.kind === 'terminal' && effect.role === 'setup' && effect.id
|
||||
)
|
||||
if (
|
||||
!setupTerminal?.id ||
|
||||
args.setup.startupPolicy !== 'start-immediately' ||
|
||||
args.setup.state !== 'running'
|
||||
) {
|
||||
return
|
||||
}
|
||||
void args.runtime
|
||||
.waitForSetupTerminalCompletion(setupTerminal.id)
|
||||
.then((completion) => {
|
||||
const setupState = completion.exitCode === 0 ? 'succeeded' : 'failed'
|
||||
const effects = args.effects.map((effect) =>
|
||||
effect.kind === 'setup' ? { ...effect, state: setupState } : effect
|
||||
)
|
||||
const evidence = args.db.updateRemoteAttachmentSetupEvidence({
|
||||
dispatchId: args.dispatchId,
|
||||
setupState,
|
||||
effects
|
||||
})
|
||||
if (!evidence.changed) {
|
||||
return
|
||||
}
|
||||
args.db.enqueueFederationRelay({
|
||||
dispatchId: args.dispatchId,
|
||||
direction: 'to_home',
|
||||
kind: 'status',
|
||||
payload: JSON.stringify({
|
||||
from: `dispatch:${args.dispatchId}`,
|
||||
subject: `Setup ${setupState} for worker ${args.dispatchId}`,
|
||||
body: '',
|
||||
type: 'status',
|
||||
priority: setupState === 'failed' ? 'high' : 'normal',
|
||||
threadId: null,
|
||||
payload: JSON.stringify({
|
||||
dispatchId: args.dispatchId,
|
||||
setupState,
|
||||
terminalHandle: setupTerminal.id
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
.catch(() => undefined)
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
import type { OrchestrationDb } from '../../orchestration/db'
|
||||
import { isFederationEffectUnknown } from './orchestration-federation-effects'
|
||||
import type { WorkerSetupReceipt } from './orchestration-worker-topology'
|
||||
|
||||
export function failFederatedAttachmentWithReceipt(args: {
|
||||
db: OrchestrationDb
|
||||
dispatchId: string
|
||||
runtimeEpoch: string
|
||||
failedStage: string
|
||||
error: unknown
|
||||
setup: WorkerSetupReceipt
|
||||
}): unknown {
|
||||
const reason = args.error instanceof Error ? args.error.message : String(args.error)
|
||||
const unknown = isFederationEffectUnknown(args.error, args.failedStage)
|
||||
const attachment = args.db.failRemoteAttachment(
|
||||
args.dispatchId,
|
||||
args.failedStage,
|
||||
reason,
|
||||
unknown
|
||||
)
|
||||
return {
|
||||
dispatchId: args.dispatchId,
|
||||
state: attachment.state === 'start_unknown' ? 'outcome_unknown' : attachment.state,
|
||||
stage: attachment.stage,
|
||||
runtimeEpoch: args.runtimeEpoch,
|
||||
failedStage: args.failedStage,
|
||||
lastError: reason,
|
||||
setup: args.setup,
|
||||
effects: JSON.parse(attachment.effects) as unknown[],
|
||||
residualResources: JSON.parse(attachment.residual_resources) as unknown[]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
import { z } from 'zod'
|
||||
import { OptionalFiniteNumber, OptionalString, requiredString } from '../schemas'
|
||||
|
||||
export const FederationAttachStartParams = z.object({
|
||||
dispatchId: requiredString('Missing Dispatch ID'),
|
||||
taskId: requiredString('Missing Task ID'),
|
||||
taskSpec: requiredString('Missing Task spec'),
|
||||
protocolVersion: z.union([z.literal(1), z.literal(2)]),
|
||||
worktree: requiredString('Missing remote worktree selector'),
|
||||
name: OptionalString,
|
||||
repo: OptionalString,
|
||||
baseBranch: OptionalString,
|
||||
displayName: OptionalString,
|
||||
comment: OptionalString,
|
||||
setup: z.enum(['run', 'skip', 'inherit']).optional(),
|
||||
setupSource: z.enum(['explicit_request', 'orchestration_default']).optional(),
|
||||
terminal: OptionalString,
|
||||
agent: OptionalString,
|
||||
timeoutMs: OptionalFiniteNumber,
|
||||
devMode: z.boolean().optional()
|
||||
})
|
||||
@@ -0,0 +1,862 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { RuntimeRpcResponse } from '../../../../shared/runtime-rpc-envelope'
|
||||
import {
|
||||
ORCHESTRATION_CONTRACT_VERSION,
|
||||
ORCHESTRATION_FEDERATION_CONTROL_MAIL_RUNTIME_CAPABILITY
|
||||
} from '../../../../shared/protocol-version'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import type { OrchestrationEnvironmentTransport } from '../../orchestration/environment-transport'
|
||||
import { RpcDispatcher } from '../dispatcher'
|
||||
import type { RpcRequest } from '../core'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
|
||||
describe('orchestration federation', () => {
|
||||
const databases: OrchestrationDb[] = []
|
||||
let homeDb: OrchestrationDb
|
||||
let workerDb: OrchestrationDb
|
||||
let homeRuntime: OrcaRuntimeService
|
||||
let workerRuntime: OrcaRuntimeService
|
||||
let homeDispatcher: RpcDispatcher
|
||||
let workerDispatcher: RpcDispatcher
|
||||
let workerCapabilities: string[]
|
||||
let workerPeerFingerprint: string
|
||||
let loseNextAckResponse: boolean
|
||||
|
||||
beforeEach(() => {
|
||||
homeDb = new OrchestrationDb(':memory:')
|
||||
workerDb = new OrchestrationDb(':memory:')
|
||||
databases.push(homeDb, workerDb)
|
||||
workerRuntime = new OrcaRuntimeService()
|
||||
workerRuntime.setOrchestrationDb(workerDb)
|
||||
workerDispatcher = new RpcDispatcher({
|
||||
runtime: workerRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
workerCapabilities = [...(workerRuntime.getStatus().capabilities ?? [])]
|
||||
workerPeerFingerprint = 'windows_peer_fingerprint'
|
||||
loseNextAckResponse = false
|
||||
const transport: OrchestrationEnvironmentTransport = {
|
||||
resolve: () => ({
|
||||
environmentId: 'environment_windows',
|
||||
name: 'windows',
|
||||
peerFingerprint: workerPeerFingerprint
|
||||
}),
|
||||
call: async (_selector, method, params, _timeoutMs, envelope) => {
|
||||
if (method === 'status.get') {
|
||||
return {
|
||||
id: 'status',
|
||||
ok: true,
|
||||
result: { ...workerRuntime.getStatus(), capabilities: workerCapabilities },
|
||||
_meta: { runtimeId: workerRuntime.getRuntimeId() }
|
||||
}
|
||||
}
|
||||
const response = (await workerDispatcher.dispatch({
|
||||
id: `remote_${method}`,
|
||||
authToken: 'run-home-device-token',
|
||||
method,
|
||||
params,
|
||||
orchestrationContractVersion: envelope?.orchestrationContractVersion,
|
||||
orchestrationRequestId: envelope?.orchestrationRequestId,
|
||||
orchestrationCapability: envelope?.orchestrationCapability
|
||||
})) as RuntimeRpcResponse<unknown>
|
||||
if (method === 'orchestration.federationAck' && loseNextAckResponse) {
|
||||
loseNextAckResponse = false
|
||||
throw new Error('connection lost after acknowledgment')
|
||||
}
|
||||
return response
|
||||
}
|
||||
}
|
||||
homeRuntime = new OrcaRuntimeService(null, undefined, {
|
||||
orchestrationEnvironmentTransport: transport
|
||||
})
|
||||
homeRuntime.setOrchestrationDb(homeDb)
|
||||
homeDispatcher = new RpcDispatcher({
|
||||
runtime: homeRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
vi.spyOn(homeRuntime, 'getTerminalPaneKey').mockImplementation((handle) =>
|
||||
handle === 'term_coord' ? 'tab_coord:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa' : null
|
||||
)
|
||||
configureWorkerRuntime(workerRuntime)
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
homeRuntime.stopOrchestrationFederationRelay()
|
||||
for (const db of databases.splice(0)) {
|
||||
db.close()
|
||||
}
|
||||
})
|
||||
|
||||
function createHomeTask() {
|
||||
const run = homeDb.createRun({
|
||||
objective: 'Mac to Windows',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
|
||||
})
|
||||
return homeDb.createTask({ spec: 'Audit Windows behavior', runId: run.id })
|
||||
}
|
||||
|
||||
function startRequest(taskId: string, overrides: Record<string, unknown> = {}): RpcRequest {
|
||||
return {
|
||||
id: 'rpc_worker_start',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'request_windows_worker',
|
||||
method: 'orchestration.workerStart',
|
||||
params: {
|
||||
task: taskId,
|
||||
from: 'term_coord',
|
||||
on: 'windows',
|
||||
worktree: 'new-top-level',
|
||||
repo: 'id:windows-repo',
|
||||
name: 'windows-audit',
|
||||
agent: 'codex',
|
||||
...overrides
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function configureWorkerRuntime(runtime: OrcaRuntimeService): void {
|
||||
vi.spyOn(runtime, 'validateOrchestrationAgentLauncher').mockImplementation(() => {})
|
||||
vi.spyOn(runtime, 'showRepo').mockResolvedValue({
|
||||
id: 'windows-repo',
|
||||
kind: 'git'
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'createManagedWorktree').mockResolvedValue({
|
||||
worktree: { id: 'repo::windows-worktree', repoId: 'repo' },
|
||||
startupTerminal: { spawned: true, handle: 'term_windows_worker' },
|
||||
setupReceipt: {
|
||||
requested: 'run',
|
||||
hookFound: true,
|
||||
startupPolicy: 'start-immediately',
|
||||
state: 'running'
|
||||
}
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'listTerminals').mockResolvedValue({
|
||||
terminals: [
|
||||
{ handle: 'term_windows_worker', title: 'Codex' },
|
||||
{ handle: 'term_windows_setup', title: 'Setup' }
|
||||
],
|
||||
totalCount: 2,
|
||||
truncated: false
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'waitForTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
condition: 'tui-idle',
|
||||
satisfied: true,
|
||||
status: 'running',
|
||||
exitCode: null
|
||||
})
|
||||
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue(
|
||||
'tab_worker:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
|
||||
)
|
||||
vi.spyOn(runtime, 'getTerminalProcessIncarnation').mockReturnValue('windows_runtime:pty:1')
|
||||
vi.spyOn(runtime, 'getTerminalOrchestrationCliCommand').mockReturnValue('orca')
|
||||
vi.spyOn(runtime, 'sendTerminalAgentPrompt').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
accepted: true,
|
||||
bytesWritten: 1
|
||||
})
|
||||
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
worktreeId: 'repo::windows-worktree',
|
||||
status: 'running'
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'readTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
status: 'running',
|
||||
entries: [{ cursor: 1, text: 'remote output' }],
|
||||
nextCursor: '1',
|
||||
limited: false
|
||||
} as never)
|
||||
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
closed: true
|
||||
} as never)
|
||||
}
|
||||
|
||||
function restartWorkerRuntime(): void {
|
||||
workerRuntime = new OrcaRuntimeService()
|
||||
workerRuntime.setOrchestrationDb(workerDb)
|
||||
configureWorkerRuntime(workerRuntime)
|
||||
workerDispatcher = new RpcDispatcher({
|
||||
runtime: workerRuntime,
|
||||
methods: ORCHESTRATION_METHODS
|
||||
})
|
||||
workerCapabilities = [...(workerRuntime.getStatus().capabilities ?? [])]
|
||||
}
|
||||
|
||||
it('starts a remote worker while keeping authoritative Task state at home', async () => {
|
||||
const task = createHomeTask()
|
||||
|
||||
const response = await homeDispatcher.dispatch(startRequest(task.id))
|
||||
|
||||
expect(response).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
taskId: task.id,
|
||||
state: 'ready',
|
||||
server: { environmentId: 'environment_windows', name: 'windows' },
|
||||
setup: { source: 'orchestration_default' },
|
||||
mutation: { requestId: 'request_windows_worker' }
|
||||
}
|
||||
})
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
expect(homeDb.getTask(task.id)?.status).toBe('dispatched')
|
||||
expect(homeDb.getFederatedDispatch(dispatch.id)).toMatchObject({
|
||||
environment_id: 'environment_windows',
|
||||
environment_name: 'windows',
|
||||
peer_fingerprint: 'windows_peer_fingerprint',
|
||||
remote_worktree_id: 'repo::windows-worktree',
|
||||
remote_terminal_handle: 'term_windows_worker'
|
||||
})
|
||||
expect(workerDb.getRemoteDispatchAttachment(dispatch.id)).toMatchObject({
|
||||
task_id: task.id,
|
||||
protocol_version: 2,
|
||||
state: 'ready',
|
||||
worktree_id: 'repo::windows-worktree',
|
||||
terminal_handle: 'term_windows_worker'
|
||||
})
|
||||
expect(JSON.parse(workerDb.getRemoteDispatchAttachment(dispatch.id)?.effects ?? '[]')).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({ kind: 'dispatch_input', state: 'accepted' })
|
||||
])
|
||||
)
|
||||
expect(workerDb.listTasks()).toHaveLength(0)
|
||||
expect(workerRuntime.sendTerminalAgentPrompt).toHaveBeenCalledWith(
|
||||
'term_windows_worker',
|
||||
expect.stringContaining(`Your task ID is: ${task.id}`)
|
||||
)
|
||||
})
|
||||
|
||||
it('preserves wait-for-setup gating on the connected worker server', async () => {
|
||||
vi.mocked(workerRuntime.createManagedWorktree).mockResolvedValueOnce({
|
||||
worktree: { id: 'repo::windows-worktree', repoId: 'repo' },
|
||||
startupTerminal: { spawned: true, handle: 'term_windows_worker' },
|
||||
setupReceipt: {
|
||||
requested: 'run',
|
||||
hookFound: true,
|
||||
startupPolicy: 'wait-for-setup',
|
||||
state: 'running'
|
||||
}
|
||||
} as never)
|
||||
const task = createHomeTask()
|
||||
|
||||
const response = await homeDispatcher.dispatch(startRequest(task.id, { setup: 'run' }))
|
||||
|
||||
expect(response).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
state: 'ready',
|
||||
setup: { startupPolicy: 'wait-for-setup', state: 'succeeded' },
|
||||
effects: expect.arrayContaining([
|
||||
expect.objectContaining({ kind: 'setup', state: 'succeeded' }),
|
||||
expect.objectContaining({ kind: 'dispatch_input', state: 'accepted' })
|
||||
])
|
||||
}
|
||||
})
|
||||
expect(response).toHaveProperty('result.setup.source', 'explicit_request')
|
||||
expect(workerRuntime.sendTerminalAgentPrompt).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('fails before remote task input when wait-for-setup fails', async () => {
|
||||
vi.mocked(workerRuntime.createManagedWorktree).mockResolvedValueOnce({
|
||||
worktree: { id: 'repo::windows-worktree', repoId: 'repo' },
|
||||
startupTerminal: { spawned: true, handle: 'term_windows_worker' },
|
||||
setupReceipt: {
|
||||
requested: 'run',
|
||||
hookFound: true,
|
||||
startupPolicy: 'wait-for-setup',
|
||||
state: 'running'
|
||||
}
|
||||
} as never)
|
||||
vi.mocked(workerRuntime.waitForTerminal).mockResolvedValueOnce({
|
||||
handle: 'term_windows_worker',
|
||||
condition: 'tui-idle',
|
||||
satisfied: false,
|
||||
status: 'exited',
|
||||
exitCode: 1
|
||||
})
|
||||
const task = createHomeTask()
|
||||
|
||||
const response = await homeDispatcher.dispatch(startRequest(task.id))
|
||||
|
||||
expect(response).toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
state: 'failed',
|
||||
failedStage: 'setup_wait',
|
||||
setup: { state: 'failed' },
|
||||
effects: expect.arrayContaining([
|
||||
expect.objectContaining({ kind: 'setup', state: 'failed' })
|
||||
])
|
||||
}
|
||||
})
|
||||
expect(homeDb.getTask(task.id)?.status).toBe('failed')
|
||||
expect(workerRuntime.sendTerminalAgentPrompt).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects control mail before queueing when the worker lacks that capability', async () => {
|
||||
workerCapabilities = workerCapabilities.filter(
|
||||
(capability) => capability !== ORCHESTRATION_FEDERATION_CONTROL_MAIL_RUNTIME_CAPABILITY
|
||||
)
|
||||
const task = createHomeTask()
|
||||
const started = await homeDispatcher.dispatch(startRequest(task.id))
|
||||
expect(started).toMatchObject({ ok: true, result: { state: 'ready' } })
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
|
||||
const sent = await homeDispatcher.dispatch({
|
||||
id: 'send-control-to-old-worker',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'send-control-to-old-worker-request',
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_coord',
|
||||
to: `dispatch:${dispatch.id}`,
|
||||
subject: 'Continue',
|
||||
body: 'This worker cannot receive control mail yet.',
|
||||
type: 'status'
|
||||
}
|
||||
})
|
||||
|
||||
expect(sent).toMatchObject({
|
||||
ok: false,
|
||||
error: { code: 'capability_unsupported' }
|
||||
})
|
||||
expect(homeDb.listPendingFederationRelay(dispatch.id, 'to_worker')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('durably relays remote completion into the home Run and acknowledges it', async () => {
|
||||
const task = createHomeTask()
|
||||
const started = await homeDispatcher.dispatch(startRequest(task.id))
|
||||
expect(started.ok).toBe(true)
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
const prompt = vi.mocked(workerRuntime.sendTerminalAgentPrompt).mock.calls[0]?.[1] ?? ''
|
||||
const capability = prompt.match(/--dispatch-capability (dcap_[A-Za-z0-9_-]+)/)?.[1]
|
||||
expect(capability).toBeTruthy()
|
||||
|
||||
const sent = await workerDispatcher.dispatch({
|
||||
id: 'rpc_worker_done',
|
||||
authToken: 'worker-local-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'worker_done_request',
|
||||
orchestrationCapability: capability,
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_windows_worker',
|
||||
subject: 'Windows audit complete',
|
||||
body: 'Audited Windows behavior. Found no blocker. Nothing remains.',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded',
|
||||
filesModified: []
|
||||
})
|
||||
}
|
||||
})
|
||||
expect(sent).toMatchObject({
|
||||
ok: true,
|
||||
result: { relay: { dispatchId: dispatch.id, accepted: true } }
|
||||
})
|
||||
expect(homeDb.getTask(task.id)?.status).toBe('dispatched')
|
||||
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
|
||||
expect(homeDb.getTask(task.id)?.status).toBe('completed')
|
||||
expect(homeDb.getWorkerDispatch(dispatch.id)?.state).toBe('succeeded')
|
||||
expect(homeDb.getRunMailboxHistory(task.run_id, 10)).toEqual(
|
||||
expect.arrayContaining([
|
||||
expect.objectContaining({
|
||||
id: expect.stringMatching(/^relay_/),
|
||||
type: 'worker_done',
|
||||
subject: 'Windows audit complete'
|
||||
})
|
||||
])
|
||||
)
|
||||
expect(
|
||||
workerDb.listFederationRelay({
|
||||
dispatchId: dispatch.id,
|
||||
direction: 'to_home',
|
||||
afterSequence: 0
|
||||
})[0]
|
||||
).toMatchObject({ acked_at: expect.any(String) })
|
||||
})
|
||||
|
||||
it('relays a worker question home and the coordinator answer back', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
const prompt = vi.mocked(workerRuntime.sendTerminalAgentPrompt).mock.calls[0]?.[1] ?? ''
|
||||
const capability = prompt.match(/--dispatch-capability (dcap_[A-Za-z0-9_-]+)/)?.[1]
|
||||
const ask = workerDispatcher.dispatch({
|
||||
id: 'rpc_remote_ask',
|
||||
authToken: 'worker-local-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'remote_question_request',
|
||||
orchestrationCapability: capability,
|
||||
method: 'orchestration.ask',
|
||||
params: {
|
||||
from: 'term_windows_worker',
|
||||
question: 'Should I include slow integration tests?',
|
||||
options: 'yes,no',
|
||||
timeoutMs: 60_000
|
||||
}
|
||||
})
|
||||
await vi.waitFor(() =>
|
||||
expect(
|
||||
workerDb.listFederationRelay({
|
||||
dispatchId: dispatch.id,
|
||||
direction: 'to_home',
|
||||
afterSequence: 0
|
||||
})
|
||||
).toHaveLength(1)
|
||||
)
|
||||
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
const question = homeDb
|
||||
.getRunMailboxHistory(task.run_id, 10)
|
||||
.find((message) => message.type === 'question')
|
||||
expect(question).toMatchObject({
|
||||
body: 'Should I include slow integration tests?'
|
||||
})
|
||||
|
||||
const reply = await homeDispatcher.dispatch({
|
||||
id: 'rpc_home_reply',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'home_reply_request',
|
||||
method: 'orchestration.reply',
|
||||
params: {
|
||||
id: question!.id,
|
||||
body: 'yes',
|
||||
from: 'term_coord'
|
||||
}
|
||||
})
|
||||
expect(reply).toMatchObject({ ok: true, result: { question: { status: 'answered' } } })
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
|
||||
await expect(ask).resolves.toMatchObject({
|
||||
ok: true,
|
||||
result: {
|
||||
answer: 'yes',
|
||||
messageId: question!.id,
|
||||
timedOut: false
|
||||
}
|
||||
})
|
||||
expect(
|
||||
homeDb.listFederationRelay({
|
||||
dispatchId: dispatch.id,
|
||||
direction: 'to_worker',
|
||||
afterSequence: 0
|
||||
})[0]
|
||||
).toMatchObject({ acked_at: expect.any(String) })
|
||||
})
|
||||
|
||||
it('keeps a timed-out remote question resumable', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const prompt = vi.mocked(workerRuntime.sendTerminalAgentPrompt).mock.calls[0]?.[1] ?? ''
|
||||
const capability = prompt.match(/--dispatch-capability (dcap_[A-Za-z0-9_-]+)/)?.[1]
|
||||
const timedOut = await workerDispatcher.dispatch({
|
||||
id: 'rpc_remote_ask_timeout',
|
||||
authToken: 'worker-local-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'remote_question_timeout_request',
|
||||
orchestrationCapability: capability,
|
||||
method: 'orchestration.ask',
|
||||
params: {
|
||||
from: 'term_windows_worker',
|
||||
question: 'Resume this later?',
|
||||
timeoutMs: 1
|
||||
}
|
||||
})
|
||||
expect(timedOut).toMatchObject({
|
||||
ok: true,
|
||||
result: { timedOut: true, messageId: expect.stringMatching(/^relay_/) }
|
||||
})
|
||||
const questionId = (timedOut as { result: { messageId: string } }).result.messageId
|
||||
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
await homeDispatcher.dispatch({
|
||||
id: 'rpc_home_late_reply',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'home_late_reply_request',
|
||||
method: 'orchestration.reply',
|
||||
params: { id: questionId, body: 'yes', from: 'term_coord' }
|
||||
})
|
||||
restartWorkerRuntime()
|
||||
const resumed = workerDispatcher.dispatch({
|
||||
id: 'rpc_remote_ask_resume',
|
||||
authToken: 'worker-local-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'remote_question_resume_request',
|
||||
orchestrationCapability: capability,
|
||||
method: 'orchestration.ask',
|
||||
params: { from: 'term_windows_worker', resume: questionId, timeoutMs: 5_000 }
|
||||
})
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
|
||||
await expect(resumed).resolves.toMatchObject({
|
||||
ok: true,
|
||||
result: { answer: 'yes', messageId: questionId, timedOut: false }
|
||||
})
|
||||
})
|
||||
|
||||
it('retries a lost relay acknowledgment without duplicating the home message', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
const prompt = vi.mocked(workerRuntime.sendTerminalAgentPrompt).mock.calls[0]?.[1] ?? ''
|
||||
const capability = prompt.match(/--dispatch-capability (dcap_[A-Za-z0-9_-]+)/)?.[1]
|
||||
await workerDispatcher.dispatch({
|
||||
id: 'rpc_remote_status',
|
||||
authToken: 'worker-local-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'remote_status_request',
|
||||
orchestrationCapability: capability,
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_windows_worker',
|
||||
subject: 'Checkpoint',
|
||||
body: 'One durable update',
|
||||
type: 'status'
|
||||
}
|
||||
})
|
||||
loseNextAckResponse = true
|
||||
|
||||
await expect(homeRuntime.syncOrchestrationFederation()).resolves.toBeUndefined()
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
|
||||
expect(
|
||||
homeDb
|
||||
.getRunMailboxHistory(task.run_id, 10)
|
||||
.filter((message) => message.subject === 'Checkpoint')
|
||||
).toHaveLength(1)
|
||||
expect(homeDb.getFederatedDispatch(dispatch.id)?.to_home_imported_sequence).toBe(1)
|
||||
})
|
||||
|
||||
it('rejects a reordered relay gap, then converges without loss or duplication', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
|
||||
expect(() =>
|
||||
homeDb.importFederatedRelayItem({
|
||||
dispatchId: dispatch.id,
|
||||
sequence: 2,
|
||||
message: {
|
||||
id: 'relay_gap',
|
||||
runId: task.run_id,
|
||||
from: `dispatch:${dispatch.id}`,
|
||||
to: `run:${task.run_id}`,
|
||||
subject: 'Gap',
|
||||
body: 'Out of order',
|
||||
type: 'status',
|
||||
priority: 'normal'
|
||||
},
|
||||
lifecycle: { kind: 'none' }
|
||||
})
|
||||
).toThrow(/not contiguous/)
|
||||
expect(homeDb.getMessageById('relay_gap')).toBeUndefined()
|
||||
expect(homeDb.getFederatedDispatch(dispatch.id)?.to_home_imported_sequence).toBe(0)
|
||||
|
||||
homeDb.importFederatedRelayItem({
|
||||
dispatchId: dispatch.id,
|
||||
sequence: 1,
|
||||
message: {
|
||||
id: 'relay_first',
|
||||
runId: task.run_id,
|
||||
from: `dispatch:${dispatch.id}`,
|
||||
to: `run:${task.run_id}`,
|
||||
subject: 'First',
|
||||
body: 'Arrived after the gap was rejected',
|
||||
type: 'status',
|
||||
priority: 'normal'
|
||||
},
|
||||
lifecycle: { kind: 'none' }
|
||||
})
|
||||
const recovered = homeDb.importFederatedRelayItem({
|
||||
dispatchId: dispatch.id,
|
||||
sequence: 2,
|
||||
message: {
|
||||
id: 'relay_gap',
|
||||
runId: task.run_id,
|
||||
from: `dispatch:${dispatch.id}`,
|
||||
to: `run:${task.run_id}`,
|
||||
subject: 'Gap',
|
||||
body: 'Out of order',
|
||||
type: 'status',
|
||||
priority: 'normal'
|
||||
},
|
||||
lifecycle: { kind: 'none' }
|
||||
})
|
||||
const duplicate = homeDb.importFederatedRelayItem({
|
||||
dispatchId: dispatch.id,
|
||||
sequence: 2,
|
||||
message: {
|
||||
id: 'relay_gap',
|
||||
runId: task.run_id,
|
||||
from: `dispatch:${dispatch.id}`,
|
||||
to: `run:${task.run_id}`,
|
||||
subject: 'Gap',
|
||||
body: 'Out of order',
|
||||
type: 'status',
|
||||
priority: 'normal'
|
||||
},
|
||||
lifecycle: { kind: 'none' }
|
||||
})
|
||||
|
||||
expect(recovered.duplicate).toBe(false)
|
||||
expect(duplicate.duplicate).toBe(true)
|
||||
expect(homeDb.getFederatedDispatch(dispatch.id)?.to_home_imported_sequence).toBe(2)
|
||||
expect(
|
||||
homeDb
|
||||
.getRunMailboxHistory(task.run_id, 10)
|
||||
.filter((message) => ['relay_first', 'relay_gap'].includes(message.id))
|
||||
).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('restarts relay polling when a federated worker is shown', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
const prompt = vi.mocked(workerRuntime.sendTerminalAgentPrompt).mock.calls[0]?.[1] ?? ''
|
||||
const capability = prompt.match(/--dispatch-capability (dcap_[A-Za-z0-9_-]+)/)?.[1]
|
||||
homeRuntime.stopOrchestrationFederationRelay()
|
||||
await workerDispatcher.dispatch({
|
||||
id: 'rpc_restart_status',
|
||||
authToken: 'worker-local-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'restart_status_request',
|
||||
orchestrationCapability: capability,
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_windows_worker',
|
||||
subject: 'After home restart',
|
||||
body: 'Relay me after worker-show',
|
||||
type: 'status'
|
||||
}
|
||||
})
|
||||
|
||||
await homeDispatcher.dispatch({
|
||||
id: 'rpc_restart_show',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerShow',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
|
||||
await vi.waitFor(() =>
|
||||
expect(
|
||||
homeDb
|
||||
.getRunMailboxHistory(task.run_id, 10)
|
||||
.some((message) => message.subject === 'After home restart')
|
||||
).toBe(true)
|
||||
)
|
||||
})
|
||||
|
||||
it('treats a worker runtime ID change as an epoch, not a new server', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
const oldEpoch = homeDb.getFederatedDispatch(dispatch.id)?.remote_runtime_epoch
|
||||
restartWorkerRuntime()
|
||||
|
||||
const shown = await homeDispatcher.dispatch({
|
||||
id: 'rpc_worker_restart_show',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerShow',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
|
||||
expect(shown).toMatchObject({
|
||||
ok: true,
|
||||
result: { observation: { status: 'running', exactWorker: true } }
|
||||
})
|
||||
expect(homeDb.getFederatedDispatch(dispatch.id)?.remote_runtime_epoch).not.toBe(oldEpoch)
|
||||
expect(homeDb.getFederatedDispatch(dispatch.id)?.peer_fingerprint).toBe(
|
||||
'windows_peer_fingerprint'
|
||||
)
|
||||
})
|
||||
|
||||
it('stops only the exact remote agent terminal', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
|
||||
const stopped = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_stop',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'request_remote_stop',
|
||||
method: 'orchestration.workerStop',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
|
||||
expect(stopped).toMatchObject({
|
||||
ok: true,
|
||||
result: { state: 'stopped', processAction: 'closed_agent_terminal' }
|
||||
})
|
||||
expect(workerRuntime.closeTerminal).toHaveBeenCalledTimes(1)
|
||||
expect(workerRuntime.closeTerminal).toHaveBeenCalledWith('term_windows_worker')
|
||||
expect(homeDb.getTask(task.id)?.status).toBe('blocked')
|
||||
|
||||
vi.mocked(workerRuntime.showTerminal).mockResolvedValue({
|
||||
handle: 'term_windows_worker',
|
||||
worktreeId: 'repo::windows-worktree',
|
||||
connected: false,
|
||||
writable: false
|
||||
} as never)
|
||||
const shown = await homeDispatcher.dispatch({
|
||||
id: 'rpc_remote_show_after_stop',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerShow',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
expect(shown).toMatchObject({
|
||||
ok: true,
|
||||
result: { observation: { status: 'exited', exactWorker: true } }
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects a re-paired server before show or stop effects', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
workerPeerFingerprint = 'replacement_windows_peer'
|
||||
|
||||
const shown = await homeDispatcher.dispatch({
|
||||
id: 'rpc_changed_peer_show',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerShow',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
const stopped = await homeDispatcher.dispatch({
|
||||
id: 'rpc_changed_peer_stop',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'request_changed_peer_stop',
|
||||
method: 'orchestration.workerStop',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
|
||||
expect(shown).toMatchObject({ ok: false, error: { code: 'peer_changed' } })
|
||||
expect(stopped).toMatchObject({ ok: false, error: { code: 'peer_changed' } })
|
||||
expect(homeDb.getWorkerDispatch(dispatch.id)?.state).toBe('ready')
|
||||
expect(workerRuntime.closeTerminal).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('coalesces overlapping relay polls for the same Dispatch', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
homeRuntime.stopOrchestrationFederationRelay()
|
||||
|
||||
let releasePull!: () => void
|
||||
const blockedPull = new Promise<void>((resolve) => {
|
||||
releasePull = resolve
|
||||
})
|
||||
let pullCount = 0
|
||||
vi.spyOn(homeRuntime, 'callOrchestrationWorkerServer').mockImplementation(
|
||||
async (_selector, method) => {
|
||||
if (method !== 'orchestration.federationPull') {
|
||||
throw new Error(`Unexpected relay method ${method}`)
|
||||
}
|
||||
pullCount += 1
|
||||
await blockedPull
|
||||
return { runtimeEpoch: workerRuntime.getRuntimeId(), items: [] }
|
||||
}
|
||||
)
|
||||
|
||||
const first = homeRuntime.syncOrchestrationFederation()
|
||||
const second = homeRuntime.syncOrchestrationFederation()
|
||||
await vi.waitFor(() => expect(pullCount).toBe(1))
|
||||
releasePull()
|
||||
await Promise.all([first, second])
|
||||
|
||||
expect(pullCount).toBe(1)
|
||||
})
|
||||
|
||||
it('warns once while a federated Dispatch remains unreachable', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
homeRuntime.stopOrchestrationFederationRelay()
|
||||
vi.spyOn(homeRuntime, 'callOrchestrationWorkerServer').mockRejectedValue(
|
||||
new Error('worker server offline')
|
||||
)
|
||||
const warn = vi.spyOn(console, 'warn').mockImplementation(() => {})
|
||||
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
await homeRuntime.syncOrchestrationFederation()
|
||||
|
||||
expect(warn).toHaveBeenCalledTimes(1)
|
||||
expect(warn).toHaveBeenCalledWith(
|
||||
expect.stringContaining('Federation sync failed'),
|
||||
expect.any(Error)
|
||||
)
|
||||
warn.mockRestore()
|
||||
})
|
||||
|
||||
it('returns stop_unknown when the worker server disconnects after the home fence', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
vi.spyOn(homeRuntime, 'callOrchestrationWorkerServer').mockRejectedValueOnce(
|
||||
new Error('connection lost')
|
||||
)
|
||||
|
||||
const stopped = await homeDispatcher.dispatch({
|
||||
id: 'rpc_disconnected_stop',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'request_disconnected_stop',
|
||||
method: 'orchestration.workerStop',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
|
||||
expect(stopped).toMatchObject({
|
||||
ok: true,
|
||||
result: { state: 'stop_unknown', processAction: 'unknown' }
|
||||
})
|
||||
expect(homeDb.getTask(task.id)?.status).toBe('blocked')
|
||||
expect(workerRuntime.closeTerminal).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('never reads or closes a same-looking replacement process', async () => {
|
||||
const task = createHomeTask()
|
||||
await homeDispatcher.dispatch(startRequest(task.id))
|
||||
const dispatch = homeDb.getDispatchContext(task.id)!
|
||||
vi.mocked(workerRuntime.getTerminalProcessIncarnation).mockReturnValue(
|
||||
'windows_runtime:pty:replacement'
|
||||
)
|
||||
|
||||
const read = await homeDispatcher.dispatch({
|
||||
id: 'rpc_replacement_read',
|
||||
authToken: 'coordinator-token',
|
||||
method: 'orchestration.workerRead',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
const stopped = await homeDispatcher.dispatch({
|
||||
id: 'rpc_replacement_stop',
|
||||
authToken: 'coordinator-token',
|
||||
orchestrationContractVersion: ORCHESTRATION_CONTRACT_VERSION,
|
||||
orchestrationRequestId: 'request_replacement_stop',
|
||||
method: 'orchestration.workerStop',
|
||||
params: { dispatch: dispatch.id }
|
||||
})
|
||||
|
||||
expect(read).toMatchObject({
|
||||
ok: false,
|
||||
error: { code: 'worker_identity_changed' }
|
||||
})
|
||||
expect(stopped).toMatchObject({
|
||||
ok: true,
|
||||
result: { state: 'stop_unknown', processAction: 'none' }
|
||||
})
|
||||
expect(workerRuntime.closeTerminal).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,297 @@
|
||||
import { isTuiAgent } from '../../../../shared/tui-agent-config'
|
||||
import type { TuiAgent } from '../../../../shared/types'
|
||||
import { buildDispatchPreamble } from '../../orchestration/preamble'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import { defineMethod, type RpcMethod } from '../core'
|
||||
import { assertOrchestrationWorktreeCreationSupported } from './orchestration-folder-worktree-placement'
|
||||
import {
|
||||
appendFederationSetupEffect,
|
||||
appendFederationTerminalEffects,
|
||||
type FederationEffect
|
||||
} from './orchestration-federation-effects'
|
||||
import type { WorkerSetupReceipt } from './orchestration-worker-topology'
|
||||
import {
|
||||
monitorFederatedSetup,
|
||||
persistFederatedReadinessStage,
|
||||
persistFederatedSetupSpawnFailure,
|
||||
persistFederatedSetupWaitOutcome
|
||||
} from './orchestration-federation-setup'
|
||||
import { FederationAttachStartParams } from './orchestration-federation-start-schema'
|
||||
import { failFederatedAttachmentWithReceipt } from './orchestration-federation-start-receipt'
|
||||
|
||||
export const ORCHESTRATION_FEDERATION_ATTACH_METHODS: RpcMethod[] = [
|
||||
defineMethod({
|
||||
name: 'orchestration.federationAttachStart',
|
||||
params: FederationAttachStartParams,
|
||||
handler: async (params, { runtime, orchestrationMutation }) => {
|
||||
if (!orchestrationMutation) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'Federated worker attachment requires a durable retry request.'
|
||||
)
|
||||
}
|
||||
if (params.worktree === 'current' || params.worktree === 'new-child') {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'A remote worker requires an exact existing worktree or new-top-level.'
|
||||
)
|
||||
}
|
||||
const createsWorktree = params.worktree === 'new-top-level'
|
||||
if (createsWorktree && (!params.name || !params.repo)) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'A remote new-top-level worktree requires --name and an explicit --repo.'
|
||||
)
|
||||
}
|
||||
if (createsWorktree && params.terminal) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'--terminal cannot combine with remote new-worktree creation.'
|
||||
)
|
||||
}
|
||||
if (
|
||||
!createsWorktree &&
|
||||
(params.name || params.repo || params.baseBranch || params.setup || params.setupSource)
|
||||
) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'Creation and setup options apply only to remote new-top-level worktrees.'
|
||||
)
|
||||
}
|
||||
if (params.terminal && params.agent) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'--terminal reuses an existing agent and cannot combine with --agent.'
|
||||
)
|
||||
}
|
||||
const agent = params.agent
|
||||
if (!params.terminal && (!agent || !isTuiAgent(agent))) {
|
||||
throw new OrchestrationError(
|
||||
'agent_unconfigured',
|
||||
'A configured --agent is required when federated worker-start creates a terminal.'
|
||||
)
|
||||
}
|
||||
if (agent) {
|
||||
runtime.validateOrchestrationAgentLauncher(agent as TuiAgent)
|
||||
}
|
||||
if (createsWorktree) {
|
||||
await assertOrchestrationWorktreeCreationSupported({
|
||||
runtime,
|
||||
repoSelector: params.repo as string,
|
||||
existingPlacement: 'an exact existing folder workspace'
|
||||
})
|
||||
}
|
||||
|
||||
const db = runtime.getOrchestrationDb()
|
||||
db.createRemoteDispatchAttachment({
|
||||
dispatchId: params.dispatchId,
|
||||
taskId: params.taskId,
|
||||
homePeerFingerprint: orchestrationMutation.callerFingerprint,
|
||||
protocolVersion: params.protocolVersion,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
mutationReceipt: orchestrationMutation
|
||||
})
|
||||
const effects: FederationEffect[] = []
|
||||
let failedStage = createsWorktree ? 'worktree_create' : 'worktree_resolve'
|
||||
let worktree
|
||||
let terminalHandle = params.terminal
|
||||
const setupSource = createsWorktree
|
||||
? (params.setupSource ?? (params.setup ? 'explicit_request' : 'orchestration_default'))
|
||||
: 'existing_worktree'
|
||||
let setup: WorkerSetupReceipt = {
|
||||
requested: createsWorktree ? (params.setup ?? 'run') : 'not_applicable',
|
||||
effective: createsWorktree ? (params.setup ?? 'run') : 'not_applicable',
|
||||
source: setupSource,
|
||||
hookFound: false,
|
||||
startupPolicy: 'start-immediately',
|
||||
state: createsWorktree ? 'not_configured' : 'not_applicable'
|
||||
}
|
||||
try {
|
||||
if (createsWorktree) {
|
||||
db.recordRemoteAttachmentStage({
|
||||
dispatchId: params.dispatchId,
|
||||
stage: 'worktree_creating'
|
||||
})
|
||||
const setupDecision = params.setup ?? 'run'
|
||||
const created = await runtime.createManagedWorktree({
|
||||
repoSelector: params.repo as string,
|
||||
name: params.name as string,
|
||||
baseBranch: params.baseBranch,
|
||||
displayName: params.displayName,
|
||||
comment: params.comment,
|
||||
runHooks: setupDecision === 'run',
|
||||
setupDecision,
|
||||
awaitTerminalProvisioning: true,
|
||||
observeSetupCompletion: true,
|
||||
createdWithAgent: agent as TuiAgent,
|
||||
startupAgent: agent as TuiAgent,
|
||||
activate: false,
|
||||
lineage: { noParent: true }
|
||||
})
|
||||
worktree = created.worktree
|
||||
terminalHandle = created.startupTerminal?.handle
|
||||
effects.push({
|
||||
kind: 'worktree',
|
||||
action: 'created_top_level',
|
||||
id: created.worktree.id
|
||||
})
|
||||
setup = {
|
||||
requested: setupDecision,
|
||||
effective: setupDecision,
|
||||
source: setupSource,
|
||||
hookFound: created.setupReceipt?.hookFound ?? false,
|
||||
startupPolicy: created.setupReceipt?.startupPolicy ?? 'start-immediately',
|
||||
state: created.setupReceipt?.state ?? 'not_configured'
|
||||
}
|
||||
if (!terminalHandle) {
|
||||
throw new Error(
|
||||
created.warning ?? 'Agent-first worktree creation returned no terminal.'
|
||||
)
|
||||
}
|
||||
const listed = await runtime.listTerminals(`id:${created.worktree.id}`)
|
||||
appendFederationTerminalEffects(
|
||||
effects,
|
||||
listed.terminals,
|
||||
terminalHandle,
|
||||
created.setupReceipt?.terminalHandle
|
||||
)
|
||||
appendFederationSetupEffect(effects, setup)
|
||||
} else {
|
||||
worktree = await runtime.showManagedWorktree(params.worktree).catch(() => {
|
||||
throw new OrchestrationError(
|
||||
'worktree_not_found_on_server',
|
||||
`Worktree ${params.worktree} was not found on the selected worker server.`
|
||||
)
|
||||
})
|
||||
effects.push(
|
||||
{ kind: 'worktree', action: 'reused', id: worktree.id },
|
||||
{ kind: 'setup', action: 'not_applicable', state: 'not_applicable' }
|
||||
)
|
||||
if (terminalHandle) {
|
||||
const terminal = await runtime.showTerminal(terminalHandle)
|
||||
if (terminal.worktreeId !== worktree.id) {
|
||||
throw new OrchestrationError(
|
||||
'terminal_worktree_mismatch',
|
||||
`Terminal ${terminalHandle} does not belong to worktree ${worktree.id}.`
|
||||
)
|
||||
}
|
||||
if (!(await runtime.isTerminalRunningAgent(terminalHandle))) {
|
||||
throw new OrchestrationError(
|
||||
'agent_unconfigured',
|
||||
`Terminal ${terminalHandle} is not running a recognized agent.`
|
||||
)
|
||||
}
|
||||
effects.push({
|
||||
kind: 'terminal',
|
||||
role: 'agent',
|
||||
action: 'reused',
|
||||
id: terminalHandle
|
||||
})
|
||||
} else {
|
||||
failedStage = 'terminal_create'
|
||||
const terminal = await runtime.createTerminal(`id:${worktree.id}`, {
|
||||
command: agent,
|
||||
title: `worker-${params.taskId}`,
|
||||
presentation: 'background'
|
||||
})
|
||||
terminalHandle = terminal.handle
|
||||
effects.push({
|
||||
kind: 'terminal',
|
||||
role: 'agent',
|
||||
action: 'created',
|
||||
id: terminal.handle
|
||||
})
|
||||
}
|
||||
}
|
||||
if (!worktree || !terminalHandle) {
|
||||
throw new Error('Federated worker topology did not resolve.')
|
||||
}
|
||||
const setupStage = {
|
||||
db,
|
||||
dispatchId: params.dispatchId,
|
||||
worktreeId: worktree.id,
|
||||
terminalHandle,
|
||||
setup,
|
||||
effects
|
||||
}
|
||||
if (persistFederatedSetupSpawnFailure(setupStage)) {
|
||||
failedStage = 'setup_start'
|
||||
throw new Error('Setup terminal failed to start before the gated agent launch.')
|
||||
}
|
||||
persistFederatedReadinessStage(setupStage)
|
||||
failedStage = 'agent_readiness'
|
||||
const wait = await runtime.waitForTerminal(terminalHandle, {
|
||||
condition: 'tui-idle',
|
||||
timeoutMs: params.timeoutMs ?? 60_000
|
||||
})
|
||||
persistFederatedSetupWaitOutcome({ ...setupStage, wait })
|
||||
if (!wait.satisfied) {
|
||||
if (setup.state === 'failed') {
|
||||
failedStage = 'setup_wait'
|
||||
}
|
||||
throw new Error(
|
||||
wait.blockedReason
|
||||
? `Agent startup blocked: ${wait.blockedReason}`
|
||||
: `Agent did not become ready (${wait.status}).`
|
||||
)
|
||||
}
|
||||
const paneKey = runtime.getTerminalPaneKey(terminalHandle)
|
||||
const processIncarnation = runtime.getTerminalProcessIncarnation(terminalHandle)
|
||||
if (!paneKey || !processIncarnation) {
|
||||
throw new Error('stable_pane_required')
|
||||
}
|
||||
const capability = db.prepareRemoteAttachmentAuthority({
|
||||
dispatchId: params.dispatchId,
|
||||
paneKey,
|
||||
processIncarnation,
|
||||
worktreeId: worktree.id,
|
||||
terminalHandle,
|
||||
setupState: setup.state,
|
||||
effects
|
||||
})
|
||||
failedStage = 'dispatch_input'
|
||||
await runtime.sendTerminalAgentPrompt(
|
||||
terminalHandle,
|
||||
buildDispatchPreamble({
|
||||
taskId: params.taskId,
|
||||
dispatchId: params.dispatchId,
|
||||
taskSpec: params.taskSpec,
|
||||
coordinatorHandle: 'Run home (relayed by Orca)',
|
||||
workerHandle: terminalHandle,
|
||||
dispatchCapability: capability,
|
||||
devMode: params.devMode,
|
||||
cliCommand: runtime.getTerminalOrchestrationCliCommand(terminalHandle)
|
||||
})
|
||||
)
|
||||
effects.push({
|
||||
kind: 'dispatch_input',
|
||||
role: 'agent',
|
||||
id: terminalHandle,
|
||||
state: 'accepted'
|
||||
})
|
||||
const attachment = db.markRemoteAttachmentReady(params.dispatchId, effects)
|
||||
monitorFederatedSetup({ ...setupStage, runtime })
|
||||
return {
|
||||
dispatchId: params.dispatchId,
|
||||
state: attachment.state,
|
||||
stage: attachment.stage,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
worktreeId: worktree.id,
|
||||
terminalHandle,
|
||||
setup,
|
||||
effects,
|
||||
residualResources: []
|
||||
}
|
||||
} catch (error) {
|
||||
return failFederatedAttachmentWithReceipt({
|
||||
db,
|
||||
dispatchId: params.dispatchId,
|
||||
runtimeEpoch: runtime.getRuntimeId(),
|
||||
failedStage,
|
||||
error,
|
||||
setup
|
||||
})
|
||||
}
|
||||
}
|
||||
})
|
||||
]
|
||||
@@ -0,0 +1,17 @@
|
||||
import { isFolderRepo } from '../../../../shared/repo-kind'
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
|
||||
export async function assertOrchestrationWorktreeCreationSupported(args: {
|
||||
runtime: OrcaRuntimeService
|
||||
repoSelector: string
|
||||
existingPlacement: string
|
||||
}): Promise<void> {
|
||||
if (!isFolderRepo(await args.runtime.showRepo(args.repoSelector))) {
|
||||
return
|
||||
}
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
`Folder projects cannot create orchestration worktrees; use ${args.existingPlacement}.`
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY,
|
||||
ORCHESTRATION_FEDERATION_RUNTIME_CAPABILITY
|
||||
} from '../../../../shared/protocol-version'
|
||||
import { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationDb } from '../../orchestration/db'
|
||||
import { RpcDispatcher } from '../dispatcher'
|
||||
import { ORCHESTRATION_METHODS } from './orchestration'
|
||||
import { startFederatedWorker } from './orchestration-federated-worker-start'
|
||||
|
||||
describe('orchestration migration behavior', () => {
|
||||
const databases: OrchestrationDb[] = []
|
||||
|
||||
afterEach(() => {
|
||||
for (const database of databases.splice(0)) {
|
||||
database.close()
|
||||
}
|
||||
})
|
||||
|
||||
function createRuntime(): { db: OrchestrationDb; runtime: OrcaRuntimeService } {
|
||||
const db = new OrchestrationDb(':memory:')
|
||||
const runtime = new OrcaRuntimeService()
|
||||
runtime.setOrchestrationDb(db)
|
||||
databases.push(db)
|
||||
return { db, runtime }
|
||||
}
|
||||
|
||||
it('lists an explicitly selected legacy Run without binding or mutation', async () => {
|
||||
const { db, runtime } = createRuntime()
|
||||
const task = db.createTask({ spec: 'pre-upgrade work' })
|
||||
const taskList = ORCHESTRATION_METHODS.find(
|
||||
(method) => method.name === 'orchestration.taskList'
|
||||
)!
|
||||
|
||||
const listed = (await taskList.handler(taskList.params!.parse({ run: 'run_legacy_local' }), {
|
||||
runtime
|
||||
})) as {
|
||||
runId: string
|
||||
legacyReadOnly: boolean
|
||||
tasks: { id: string }[]
|
||||
}
|
||||
|
||||
expect(listed).toMatchObject({
|
||||
runId: 'run_legacy_local',
|
||||
legacyReadOnly: true,
|
||||
tasks: [{ id: task.id }]
|
||||
})
|
||||
expect(db.getTask(task.id)?.status).toBe('ready')
|
||||
})
|
||||
|
||||
it('rejects a pre-contract worker_done before message or lifecycle mutation', async () => {
|
||||
const { db, runtime } = createRuntime()
|
||||
const run = db.createRun({
|
||||
objective: 'legacy worker',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:leaf_coord'
|
||||
})
|
||||
const task = db.createTask({ spec: 'legacy worker', runId: run.id })
|
||||
const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker')
|
||||
const dispatcher = new RpcDispatcher({ runtime, methods: ORCHESTRATION_METHODS })
|
||||
|
||||
const response = await dispatcher.dispatch({
|
||||
id: 'legacy_worker_done',
|
||||
authToken: 'worker-token',
|
||||
method: 'orchestration.send',
|
||||
params: {
|
||||
from: 'term_worker',
|
||||
subject: 'done',
|
||||
type: 'worker_done',
|
||||
payload: JSON.stringify({
|
||||
taskId: task.id,
|
||||
dispatchId: dispatch.id,
|
||||
outcome: 'succeeded'
|
||||
})
|
||||
}
|
||||
})
|
||||
|
||||
expect(response).toMatchObject({
|
||||
ok: false,
|
||||
error: {
|
||||
code: 'orchestration_migration_required',
|
||||
data: { effectsApplied: false }
|
||||
}
|
||||
})
|
||||
expect(db.getInbox(100)).toHaveLength(0)
|
||||
expect(db.getTask(task.id)?.status).toBe('dispatched')
|
||||
expect(db.getDispatchContextById(dispatch.id)?.status).toBe('dispatched')
|
||||
})
|
||||
|
||||
it('rejects a connected server missing the contract before home or remote effects', async () => {
|
||||
const { db, runtime } = createRuntime()
|
||||
const run = db.createRun({
|
||||
objective: 'mixed-version worker',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:leaf_coord'
|
||||
})
|
||||
const task = db.createTask({ spec: 'remote work', runId: run.id })
|
||||
vi.spyOn(runtime, 'resolveOrchestrationWorkerServer').mockReturnValue({
|
||||
environmentId: 'environment_windows',
|
||||
name: 'windows',
|
||||
peerFingerprint: 'windows_peer'
|
||||
})
|
||||
vi.spyOn(runtime, 'callOrchestrationWorkerServer').mockResolvedValue({
|
||||
capabilities: [ORCHESTRATION_FEDERATION_RUNTIME_CAPABILITY]
|
||||
})
|
||||
|
||||
await expect(
|
||||
startFederatedWorker({
|
||||
params: {
|
||||
task: task.id,
|
||||
from: 'term_coord',
|
||||
on: 'windows',
|
||||
worktree: 'new-top-level',
|
||||
repo: 'id:windows-repo',
|
||||
name: 'remote-work',
|
||||
agent: 'codex'
|
||||
},
|
||||
runtime,
|
||||
db,
|
||||
runId: run.id,
|
||||
task,
|
||||
orchestrationMutation: {
|
||||
callerFingerprint: 'caller',
|
||||
requestId: 'remote_start',
|
||||
method: 'orchestration.workerStart',
|
||||
payloadHash: 'payload'
|
||||
}
|
||||
})
|
||||
).rejects.toMatchObject({
|
||||
code: 'orchestration_migration_required',
|
||||
data: { reason: 'runtime_capability_missing', effectsApplied: false }
|
||||
})
|
||||
expect(db.getTask(task.id)?.status).toBe('ready')
|
||||
expect(db.getDispatchContext(task.id)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects a connected server missing federation support before Task mutation', async () => {
|
||||
const { db, runtime } = createRuntime()
|
||||
const run = db.createRun({
|
||||
objective: 'unsupported worker',
|
||||
coordinatorHandle: 'term_coord',
|
||||
coordinatorPaneKey: 'tab_coord:leaf_coord'
|
||||
})
|
||||
const task = db.createTask({ spec: 'remote work', runId: run.id })
|
||||
vi.spyOn(runtime, 'resolveOrchestrationWorkerServer').mockReturnValue({
|
||||
environmentId: 'environment_windows',
|
||||
name: 'windows',
|
||||
peerFingerprint: 'windows_peer'
|
||||
})
|
||||
vi.spyOn(runtime, 'callOrchestrationWorkerServer').mockResolvedValue({
|
||||
capabilities: [ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY]
|
||||
})
|
||||
|
||||
await expect(
|
||||
startFederatedWorker({
|
||||
params: {
|
||||
task: task.id,
|
||||
from: 'term_coord',
|
||||
on: 'windows',
|
||||
worktree: 'new-top-level',
|
||||
repo: 'id:windows-repo',
|
||||
name: 'remote-work',
|
||||
agent: 'codex'
|
||||
},
|
||||
runtime,
|
||||
db,
|
||||
runId: run.id,
|
||||
task,
|
||||
orchestrationMutation: {
|
||||
callerFingerprint: 'caller',
|
||||
requestId: 'remote_start',
|
||||
method: 'orchestration.workerStart',
|
||||
payloadHash: 'payload'
|
||||
}
|
||||
})
|
||||
).rejects.toMatchObject({ code: 'capability_unsupported' })
|
||||
expect(db.getTask(task.id)?.status).toBe('ready')
|
||||
expect(db.getDispatchContext(task.id)).toBeUndefined()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,100 @@
|
||||
import { z } from 'zod'
|
||||
import { defineMethod, type RpcMethod } from '../core'
|
||||
import { OptionalString, requiredString } from '../schemas'
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
|
||||
const RunCreateParams = z.object({
|
||||
objective: requiredString('Missing --objective'),
|
||||
from: requiredString('Missing coordinator terminal')
|
||||
})
|
||||
|
||||
const RunUseParams = z.object({
|
||||
id: requiredString('Missing --id'),
|
||||
from: requiredString('Missing coordinator terminal')
|
||||
})
|
||||
|
||||
const RunCurrentParams = z.object({ from: requiredString('Missing coordinator terminal') })
|
||||
const RunListParams = z.object({})
|
||||
const RunShowParams = z.object({ id: requiredString('Missing --id'), from: OptionalString })
|
||||
|
||||
function requireCallerPane(runtime: OrcaRuntimeService, handle: string): string {
|
||||
const paneKey = runtime.getTerminalPaneKey(handle)
|
||||
if (!paneKey) {
|
||||
throw new OrchestrationError(
|
||||
'stable_pane_required',
|
||||
'The coordinator terminal has no stable pane identity. Run this command inside a live Orca terminal.'
|
||||
)
|
||||
}
|
||||
return paneKey
|
||||
}
|
||||
|
||||
export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [
|
||||
defineMethod({
|
||||
name: 'orchestration.runCreate',
|
||||
params: RunCreateParams,
|
||||
handler: (params, { runtime }) => {
|
||||
const paneKey = requireCallerPane(runtime, params.from)
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const priorRun = db.getCurrentRunForPane(paneKey)
|
||||
const run = db.createRun({
|
||||
objective: params.objective,
|
||||
coordinatorHandle: params.from,
|
||||
coordinatorPaneKey: paneKey
|
||||
})
|
||||
if (priorRun) {
|
||||
runtime.cancelMessageWaiters(`run:${priorRun.id}`)
|
||||
}
|
||||
return { run, binding: { consumerGeneration: run.consumer_generation } }
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.runUse',
|
||||
params: RunUseParams,
|
||||
handler: (params, { runtime }) => {
|
||||
const paneKey = requireCallerPane(runtime, params.from)
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const priorRun = db.getCurrentRunForPane(paneKey)
|
||||
const run = db.bindRun({
|
||||
runId: params.id,
|
||||
coordinatorHandle: params.from,
|
||||
coordinatorPaneKey: paneKey
|
||||
})
|
||||
if (!run) {
|
||||
throw new OrchestrationError(
|
||||
'run_not_found',
|
||||
`Run ${params.id} was not found or is inspect-only.`
|
||||
)
|
||||
}
|
||||
runtime.cancelMessageWaiters(`run:${params.id}`)
|
||||
if (priorRun && priorRun.id !== params.id) {
|
||||
runtime.cancelMessageWaiters(`run:${priorRun.id}`)
|
||||
}
|
||||
return { run, binding: { consumerGeneration: run.consumer_generation } }
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.runCurrent',
|
||||
params: RunCurrentParams,
|
||||
handler: (params, { runtime }) => {
|
||||
const paneKey = requireCallerPane(runtime, params.from)
|
||||
return { run: runtime.getOrchestrationDb().getCurrentRunForPane(paneKey) ?? null }
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.runList',
|
||||
params: RunListParams,
|
||||
handler: (_params, { runtime }) => ({ runs: runtime.getOrchestrationDb().listRuns() })
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.runShow',
|
||||
params: RunShowParams,
|
||||
handler: (params, { runtime }) => {
|
||||
const run = runtime.getOrchestrationDb().getRun(params.id)
|
||||
if (!run) {
|
||||
throw new OrchestrationError('run_not_found', `Run ${params.id} was not found.`)
|
||||
}
|
||||
return { run }
|
||||
}
|
||||
})
|
||||
]
|
||||
@@ -0,0 +1,294 @@
|
||||
import { z } from 'zod'
|
||||
import {
|
||||
ORCHESTRATION_WORKER_READ_SOURCES,
|
||||
type OrchestrationWorkerReadResult
|
||||
} from '../../../../shared/orchestration-worker-output'
|
||||
import type { RuntimeTerminalRead } from '../../../../shared/runtime-types'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import { syncFederatedDispatch } from '../../orchestration/federation-sync'
|
||||
import {
|
||||
createWorkerOutputSourceIdentity,
|
||||
decodeWorkerOutputCursor,
|
||||
encodeWorkerOutputCursor
|
||||
} from '../../orchestration/worker-output-cursor'
|
||||
import { defineMethod, type RpcMethod } from '../core'
|
||||
import { OptionalFiniteNumber, requiredString } from '../schemas'
|
||||
import {
|
||||
callFederatedWorkerShow,
|
||||
exposeWorker,
|
||||
inspectWorkerTerminal,
|
||||
resolvePinnedFederatedServer
|
||||
} from './orchestration-worker-observation'
|
||||
import { readExactWorkerOutput } from './orchestration-worker-output'
|
||||
|
||||
const WorkerDispatchParams = z.object({ dispatch: requiredString('Missing --dispatch') })
|
||||
const WorkerReadParams = WorkerDispatchParams.extend({
|
||||
cursor: z.union([z.number().int().nonnegative(), z.string().min(1).max(2_048)]).optional(),
|
||||
limit: OptionalFiniteNumber,
|
||||
source: z.enum(ORCHESTRATION_WORKER_READ_SOURCES).optional()
|
||||
})
|
||||
|
||||
export const ORCHESTRATION_WORKER_CONTROL_METHODS: RpcMethod[] = [
|
||||
defineMethod({
|
||||
name: 'orchestration.workerShow',
|
||||
params: WorkerDispatchParams,
|
||||
handler: async (params, { runtime }) => {
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const dispatch = db.getDispatchContextById(params.dispatch)
|
||||
let worker = db.getWorkerDispatch(params.dispatch)
|
||||
if (!dispatch || !worker) {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_not_found',
|
||||
`Worker Dispatch ${params.dispatch} was not found.`
|
||||
)
|
||||
}
|
||||
const federated = db.getFederatedDispatch(params.dispatch)
|
||||
if (federated) {
|
||||
const server = resolvePinnedFederatedServer(runtime, federated)
|
||||
runtime.ensureOrchestrationFederationRelay(dispatch.run_id)
|
||||
const remote = await callFederatedWorkerShow(runtime, federated)
|
||||
const attachment = remote.attachment
|
||||
worker = db.updateWorkerSetupEvidence({
|
||||
dispatchId: params.dispatch,
|
||||
setupState: attachment.setup_state,
|
||||
effects: attachment.effects
|
||||
}).worker
|
||||
if (
|
||||
attachment.state === 'succeeded' ||
|
||||
(attachment.state === 'failed' && attachment.stage === 'worker_report_queued')
|
||||
) {
|
||||
await syncFederatedDispatch(runtime, params.dispatch).catch(() => undefined)
|
||||
} else if (
|
||||
attachment.state === 'stopped' &&
|
||||
['stopping', 'stop_unknown'].includes(worker.state)
|
||||
) {
|
||||
worker = db.reconcileFederatedWorkerStop(params.dispatch)
|
||||
} else if (['ready', 'failed', 'stopped', 'start_unknown'].includes(attachment.state)) {
|
||||
worker = db.reconcileFederatedWorkerStart({
|
||||
dispatchId: params.dispatch,
|
||||
state: attachment.state as 'ready' | 'failed' | 'stopped' | 'start_unknown',
|
||||
stage: attachment.stage,
|
||||
lastError: attachment.last_error,
|
||||
worktreeId: attachment.worktree_id,
|
||||
terminalHandle: attachment.terminal_handle,
|
||||
setupState: attachment.setup_state,
|
||||
effects: attachment.effects,
|
||||
residualResources: attachment.residualResources
|
||||
})
|
||||
if (
|
||||
attachment.state === 'ready' &&
|
||||
attachment.worktree_id &&
|
||||
attachment.terminal_handle
|
||||
) {
|
||||
db.updateFederatedDispatchResources({
|
||||
dispatchId: params.dispatch,
|
||||
remoteRuntimeEpoch: remote.runtimeEpoch,
|
||||
worktreeId: attachment.worktree_id,
|
||||
terminalHandle: attachment.terminal_handle
|
||||
})
|
||||
}
|
||||
}
|
||||
worker = db.getWorkerDispatch(params.dispatch)
|
||||
if (!worker) {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_not_found',
|
||||
`Worker Dispatch ${params.dispatch} was not found after remote reconciliation.`
|
||||
)
|
||||
}
|
||||
return {
|
||||
dispatch: db.getDispatchContextById(params.dispatch),
|
||||
worker: exposeWorker(worker),
|
||||
server: { environmentId: server.environmentId, name: server.name },
|
||||
remoteRuntimeEpoch: remote.runtimeEpoch,
|
||||
terminal: remote.terminal,
|
||||
observation: remote.observation
|
||||
}
|
||||
}
|
||||
if (worker.runtime_epoch && worker.runtime_epoch !== runtime.getRuntimeId()) {
|
||||
if (worker.state === 'starting') {
|
||||
worker = db.markWorkerStartUnknown(
|
||||
params.dispatch,
|
||||
worker.stage,
|
||||
'The runtime restarted before worker-start reached a terminal receipt.'
|
||||
)
|
||||
} else if (worker.state === 'stopping') {
|
||||
worker = db.markWorkerStopUnknown(
|
||||
params.dispatch,
|
||||
'The runtime restarted before worker-stop reached a terminal receipt.'
|
||||
)
|
||||
}
|
||||
}
|
||||
const observation = await inspectWorkerTerminal(runtime, db, params.dispatch)
|
||||
return {
|
||||
dispatch,
|
||||
worker: exposeWorker(worker),
|
||||
terminal: observation.exact ? observation.terminal : null,
|
||||
observation: { status: observation.status, exactWorker: observation.exact }
|
||||
}
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.workerRead',
|
||||
params: WorkerReadParams,
|
||||
handler: async (params, { runtime }) => {
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const federated = db.getFederatedDispatch(params.dispatch)
|
||||
if (federated) {
|
||||
const server = resolvePinnedFederatedServer(runtime, federated)
|
||||
try {
|
||||
const remote = (await runtime.callOrchestrationWorkerServer(
|
||||
server.environmentId,
|
||||
'orchestration.federationReadOutput',
|
||||
{
|
||||
dispatchId: params.dispatch,
|
||||
cursor: params.cursor,
|
||||
limit: params.limit,
|
||||
source: params.source
|
||||
},
|
||||
15_000
|
||||
)) as { runtimeEpoch: string; output: OrchestrationWorkerReadResult }
|
||||
return {
|
||||
...remote.output,
|
||||
server: { environmentId: server.environmentId, name: server.name },
|
||||
remoteRuntimeEpoch: remote.runtimeEpoch
|
||||
}
|
||||
} catch (error) {
|
||||
if (!(error instanceof OrchestrationError) || error.code !== 'method_not_found') {
|
||||
throw error
|
||||
}
|
||||
return readLegacyFederatedTerminal({
|
||||
runtime,
|
||||
server,
|
||||
federated,
|
||||
workerState: db.getWorkerDispatch(params.dispatch)?.state ?? 'unknown',
|
||||
dispatchId: params.dispatch,
|
||||
source: params.source,
|
||||
cursor: params.cursor,
|
||||
limit: params.limit
|
||||
})
|
||||
}
|
||||
}
|
||||
const worker = db.getWorkerDispatch(params.dispatch)
|
||||
if (!worker?.agent_terminal_handle) {
|
||||
throw new OrchestrationError(
|
||||
'dispatch_not_found',
|
||||
`Worker Dispatch ${params.dispatch} has no agent terminal.`
|
||||
)
|
||||
}
|
||||
const observation = await inspectWorkerTerminal(runtime, db, params.dispatch)
|
||||
if (!observation.exact) {
|
||||
throw new OrchestrationError(
|
||||
'worker_identity_changed',
|
||||
`Worker Dispatch ${params.dispatch} no longer resolves to its exact process.`
|
||||
)
|
||||
}
|
||||
const output = await readExactWorkerOutput({
|
||||
runtime,
|
||||
dispatchId: params.dispatch,
|
||||
terminalHandle: worker.agent_terminal_handle,
|
||||
workerState: worker.state,
|
||||
terminalStatus: observation.status === 'exited' ? 'exited' : 'running',
|
||||
attachedAt: worker.created_at,
|
||||
source: params.source,
|
||||
cursor: params.cursor,
|
||||
limit: params.limit
|
||||
})
|
||||
const afterRead = await inspectWorkerTerminal(runtime, db, params.dispatch)
|
||||
if (!afterRead.exact) {
|
||||
throw new OrchestrationError(
|
||||
'worker_identity_changed',
|
||||
`Worker Dispatch ${params.dispatch} changed process while output was read.`
|
||||
)
|
||||
}
|
||||
return output
|
||||
}
|
||||
}),
|
||||
defineMethod({
|
||||
name: 'orchestration.workerAbandon',
|
||||
params: WorkerDispatchParams,
|
||||
handler: (params, { runtime }) => {
|
||||
const abandoned = runtime.getOrchestrationDb().abandonWorkerDispatch(params.dispatch)
|
||||
const worker = abandoned.worker
|
||||
if (abandoned.disposition === 'abandoned') {
|
||||
runtime.notifyMessageArrived(`dispatch:${params.dispatch}`, 'status')
|
||||
}
|
||||
return {
|
||||
dispatchId: params.dispatch,
|
||||
state: worker.state,
|
||||
alreadySettled: abandoned.disposition !== 'abandoned',
|
||||
stale: abandoned.disposition === 'stale',
|
||||
processAction: 'none',
|
||||
warning:
|
||||
abandoned.disposition === 'stale'
|
||||
? 'The Dispatch is no longer current; no state or process changed.'
|
||||
: 'Possibly-live resources were retained; no process was stopped or deleted.',
|
||||
residualResources: JSON.parse(worker.residual_resources) as unknown[]
|
||||
}
|
||||
}
|
||||
})
|
||||
]
|
||||
|
||||
async function readLegacyFederatedTerminal(args: {
|
||||
runtime: Parameters<typeof resolvePinnedFederatedServer>[0]
|
||||
server: ReturnType<typeof resolvePinnedFederatedServer>
|
||||
federated: Parameters<typeof resolvePinnedFederatedServer>[1]
|
||||
workerState: string
|
||||
dispatchId: string
|
||||
source: (typeof ORCHESTRATION_WORKER_READ_SOURCES)[number] | undefined
|
||||
cursor: string | number | undefined
|
||||
limit: number | undefined
|
||||
}) {
|
||||
const cursor = decodeWorkerOutputCursor(args.cursor, args.dispatchId)
|
||||
if (args.source === 'transcript' || cursor?.source === 'transcript') {
|
||||
throw new OrchestrationError(
|
||||
'transcript_required',
|
||||
`Connected server ${args.server.name} does not support structured worker output.`,
|
||||
{ reason: 'remote_capability_unavailable' }
|
||||
)
|
||||
}
|
||||
const remote = (await args.runtime.callOrchestrationWorkerServer(
|
||||
args.server.environmentId,
|
||||
'orchestration.federationRead',
|
||||
{
|
||||
dispatchId: args.dispatchId,
|
||||
cursor: cursor?.source === 'terminal' ? cursor.position : undefined,
|
||||
limit: args.limit
|
||||
},
|
||||
15_000
|
||||
)) as { runtimeEpoch: string; terminal: RuntimeTerminalRead }
|
||||
const sourceIdentity = createWorkerOutputSourceIdentity([
|
||||
'legacy-remote-terminal',
|
||||
args.federated.peer_fingerprint,
|
||||
args.dispatchId,
|
||||
remote.runtimeEpoch
|
||||
])
|
||||
if (
|
||||
cursor?.source === 'terminal' &&
|
||||
cursor.sourceIdentity !== null &&
|
||||
cursor.sourceIdentity !== sourceIdentity
|
||||
) {
|
||||
throw new OrchestrationError(
|
||||
'source_changed',
|
||||
'The worker output source changed. Start a fresh worker-read without the old cursor.'
|
||||
)
|
||||
}
|
||||
const nextPosition =
|
||||
remote.terminal.nextCursor !== null && /^\d+$/.test(remote.terminal.nextCursor)
|
||||
? Number.parseInt(remote.terminal.nextCursor, 10)
|
||||
: null
|
||||
return {
|
||||
dispatchId: args.dispatchId,
|
||||
source: 'terminal' as const,
|
||||
sourceIdentity,
|
||||
terminal: remote.terminal,
|
||||
cursor:
|
||||
nextPosition === null
|
||||
? null
|
||||
: encodeWorkerOutputCursor(args.dispatchId, 'terminal', sourceIdentity, nextPosition),
|
||||
status: { worker: args.workerState, terminal: remote.terminal.status },
|
||||
fallbackReason: 'remote_capability_unavailable' as const,
|
||||
warnings: [],
|
||||
server: { environmentId: args.server.environmentId, name: args.server.name },
|
||||
remoteRuntimeEpoch: remote.runtimeEpoch
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import type { RpcMethod } from '../core'
|
||||
import { ORCHESTRATION_WORKER_CONTROL_METHODS } from './orchestration-worker-control'
|
||||
import { ORCHESTRATION_WORKER_STOP_METHODS } from './orchestration-worker-stop'
|
||||
import { ORCHESTRATION_WORKER_START_METHODS } from './orchestration-workers'
|
||||
|
||||
export const ORCHESTRATION_WORKER_METHODS: RpcMethod[] = [
|
||||
...ORCHESTRATION_WORKER_START_METHODS,
|
||||
...ORCHESTRATION_WORKER_CONTROL_METHODS,
|
||||
...ORCHESTRATION_WORKER_STOP_METHODS
|
||||
]
|
||||
@@ -0,0 +1,82 @@
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import type { OrchestrationDb } from '../../orchestration/db'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import type { FederatedDispatchRow, WorkerDispatchRow } from '../../orchestration/types'
|
||||
|
||||
export async function inspectWorkerTerminal(
|
||||
runtime: OrcaRuntimeService,
|
||||
db: OrchestrationDb,
|
||||
dispatchId: string
|
||||
): Promise<{
|
||||
terminal: Awaited<ReturnType<OrcaRuntimeService['showTerminal']>> | null
|
||||
exact: boolean
|
||||
status: 'unattached' | 'missing' | 'identity_changed' | 'running' | 'exited'
|
||||
}> {
|
||||
const worker = db.getWorkerDispatch(dispatchId)
|
||||
if (!worker?.agent_terminal_handle) {
|
||||
return { terminal: null, exact: false, status: 'unattached' }
|
||||
}
|
||||
const terminal = await runtime.showTerminal(worker.agent_terminal_handle).catch(() => null)
|
||||
if (!terminal) {
|
||||
return { terminal: null, exact: false, status: 'missing' }
|
||||
}
|
||||
const exact = db.isDispatchProcessCurrent({
|
||||
dispatchId,
|
||||
paneKey: runtime.getTerminalPaneKey(worker.agent_terminal_handle),
|
||||
processIncarnation: runtime.getTerminalProcessIncarnation(worker.agent_terminal_handle)
|
||||
})
|
||||
return {
|
||||
terminal,
|
||||
exact,
|
||||
status: exact ? (terminal.connected === false ? 'exited' : 'running') : 'identity_changed'
|
||||
}
|
||||
}
|
||||
|
||||
export function exposeWorker(worker: WorkerDispatchRow) {
|
||||
return {
|
||||
...worker,
|
||||
effects: JSON.parse(worker.effects) as unknown[],
|
||||
residualResources: JSON.parse(worker.residual_resources) as unknown[],
|
||||
startOptions: JSON.parse(worker.start_options) as unknown
|
||||
}
|
||||
}
|
||||
|
||||
export function resolvePinnedFederatedServer(
|
||||
runtime: OrcaRuntimeService,
|
||||
federated: FederatedDispatchRow
|
||||
) {
|
||||
const server = runtime.resolveOrchestrationWorkerServer(federated.environment_id)
|
||||
if (server.peerFingerprint !== federated.peer_fingerprint) {
|
||||
throw new OrchestrationError(
|
||||
'peer_changed',
|
||||
`Saved environment ${federated.environment_name} now identifies a different Orca server.`
|
||||
)
|
||||
}
|
||||
return server
|
||||
}
|
||||
|
||||
export async function callFederatedWorkerShow(
|
||||
runtime: OrcaRuntimeService,
|
||||
federated: FederatedDispatchRow
|
||||
): Promise<{
|
||||
runtimeEpoch: string
|
||||
attachment: {
|
||||
state: string
|
||||
stage: string
|
||||
last_error: string | null
|
||||
worktree_id: string | null
|
||||
terminal_handle: string | null
|
||||
setup_state: string
|
||||
effects: unknown[]
|
||||
residualResources: unknown[]
|
||||
}
|
||||
terminal: unknown
|
||||
observation: { status: string; exactWorker: boolean }
|
||||
}> {
|
||||
return (await runtime.callOrchestrationWorkerServer(
|
||||
federated.environment_id,
|
||||
'orchestration.federationShow',
|
||||
{ dispatchId: federated.dispatch_id },
|
||||
15_000
|
||||
)) as Awaited<ReturnType<typeof callFederatedWorkerShow>>
|
||||
}
|
||||
@@ -0,0 +1,205 @@
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { readExactWorkerOutput } from './orchestration-worker-output'
|
||||
|
||||
function codexMessage(id: string, text: string): string {
|
||||
return JSON.stringify({
|
||||
type: 'event_msg',
|
||||
payload: { id, type: 'agent_message', message: text }
|
||||
})
|
||||
}
|
||||
|
||||
describe('exact orchestration worker output', () => {
|
||||
let directory: string
|
||||
let transcriptA: string
|
||||
let transcriptB: string
|
||||
let providerSession: ReturnType<OrcaRuntimeService['getExactWorkerProviderSession']>
|
||||
let runtime: OrcaRuntimeService
|
||||
const readTerminal = vi.fn()
|
||||
|
||||
beforeEach(async () => {
|
||||
directory = await mkdtemp(join(tmpdir(), 'orca-worker-output-'))
|
||||
transcriptA = join(directory, 'session-a.jsonl')
|
||||
transcriptB = join(directory, 'session-b.jsonl')
|
||||
await writeFile(transcriptA, `${codexMessage('a', 'worker A only')}\n`)
|
||||
await writeFile(transcriptB, `${codexMessage('b', 'worker B only')}\n`)
|
||||
providerSession = {
|
||||
paneKey: 'tab:worker',
|
||||
processIncarnation: 'pty:incarnation-1',
|
||||
agent: 'codex',
|
||||
providerSession: {
|
||||
key: 'session_id',
|
||||
id: 'session-a',
|
||||
transcriptPath: transcriptA
|
||||
},
|
||||
observedAt: Date.now()
|
||||
}
|
||||
readTerminal.mockReset()
|
||||
readTerminal.mockResolvedValue({
|
||||
handle: 'term_worker',
|
||||
status: 'running',
|
||||
tail: ['terminal output'],
|
||||
truncated: false,
|
||||
nextCursor: '9'
|
||||
})
|
||||
runtime = {
|
||||
getExactWorkerProviderSession: vi.fn(() => providerSession),
|
||||
getTerminalProcessIncarnation: vi.fn(() => 'pty:incarnation-1'),
|
||||
getTerminalPaneKey: vi.fn(() => 'tab:worker'),
|
||||
readTerminal
|
||||
} as unknown as OrcaRuntimeService
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await rm(directory, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
const read = (overrides: Partial<Parameters<typeof readExactWorkerOutput>[0]> = {}) =>
|
||||
readExactWorkerOutput({
|
||||
runtime,
|
||||
dispatchId: 'dispatch_1',
|
||||
terminalHandle: 'term_worker',
|
||||
workerState: 'ready',
|
||||
terminalStatus: 'running',
|
||||
attachedAt: '2026-07-24 00:00:00',
|
||||
...overrides
|
||||
})
|
||||
|
||||
it('reads only the exact pane session and keeps its local path private', async () => {
|
||||
const result = await read()
|
||||
|
||||
expect(result).toMatchObject({
|
||||
source: 'transcript',
|
||||
provider: 'codex',
|
||||
transcript: {
|
||||
messages: [{ id: 'a', blocks: [{ type: 'text', text: 'worker A only' }] }]
|
||||
}
|
||||
})
|
||||
expect(JSON.stringify(result)).not.toContain(transcriptA)
|
||||
expect(JSON.stringify(result)).not.toContain('worker B only')
|
||||
expect(readTerminal).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('reads Grok through the shared Native Chat transcript decoder', async () => {
|
||||
await writeFile(
|
||||
transcriptA,
|
||||
`${JSON.stringify({
|
||||
id: 'grok-a',
|
||||
type: 'assistant',
|
||||
content: 'Grok worker only'
|
||||
})}\n`
|
||||
)
|
||||
providerSession = {
|
||||
...providerSession!,
|
||||
agent: 'grok',
|
||||
providerSession: {
|
||||
key: 'session_id',
|
||||
id: 'session-grok',
|
||||
transcriptPath: transcriptA
|
||||
}
|
||||
}
|
||||
|
||||
const result = await read()
|
||||
|
||||
expect(result).toMatchObject({
|
||||
source: 'transcript',
|
||||
provider: 'grok',
|
||||
transcript: {
|
||||
messages: [{ role: 'assistant', blocks: [{ type: 'text', text: 'Grok worker only' }] }]
|
||||
}
|
||||
})
|
||||
expect(readTerminal).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('labels OpenCode as a terminal fallback when no transcript decoder exists', async () => {
|
||||
const capability = `dcap_${'A'.repeat(43)}`
|
||||
readTerminal.mockResolvedValue({
|
||||
handle: 'term_worker',
|
||||
status: 'running',
|
||||
tail: [`opencode --dispatch-capability ${capability}`],
|
||||
truncated: false,
|
||||
nextCursor: '9'
|
||||
})
|
||||
providerSession = {
|
||||
...providerSession!,
|
||||
agent: 'opencode',
|
||||
providerSession: {
|
||||
key: 'session_id',
|
||||
id: 'session-opencode',
|
||||
transcriptPath: transcriptA
|
||||
}
|
||||
}
|
||||
|
||||
const result = await read()
|
||||
|
||||
expect(result).toMatchObject({
|
||||
source: 'terminal',
|
||||
fallbackReason: 'provider_unsupported',
|
||||
terminal: { tail: ['opencode --dispatch-capability [dispatch capability redacted]'] },
|
||||
warnings: ['Dispatch capability tokens were redacted from terminal output.']
|
||||
})
|
||||
expect(JSON.stringify(result)).not.toContain(capability)
|
||||
})
|
||||
|
||||
it('rejects an old cursor after the exact provider session changes', async () => {
|
||||
const initial = await read()
|
||||
if (initial.source !== 'transcript') {
|
||||
throw new Error('Expected transcript output')
|
||||
}
|
||||
providerSession = {
|
||||
...providerSession!,
|
||||
providerSession: {
|
||||
key: 'session_id',
|
||||
id: 'session-b',
|
||||
transcriptPath: transcriptB
|
||||
}
|
||||
}
|
||||
|
||||
await expect(read({ cursor: initial.cursor })).rejects.toMatchObject({
|
||||
code: 'source_changed'
|
||||
})
|
||||
})
|
||||
|
||||
it('uses a labeled terminal fallback and keeps its cursor pinned', async () => {
|
||||
providerSession = null
|
||||
const fallback = await read()
|
||||
|
||||
expect(fallback).toMatchObject({
|
||||
source: 'terminal',
|
||||
fallbackReason: 'session_not_reported',
|
||||
terminal: { tail: ['terminal output'] }
|
||||
})
|
||||
expect(fallback.cursor).toMatch(/^owr1_/)
|
||||
|
||||
providerSession = {
|
||||
paneKey: 'tab:worker',
|
||||
processIncarnation: 'pty:incarnation-1',
|
||||
agent: 'codex',
|
||||
providerSession: {
|
||||
key: 'session_id',
|
||||
id: 'session-a',
|
||||
transcriptPath: transcriptA
|
||||
},
|
||||
observedAt: Date.now()
|
||||
}
|
||||
await read({ cursor: fallback.cursor ?? undefined })
|
||||
|
||||
expect(readTerminal).toHaveBeenLastCalledWith('term_worker', {
|
||||
cursor: 9,
|
||||
limit: undefined
|
||||
})
|
||||
})
|
||||
|
||||
it('fails instead of falling back when transcript output is required', async () => {
|
||||
providerSession = null
|
||||
|
||||
await expect(read({ source: 'transcript' })).rejects.toMatchObject({
|
||||
code: 'transcript_required',
|
||||
data: { reason: 'session_not_reported' }
|
||||
})
|
||||
expect(readTerminal).not.toHaveBeenCalled()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,208 @@
|
||||
import type {
|
||||
OrchestrationWorkerReadFallbackReason,
|
||||
OrchestrationWorkerReadResult,
|
||||
OrchestrationWorkerReadSource
|
||||
} from '../../../../shared/orchestration-worker-output'
|
||||
import type { RuntimeTerminalState } from '../../../../shared/runtime-types'
|
||||
import type { OrcaRuntimeService } from '../../orca-runtime'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import {
|
||||
createWorkerOutputSourceIdentity,
|
||||
decodeWorkerOutputCursor,
|
||||
encodeWorkerOutputCursor
|
||||
} from '../../orchestration/worker-output-cursor'
|
||||
import { redactWorkerTerminalLines } from '../../orchestration/worker-transcript-payload'
|
||||
import { readWorkerTranscript } from '../../orchestration/worker-transcript-read'
|
||||
|
||||
export async function readExactWorkerOutput(args: {
|
||||
runtime: OrcaRuntimeService
|
||||
dispatchId: string
|
||||
terminalHandle: string
|
||||
workerState: string
|
||||
terminalStatus: RuntimeTerminalState
|
||||
attachedAt: string
|
||||
source?: OrchestrationWorkerReadSource
|
||||
cursor?: string | number
|
||||
limit?: number
|
||||
}): Promise<OrchestrationWorkerReadResult> {
|
||||
const source = args.source ?? 'auto'
|
||||
const cursor = decodeWorkerOutputCursor(args.cursor, args.dispatchId)
|
||||
assertCursorSourceMatchesRequest(cursor?.source, source)
|
||||
|
||||
if (cursor?.source === 'terminal' || source === 'terminal') {
|
||||
return readTerminalOutput(args, cursor)
|
||||
}
|
||||
|
||||
const observedAfter = orchestrationTimestampToMs(args.attachedAt)
|
||||
const session = args.runtime.getExactWorkerProviderSession(args.terminalHandle, observedAfter)
|
||||
if (!session) {
|
||||
if (cursor?.source === 'transcript') {
|
||||
throw sourceChanged()
|
||||
}
|
||||
return fallbackOrThrow(args, 'session_not_reported')
|
||||
}
|
||||
const transcript = await readWorkerTranscript({
|
||||
agent: session.agent,
|
||||
sessionId: session.providerSession.id,
|
||||
transcriptPath: session.providerSession.transcriptPath,
|
||||
offset: cursor?.source === 'transcript' ? cursor.position : undefined,
|
||||
limit: args.limit
|
||||
})
|
||||
if (!transcript.ok) {
|
||||
if (transcript.reason === 'source_changed') {
|
||||
throw sourceChanged()
|
||||
}
|
||||
if (cursor?.source === 'transcript') {
|
||||
throw transcriptRequired(args.dispatchId, transcript.reason)
|
||||
}
|
||||
return fallbackOrThrow(args, transcript.reason, transcript.warnings)
|
||||
}
|
||||
const sourceIdentity = createWorkerOutputSourceIdentity([
|
||||
'transcript',
|
||||
session.processIncarnation,
|
||||
session.agent,
|
||||
session.providerSession.key,
|
||||
session.providerSession.id,
|
||||
transcript.filePath
|
||||
])
|
||||
if (cursor?.source === 'transcript' && cursor.sourceIdentity !== sourceIdentity) {
|
||||
throw sourceChanged()
|
||||
}
|
||||
const sessionAfterRead = args.runtime.getExactWorkerProviderSession(
|
||||
args.terminalHandle,
|
||||
observedAfter
|
||||
)
|
||||
if (
|
||||
!sessionAfterRead ||
|
||||
sessionAfterRead.processIncarnation !== session.processIncarnation ||
|
||||
sessionAfterRead.agent !== session.agent ||
|
||||
sessionAfterRead.providerSession.key !== session.providerSession.key ||
|
||||
sessionAfterRead.providerSession.id !== session.providerSession.id ||
|
||||
sessionAfterRead.providerSession.transcriptPath !== session.providerSession.transcriptPath
|
||||
) {
|
||||
throw sourceChanged()
|
||||
}
|
||||
const nextCursor = encodeWorkerOutputCursor(
|
||||
args.dispatchId,
|
||||
'transcript',
|
||||
sourceIdentity,
|
||||
transcript.nextOffset
|
||||
)
|
||||
return {
|
||||
dispatchId: args.dispatchId,
|
||||
source: 'transcript',
|
||||
sourceIdentity,
|
||||
provider: session.agent,
|
||||
transcript: {
|
||||
messages: transcript.messages,
|
||||
nextCursor,
|
||||
limited: transcript.limited,
|
||||
returnedMessageCount: transcript.messages.length
|
||||
},
|
||||
cursor: nextCursor,
|
||||
status: { worker: args.workerState, terminal: args.terminalStatus },
|
||||
fallbackReason: null,
|
||||
warnings: transcript.warnings
|
||||
}
|
||||
}
|
||||
|
||||
async function readTerminalOutput(
|
||||
args: Parameters<typeof readExactWorkerOutput>[0],
|
||||
cursor: ReturnType<typeof decodeWorkerOutputCursor>
|
||||
): Promise<OrchestrationWorkerReadResult> {
|
||||
const processIncarnation = args.runtime.getTerminalProcessIncarnation(args.terminalHandle)
|
||||
const paneKey = args.runtime.getTerminalPaneKey(args.terminalHandle)
|
||||
if (!processIncarnation || !paneKey) {
|
||||
throw new OrchestrationError(
|
||||
'worker_identity_changed',
|
||||
`Worker Dispatch ${args.dispatchId} no longer resolves to its exact process.`
|
||||
)
|
||||
}
|
||||
const sourceIdentity = createWorkerOutputSourceIdentity(['terminal', processIncarnation, paneKey])
|
||||
if (
|
||||
cursor?.source === 'terminal' &&
|
||||
cursor.sourceIdentity !== null &&
|
||||
cursor.sourceIdentity !== sourceIdentity
|
||||
) {
|
||||
throw sourceChanged()
|
||||
}
|
||||
const terminal = await args.runtime.readTerminal(args.terminalHandle, {
|
||||
cursor: cursor?.source === 'terminal' ? cursor.position : undefined,
|
||||
limit: args.limit
|
||||
})
|
||||
const redactedTerminal = redactWorkerTerminalLines(terminal.tail)
|
||||
const position =
|
||||
terminal.nextCursor !== null && /^\d+$/.test(terminal.nextCursor)
|
||||
? Number.parseInt(terminal.nextCursor, 10)
|
||||
: null
|
||||
const nextCursor =
|
||||
position === null
|
||||
? null
|
||||
: encodeWorkerOutputCursor(args.dispatchId, 'terminal', sourceIdentity, position)
|
||||
return {
|
||||
dispatchId: args.dispatchId,
|
||||
source: 'terminal',
|
||||
sourceIdentity,
|
||||
terminal: { ...terminal, tail: redactedTerminal.lines },
|
||||
cursor: nextCursor,
|
||||
status: { worker: args.workerState, terminal: terminal.status },
|
||||
fallbackReason: null,
|
||||
warnings: redactedTerminal.warnings
|
||||
}
|
||||
}
|
||||
|
||||
async function fallbackOrThrow(
|
||||
args: Parameters<typeof readExactWorkerOutput>[0],
|
||||
reason: OrchestrationWorkerReadFallbackReason,
|
||||
warnings: string[] = []
|
||||
): Promise<OrchestrationWorkerReadResult> {
|
||||
if (args.source === 'transcript') {
|
||||
throw transcriptRequired(args.dispatchId, reason)
|
||||
}
|
||||
const fallback = await readTerminalOutput(args, null)
|
||||
return fallback.source === 'terminal'
|
||||
? {
|
||||
...fallback,
|
||||
fallbackReason: reason,
|
||||
warnings: [...new Set([...fallback.warnings, ...warnings])]
|
||||
}
|
||||
: fallback
|
||||
}
|
||||
|
||||
function assertCursorSourceMatchesRequest(
|
||||
cursorSource: 'terminal' | 'transcript' | undefined,
|
||||
requestedSource: OrchestrationWorkerReadSource
|
||||
): void {
|
||||
if (cursorSource && requestedSource !== 'auto' && cursorSource !== requestedSource) {
|
||||
throw new OrchestrationError(
|
||||
'cursor_invalid',
|
||||
`The worker-read cursor is pinned to ${cursorSource} output.`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function orchestrationTimestampToMs(value: string): number {
|
||||
const normalized = /^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}/.test(value)
|
||||
? `${value.replace(' ', 'T')}Z`
|
||||
: value
|
||||
const parsed = Date.parse(normalized)
|
||||
return Number.isFinite(parsed) ? parsed : 0
|
||||
}
|
||||
|
||||
function sourceChanged(): OrchestrationError {
|
||||
return new OrchestrationError(
|
||||
'source_changed',
|
||||
'The worker output source changed. Start a fresh worker-read without the old cursor.'
|
||||
)
|
||||
}
|
||||
|
||||
function transcriptRequired(
|
||||
dispatchId: string,
|
||||
reason: OrchestrationWorkerReadFallbackReason
|
||||
): OrchestrationError {
|
||||
return new OrchestrationError(
|
||||
'transcript_required',
|
||||
`Structured output is unavailable for Dispatch ${dispatchId}: ${reason}.`,
|
||||
{ reason }
|
||||
)
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
import type { OrchestrationDb } from '../../orchestration/db'
|
||||
import {
|
||||
applyWaitForSetupOutcome,
|
||||
type WorkerEffect,
|
||||
type WorkerSetupReceipt
|
||||
} from './orchestration-worker-topology'
|
||||
|
||||
function residualWorkerEffects(effects: WorkerEffect[]): WorkerEffect[] {
|
||||
return effects.filter(
|
||||
(effect) => effect.action?.startsWith('created') || effect.action === 'reused_agent_terminal'
|
||||
)
|
||||
}
|
||||
|
||||
type WorkerSetupStageArgs = {
|
||||
db: OrchestrationDb
|
||||
dispatchId: string
|
||||
worktreeId: string
|
||||
terminalHandle: string
|
||||
setup: WorkerSetupReceipt
|
||||
effects: WorkerEffect[]
|
||||
}
|
||||
|
||||
export function persistWorkerReadinessStage(args: WorkerSetupStageArgs): void {
|
||||
args.db.recordWorkerStage({
|
||||
dispatchId: args.dispatchId,
|
||||
stage: 'terminal_readying',
|
||||
worktreeId: args.worktreeId,
|
||||
terminalHandle: args.terminalHandle,
|
||||
setupState: args.setup.state,
|
||||
effects: args.effects,
|
||||
residualResources: residualWorkerEffects(args.effects)
|
||||
})
|
||||
}
|
||||
|
||||
export function persistGatedSetupSpawnFailure(args: WorkerSetupStageArgs): boolean {
|
||||
if (args.setup.startupPolicy !== 'wait-for-setup' || args.setup.state !== 'spawn_failed') {
|
||||
return false
|
||||
}
|
||||
args.db.recordWorkerStage({
|
||||
dispatchId: args.dispatchId,
|
||||
stage: 'setup_start',
|
||||
worktreeId: args.worktreeId,
|
||||
terminalHandle: args.terminalHandle,
|
||||
setupState: args.setup.state,
|
||||
effects: args.effects,
|
||||
residualResources: residualWorkerEffects(args.effects)
|
||||
})
|
||||
return true
|
||||
}
|
||||
|
||||
export function persistWorkerSetupWaitOutcome(
|
||||
args: WorkerSetupStageArgs & { wait: { satisfied: boolean; status: string } }
|
||||
): void {
|
||||
applyWaitForSetupOutcome(args.setup, args.effects, args.wait)
|
||||
if (args.setup.startupPolicy !== 'wait-for-setup') {
|
||||
return
|
||||
}
|
||||
args.db.recordWorkerStage({
|
||||
dispatchId: args.dispatchId,
|
||||
stage: args.setup.state === 'failed' ? 'setup_failed' : 'setup_settled',
|
||||
worktreeId: args.worktreeId,
|
||||
terminalHandle: args.terminalHandle,
|
||||
setupState: args.setup.state,
|
||||
effects: args.effects,
|
||||
residualResources: residualWorkerEffects(args.effects)
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
import type { OrchestrationDb } from '../../orchestration/db'
|
||||
import {
|
||||
isUnknownWorkerStartOutcome,
|
||||
type WorkerSetupReceipt
|
||||
} from './orchestration-worker-topology'
|
||||
|
||||
export function failWorkerStartWithReceipt(args: {
|
||||
db: OrchestrationDb
|
||||
runId: string
|
||||
taskId: string
|
||||
dispatchId: string
|
||||
failedStage: string
|
||||
error: unknown
|
||||
setup: WorkerSetupReceipt
|
||||
}): unknown {
|
||||
const reason = args.error instanceof Error ? args.error.message : String(args.error)
|
||||
const unknown = isUnknownWorkerStartOutcome(args.error, args.failedStage)
|
||||
const worker = unknown
|
||||
? args.db.markWorkerStartUnknown(args.dispatchId, args.failedStage, reason)
|
||||
: args.db.failWorkerStart(args.dispatchId, args.failedStage, reason)
|
||||
return {
|
||||
runId: args.runId,
|
||||
taskId: args.taskId,
|
||||
dispatchId: args.dispatchId,
|
||||
state: worker.state === 'start_unknown' ? 'outcome_unknown' : worker.state,
|
||||
stage: worker.stage,
|
||||
failedStage: args.failedStage,
|
||||
lastError: reason,
|
||||
setup: args.setup,
|
||||
effects: JSON.parse(worker.effects) as unknown[],
|
||||
residualResources: JSON.parse(worker.residual_resources) as unknown[],
|
||||
...(unknown
|
||||
? {
|
||||
nextCommands: [
|
||||
`orca orchestration worker-show --dispatch ${args.dispatchId} --json`,
|
||||
`orca orchestration worker-abandon --dispatch ${args.dispatchId} --json`
|
||||
]
|
||||
}
|
||||
: {})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
import { z } from 'zod'
|
||||
import { OptionalFiniteNumber, OptionalString, requiredString } from '../schemas'
|
||||
|
||||
export const WorkerStartParams = z.object({
|
||||
task: requiredString('Missing --task'),
|
||||
on: OptionalString,
|
||||
run: OptionalString,
|
||||
from: requiredString('Missing --from'),
|
||||
worktree: OptionalString,
|
||||
name: OptionalString,
|
||||
repo: OptionalString,
|
||||
baseBranch: OptionalString,
|
||||
displayName: OptionalString,
|
||||
comment: OptionalString,
|
||||
setup: z.enum(['run', 'skip', 'inherit']).optional(),
|
||||
terminal: OptionalString,
|
||||
agent: OptionalString,
|
||||
retryOf: OptionalString,
|
||||
timeoutMs: OptionalFiniteNumber,
|
||||
devMode: z.boolean().optional()
|
||||
})
|
||||
|
||||
export type WorkerStartInput = z.infer<typeof WorkerStartParams>
|
||||
@@ -0,0 +1,148 @@
|
||||
import { z } from 'zod'
|
||||
import { OrchestrationError } from '../../orchestration/orchestration-error'
|
||||
import { syncFederatedDispatch } from '../../orchestration/federation-sync'
|
||||
import { defineMethod, type RpcMethod } from '../core'
|
||||
import { requiredString } from '../schemas'
|
||||
import {
|
||||
inspectWorkerTerminal,
|
||||
resolvePinnedFederatedServer
|
||||
} from './orchestration-worker-observation'
|
||||
|
||||
const WorkerDispatchParams = z.object({ dispatch: requiredString('Missing --dispatch') })
|
||||
|
||||
export const ORCHESTRATION_WORKER_STOP_METHODS: RpcMethod[] = [
|
||||
defineMethod({
|
||||
name: 'orchestration.workerStop',
|
||||
params: WorkerDispatchParams,
|
||||
handler: async (params, { runtime, orchestrationMutation }) => {
|
||||
const db = runtime.getOrchestrationDb()
|
||||
const federated = db.getFederatedDispatch(params.dispatch)
|
||||
if (federated) {
|
||||
if (!orchestrationMutation) {
|
||||
throw new OrchestrationError(
|
||||
'invalid_argument',
|
||||
'Remote worker-stop requires a durable retry request.'
|
||||
)
|
||||
}
|
||||
const server = resolvePinnedFederatedServer(runtime, federated)
|
||||
const begun = db.beginWorkerStop(params.dispatch)
|
||||
if (begun.disposition === 'already_settled') {
|
||||
return settledReceipt(params.dispatch, begun.worker.state)
|
||||
}
|
||||
try {
|
||||
const remote = (await runtime.callOrchestrationWorkerServer(
|
||||
server.environmentId,
|
||||
'orchestration.federationStop',
|
||||
{ dispatchId: params.dispatch },
|
||||
30_000,
|
||||
{ orchestrationRequestId: orchestrationMutation.requestId }
|
||||
)) as RemoteStopReceipt
|
||||
if (remote.state === 'stopped') {
|
||||
const worker = db.reconcileFederatedWorkerStop(params.dispatch)
|
||||
return {
|
||||
dispatchId: params.dispatch,
|
||||
state: worker.state,
|
||||
alreadySettled: remote.alreadySettled,
|
||||
processAction: remote.processAction,
|
||||
close: remote.close
|
||||
}
|
||||
}
|
||||
if (remote.state === 'succeeded' || remote.state === 'failed') {
|
||||
db.resumeFederatedWorkerForTerminalRelay(params.dispatch)
|
||||
await syncFederatedDispatch(runtime, params.dispatch).catch(() => undefined)
|
||||
return {
|
||||
dispatchId: params.dispatch,
|
||||
state: db.getWorkerDispatch(params.dispatch)?.state ?? remote.state,
|
||||
alreadySettled: true,
|
||||
processAction: 'none'
|
||||
}
|
||||
}
|
||||
return unknownReceipt(
|
||||
params.dispatch,
|
||||
db.markWorkerStopUnknown(
|
||||
params.dispatch,
|
||||
remote.lastError ?? `The worker server returned ${remote.state}.`
|
||||
),
|
||||
remote.processAction
|
||||
)
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error)
|
||||
return unknownReceipt(
|
||||
params.dispatch,
|
||||
db.markWorkerStopUnknown(params.dispatch, reason),
|
||||
'unknown'
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
const begun = db.beginWorkerStop(params.dispatch)
|
||||
if (begun.disposition === 'already_settled') {
|
||||
return settledReceipt(params.dispatch, begun.worker.state)
|
||||
}
|
||||
const handle = begun.worker.agent_terminal_handle
|
||||
if (!handle) {
|
||||
return unknownReceipt(
|
||||
params.dispatch,
|
||||
db.markWorkerStopUnknown(params.dispatch, 'The Dispatch has no recorded agent terminal.'),
|
||||
'unknown'
|
||||
)
|
||||
}
|
||||
const observation = await inspectWorkerTerminal(runtime, db, params.dispatch)
|
||||
if (!observation.exact || observation.status !== 'running') {
|
||||
return unknownReceipt(
|
||||
params.dispatch,
|
||||
db.markWorkerStopUnknown(
|
||||
params.dispatch,
|
||||
`The recorded worker process is ${observation.status}; no terminal was closed.`
|
||||
),
|
||||
'none'
|
||||
)
|
||||
}
|
||||
try {
|
||||
const close = await runtime.closeTerminal(handle)
|
||||
const worker = db.settleWorkerStop(params.dispatch)
|
||||
runtime.notifyMessageArrived(`dispatch:${params.dispatch}`, 'status')
|
||||
return {
|
||||
dispatchId: params.dispatch,
|
||||
state: worker.state,
|
||||
alreadySettled: false,
|
||||
processAction: 'closed_agent_terminal',
|
||||
close
|
||||
}
|
||||
} catch (error) {
|
||||
const reason = error instanceof Error ? error.message : String(error)
|
||||
return unknownReceipt(
|
||||
params.dispatch,
|
||||
db.markWorkerStopUnknown(params.dispatch, reason),
|
||||
'unknown'
|
||||
)
|
||||
}
|
||||
}
|
||||
})
|
||||
]
|
||||
|
||||
type RemoteStopReceipt = {
|
||||
state: string
|
||||
alreadySettled: boolean
|
||||
processAction: string
|
||||
close?: unknown
|
||||
lastError?: string | null
|
||||
}
|
||||
|
||||
function settledReceipt(dispatchId: string, state: string) {
|
||||
return { dispatchId, state, alreadySettled: true, processAction: 'none' }
|
||||
}
|
||||
|
||||
function unknownReceipt(
|
||||
dispatchId: string,
|
||||
worker: { state: string; last_error: string | null },
|
||||
processAction: string
|
||||
) {
|
||||
return {
|
||||
dispatchId,
|
||||
state: worker.state,
|
||||
alreadySettled: false,
|
||||
processAction,
|
||||
lastError: worker.last_error
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user