diff --git a/skill-guides/orchestration.md b/skill-guides/orchestration.md index 4e65a07e733..ef8181dd4e3 100644 --- a/skill-guides/orchestration.md +++ b/skill-guides/orchestration.md @@ -176,6 +176,33 @@ Dispatch rules: - After 3 consecutive failures on one task, the dispatch context circuit-breaks and the task is marked failed. - Use `task-list --brief --json` for coordinator sweeps; it collapses whitespace and caps each echoed spec at 160 characters (`spec_truncated` marks shortened rows). Omit `--brief` when the full spec is required, or when an older CLI rejects it as an unknown flag. +## How deep workers can nest + +A dispatched worker normally cannot dispatch sub-workers. Attempting it fails with +`nested_worker_depth_exceeded` and a message telling the worker to complete the task +itself. Do that — do not try to route around it. + +The limit is a number, not an on/off switch. `Settings -> Agents -> Nested worker depth` +sets how many generations are allowed: + +- `1` (default): a coordinator dispatches workers; those workers do not dispatch. +- `2`: workers may dispatch one further generation. + +Depth is counted from the terminal that issues the command, not from the Run. Creating a +new Run does not reset it — a worker that runs `run-create` then `worker-start` is still a +worker, and still counted. This is the part that changed: the old behaviour rejected +sub-dispatch only because a worker's terminal was not bound to a Run, so creating a Run was +enough to slip past it. + +Two limits worth knowing: + +- **It is a guardrail, not a security boundary.** A caller that declares another terminal's + handle while its own launch evidence is unverifiable (an ordinary restored terminal, for + example) can be counted as that terminal instead. Orca does not treat workers as hostile. +- **It applies while a Dispatch is active.** After `worker_done`, or after a coordinator + settles the task, the terminal is no longer a worker and is counted as a root again. The + process may still be alive; that is the documented boundary, not an accident. + ## Preferred Supervised Worker Loop Use `worker-start` for the normal supervised path. It composes the existing worktree, terminal, readiness, and dispatch primitives while returning exact created/reused effects. Agents still choose placement and concurrency; Orca does not schedule workers or infer conflicts. diff --git a/src/cli/bundled-skill-guides.ts b/src/cli/bundled-skill-guides.ts index 33f3e18f3a7..66f55c065e7 100644 --- a/src/cli/bundled-skill-guides.ts +++ b/src/cli/bundled-skill-guides.ts @@ -30,7 +30,7 @@ const ORCA_LINEAR_MARKDOWN = "---\nname: orca-linear\ndescription: >-\n Use Orc const ORCA_PER_WORKSPACE_ENV_MARKDOWN = "---\nname: orca-per-workspace-env\ndescription: >-\n Set up, review, debug, or validate Orca per-workspace environment recipes —\n on-demand, disposable runtimes (cloud sandboxes, VMs, or local) created fresh\n for each workspace. Covers first-time setup (provider prerequisites, the\n reusable base snapshot, the coding-agent auth snapshot, credentials, and\n state), not just the per-workspace lifecycle scripts. Use to stand up\n per-workspace environments, fix an `environmentRecipes` entry in `orca.yaml`, scaffold\n provider lifecycle scripts, or resolve an `orca vm recipe doctor` failure.\n---\n\n# Per-Workspace Environments\n\nHelp a user stand up and maintain a repo-owned per-workspace environment recipe end to end. Each\nworkspace gets its own on-demand, disposable runtime (a cloud sandbox, a VM, or a local one),\ncreated fresh and torn down after.\n\nOrca is a **thin wrapper**: you guide, detect, and scaffold; you never own the user's cloud account,\nbilling, images, or credentials.\n\n- **You DO:** sequence the setup, detect what's detectable (provider CLI present/logged-in? recipe\n present? `doctor` passing?), scaffold provider-templated scripts the user fills in, drive the slow\n snapshot/auth phases with the user, and always show the next action.\n- **You DO NOT:** create accounts, choose plans/regions, invent org/project/scope ids, store or print\n secrets, or run anything that spends money without an explicit user OK.\n\nFirst-time setup has **four phases before the per-workspace recipe runs** — easy to miss, so walk\nthem in order:\n\n1. **Prerequisites** — cloud account, provider CLI, scope/project, plan limits, git token (§2).\n2. **Base snapshot** — reusable image: tools + repo + headless build, snapshotted once (§3).\n3. **Agent-auth snapshot** — boot the base, run interactive device-auth, re-snapshot (§4).\n4. **State** — thread snapshot id / scope / project / port between phases via a state file (§6).\n\nThen the **per-workspace contract** (create/suspend/resume/destroy) runs fast (§8).\n\n**The one branch that shapes everything — connection mode:** **Orca-server** (`create` runs `orca serve`\nin the env and emits a `pairingCode`; §7c/§7f) vs **SSH** (`create` runs no server and emits a\n`connection.type:\"ssh\"` block Orca dials into; §7g/§7h). Settle this first — it changes the `create`\noutput shape and half the templates.\n\nKeep Orca's checkout behavior unchanged by default: omit `checkoutMode`, emit schema version 1, and\nlet Orca create a linked worktree. Only use `checkoutMode: provisioned-root` when the user explicitly\nwants one ephemeral machine to clone the finished workspace itself. This niche mode currently requires\ndirect SSH, an ordinary non-bare/non-sparse primary checkout at `projectRoot`, and schema version 2.\n\n**Quick-start (happy path):** interview the user (connection mode Orca-server vs SSH, provider, agent CLI,\ngit auth — §1.2) + read the provider's CLI docs → scaffold `scripts/orca-vm/` from §7 → run the\nbase-snapshot script, then the auth script (you invoke these by hand; not via `orca.yaml`) → wire\n`environmentRecipes` in `orca.yaml` → `orca vm recipe doctor --json` (free) → then the `--provision`\nself-test loop (§9) until it passes.\n\n---\n\n## 1. Setup workflow\n\nDrive these with the user. **[CHECKPOINT]** steps need explicit confirmation — they spend money, take\na long time, or need the user at the keyboard. Never create an Orca workspace or commit unless asked.\n\n1. **Inspect the repo** for an existing `environmentRecipes` entry, `scripts/orca-vm/`, a state file, or setup\n notes. If a working recipe exists, jump to Doctor (§9) instead of rebuilding.\n2. **Interview the user up front** — gather these choices and confirm them back before scaffolding\n anything. Don't pick for them (§11); don't guess.\n - **Connection mode:** how Orca attaches to the environment — an **Orca server** (the VM runs\n `orca serve` and Orca pairs over its pairing URL; worked example §7f) or **SSH** (Orca connects to\n the host over SSH; §7g). This decides the recipe's connection shape, so settle it first.\n - **Checkout ownership:** do not ask by default. Only when the user requires the environment to\n create the exact final checkout, confirm `provisioned-root` and direct SSH; otherwise omit it.\n - **Provider:** Vercel Sandbox, Fly, Modal, an existing SSH host, … For non-obvious providers, also\n ask scope/project/region and plan limits (§2). Then **read that provider's CLI/SDK docs** (or\n ` --help`) before scaffolding — you need its exact create/exec/snapshot/remove verbs.\n If a provider advertises `ssh`, verify whether it exposes a real dialable SSH target\n (host/port/user/key or proxy command) or only a provider-mediated interactive shell; Orca SSH mode\n needs the former.\n - **Coding-agent CLI + account:** which agent runs in the VM (`codex`, `claude`, …) and that the user\n has an account for it — it gets logged in during the Phase-3 auth snapshot (§4).\n - **Git auth:** the token source for cloning a private repo (`GH_TOKEN`/`GITHUB_TOKEN` or `gh auth\n token`; §5).\n3. **Check prerequisites (§2)** — detect the provider CLI + auth and confirm the items above are in\n place before any paid step.\n4. **Scaffold scripts + state file** from §7 (worked Vercel example: §7f; SSH host: §7g; Docker SSH:\n §7h; Windows: §7i), filling in the provider's real commands. Make them executable.\n5. **[CHECKPOINT] Build the base snapshot (§3)** — paid, slow.\n6. **[CHECKPOINT] Authenticate the agent (§4)** — interactive; the user follows a URL/code. **You cannot\n drive this step** — you run commands non-interactively, so there's no TTY for `docker exec -it` /\n `ssh -t` to prompt against. The **user** runs the Phase-3 login in their own terminal (or via the\n Claude Code harness bang-prefix — `! `, with the required space after `!`); you scaffold and drive\n the non-interactive phases around it. After kicking it off, **ask the user to report back once the login\n finishes** — you can't observe it completing, and you need that confirmation before resuming the\n non-interactive steps (base/auth commit, doctor, provision).\n7. **Wire the recipe** so `orca.yaml` points create/suspend/resume/destroy at the scripts (§8). The\n workspace composer reads `environmentRecipes` from the project's primary checkout of `orca.yaml`, **not** from\n a feature branch or worktree. So a recipe added only on a branch won't appear as a \"Run on\" option\n until that `orca.yaml` change is committed and merged to the project's primary branch. Tell the user\n this up front: `doctor`/`--provision` validate the scripts from the working copy on any branch, but\n creating a workspace from the recipe in the picker needs it on primary.\n8. **Dry-run doctor** — `orca vm recipe doctor --repo-path --json` (free, static; §9).\n Fix every failure before going live.\n9. **[CHECKPOINT] Live self-test** — get the user's OK once, then run\n `orca vm recipe doctor --provision --json` as a loop: it runs create → validates →\n destroys, and on failure returns a full transcript. Read it, fix the scripts, and re-run yourself until\n it passes (§9). Spends cloud money; the one approval covers the loop.\n10. **[CHECKPOINT] Optional workspace test** — only if asked: create a workspace via the picker, then\n verify sleep/wake/delete.\n\n---\n\n## 2. Phase 1 — Prerequisites\n\nThe user's responsibility; verify what's verifiable, ask for the rest, invent nothing. State which\nitems you verified vs. which the user asserted.\n\n- **Connection mode** (Orca server vs SSH) confirmed with the user — see §1 step 2; it shapes the recipe.\n- **Cloud account + plan** that allows sandboxes/VMs. Ask.\n- **Provider CLI installed + authenticated** — detect (`command -v `), check auth (e.g.\n `vercel whoami`). If missing, point at the provider's docs; don't log them in.\n- **Scope / project / region** the sandboxes live under. Ask; flows into every script via state.\n- **Plan / timeout / RAM caps.** Record them — e.g. Vercel Hobby caps sandbox timeout at **45m**,\n which limits both the base build and per-workspace runtime (see §10).\n- **Git token for private repos** (`GH_TOKEN`/`GITHUB_TOKEN`, or the provider's git auth; can fall back\n to `gh auth token`). See §5.\n- **Coding-agent CLI choice** (`codex`, `claude`…) and that the user has an account — it gets\n authenticated into the VM in Phase 3.\n\n---\n\n## 3. Phase 2 — Base snapshot (the reusable image)\n\nBuild **once**, snapshot, and every workspace boots from it in seconds instead of rebuilding.\nProvisioning + building takes a while (often ~20–30 min), so it runs behind a checkpoint. The script\nshape is §7a; key points:\n\n- Build the **headless Electron main only** (not the renderer) so it fits in plan RAM.\n- Use the VM image's package manager (`apt`/`dnf`/`apk`, per the base distro — not the provider brand).\n- Clone with the git token via `GIT_ASKPASS` (§5).\n- **Trap errors and remove the half-built sandbox** so a crash doesn't leave a paid resource running.\n- Snapshot the stopped sandbox, parse the snapshot id, and write it + scope/project/port/repo to state.\n\n---\n\n## 4. Phase 3 — Agent-auth snapshot (interactive)\n\nThe base snapshot has the agent CLI installed but **not logged in**, and per-workspace VMs are\nephemeral — so authenticate once and bake it into a second snapshot layer. Script shape is §7b:\n\n1. Boot a sandbox from the base `snapshotId` (from state).\n2. Run the agent's login **interactively** (`--interactive --tty`); the user completes the URL/code in\n their browser. On a **headless VM this must be the device-auth flow** (e.g. `codex login --device-auth`),\n **not** plain `codex login`: the default OAuth login starts a loopback callback server on a container\n port the host browser can't reach, so it hangs. Device-auth instead prints a URL + code the user opens\n on the **host**.\n3. Verify login; **refuse to snapshot an unauthenticated VM.** Prefer the status command's **exit code**\n (most agent CLIs exit non-zero when unauthenticated). If you grep instead, agent status often goes to\n **stderr** (e.g. `codex login status` prints \"Logged in using ChatGPT\" there), so **fold stderr first**\n (`... 2>&1 | grep …`) and match the agent's **exact success line** — never `grep -qi 'logged in'`, which\n also matches \"**not** logged in\" and would commit an unauthenticated image.\n4. Re-snapshot, parse the new id, and overwrite `snapshotId` in state to the authenticated image\n (recording `authSourceSnapshotId`). Remove the auth sandbox.\n\n**You can't drive step 2 yourself** (you run commands non-interactively — no TTY). The **user** runs it in\ntheir own terminal, or via the Claude Code harness bang-prefix (`! `, with the required space after\n`!`). You scaffold/boot the sandbox and run steps 3–4, but **you cannot observe the interactive login\nfinishing** — so **ask the user to tell you when it's done** before you verify and re-snapshot.\n\nIf the agent's credentials are short-lived, warn that the snapshot may need periodic re-auth (§10).\n\nFor disposable runtimes, do **not** treat a host agent config directory (for example `~/.codex`) as the\nauth snapshot by bind-mounting or copying it wholesale. Agent homes often contain sqlite state, hook\napproval state, caches, logs, and host-specific env/config. Instead, authenticate/configure the agent\ninside the disposable runtime and snapshot/commit that runtime layer.\n\n---\n\n## 5. Credentials\n\n- **Never** commit secrets or put them in `userData`, recipe JSON, comments, docs, or the state file.\n- **Git token:** read from env (`GH_TOKEN`/`GITHUB_TOKEN`), falling back to `gh auth token`. Pass to the\n VM only via the provider's ephemeral `--env`. Inside the VM, use a `GIT_ASKPASS` helper with\n `x-access-token` (not the token in the clone URL) and `GIT_TERMINAL_PROMPT=0` so a missing token fails\n fast instead of hanging. When you write the helper from inside `bash -lc` under `set -u`, escape the\n positional arg and the token (`\\$1`, `\\$GH_TOKEN`) so they land **literally** and resolve at git-runtime\n — an unescaped `$1` aborts with \"unbound variable\", and a literal `$GH_TOKEN` keeps the real token out of\n the written file. `rm -f` the helper after the clone/fetch.\n- **Provider auth:** rely on the provider CLI's logged-in session, not checked-in keys.\n- **Agent auth:** lives in the authenticated snapshot (Phase 3) — never a file you write or commit.\n- State holds only **non-secret** wiring (snapshot ids, scope, project, port, repo url/ref).\n\n---\n\n## 6. State file\n\nA repo-local JSON file (e.g. `scripts/orca-vm/-state.json`) threads non-secret values between\nphases. Each script resolves values as **env var → state → built-in fallback**, and merges its outputs\nback. Phase 2 writes the base `snapshotId`; Phase 3 overwrites it with the authenticated snapshot;\nper-workspace `create` boots from `snapshotId`.\n\n```json\n{\n \"baseName\": \"orca-base\",\n \"snapshotId\": \"snap_authenticated_image_id\",\n \"authSourceSnapshotId\": \"snap_base_image_id\",\n \"scope\": \"\",\n \"project\": \"\",\n \"port\": 7331,\n \"repoUrl\": \"https://host/org/repo.git\",\n \"repoRef\": \"main\",\n \"projectRoot\": \"/abs/path/on/remote/repo\"\n}\n```\n\n---\n\n## 7. Script templates (provider-agnostic shapes)\n\nScaffold under `scripts/orca-vm/`. These are **shapes** — fill in the provider's real commands. All\nreserve stdout for the final JSON and log progress to stderr. Include a shared `json_value ` /\n`env_value ` reader (env → state → fallback) in each.\n\n**Where each script runs:**\n\n- **Local-side** (`create`/`suspend`/`resume`/`destroy` + the base-snapshot/auth scripts the user\n invokes) runs **on the user's desktop**, so it must run on their OS. macOS/Linux: `#!/usr/bin/env\n bash`, `set -euo pipefail`, quoted paths. **Windows:** a bare `.sh` won't run — scaffold `.ps1`/`.cmd`\n or require WSL/Git-Bash and point `orca.yaml` at the right launcher.\n- **Remote-side** (commands you `exec` *inside* the Linux VM) always runs in the VM's Linux shell, so\n bash is fine there regardless of the user's OS.\n\n### 7a. Base-snapshot (`-base-snapshot.sh`) — Phase 2\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\n# resolve base_name/repo_url/repo_ref/project_root/port/scope/project/timeout (env→state→fallback)\n# resolve gh token: GH_TOKEN | GITHUB_TOKEN | `gh auth token`\n# 1. provision a sandbox (timeout/vcpus/published port/snapshot retention); trap: remove on error\n# 2. remote exec (long timeout): install pkgs + gh + corepack/pnpm + agent CLI;\n# clone with GIT_ASKPASS(token); write headless main-only build config;\n# dev setup; pnpm install; build CLI; build headless electron main; smoke-check tools\n# 3. snapshot stopped sandbox; parse snapshot id (fail if unparseable)\n# 4. merge { baseName, snapshotId, projectRoot, repoUrl, repoRef, port, scope, project } into state\n# print only the state JSON to stdout\n```\n\nWorked Vercel commands for this phase are in §7f. You run this script by hand (not via `orca.yaml`),\nafter exporting the first-run inputs the state file doesn't have yet — e.g. provider scope/project, the\nrepo URL/ref, and a git token (`GH_TOKEN`); later runs read them back from state.\n\n### 7b. Auth (`-base-auth.sh`) — Phase 3\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\n# read source snapshot from state.snapshotId (fail if absent); auth_name=\"${base_name}-auth\"\n# 1. boot sandbox from source snapshot; trap: remove on error\n# 2. INTERACTIVE/TTY remote exec: agent login — user completes URL/code. Headless VM: MUST use the\n# device-auth flow (e.g. `codex login --device-auth`) — plain OAuth login binds a loopback callback\n# port the host can't reach and hangs. User runs this themselves (you have no interactive TTY); ask\n# them to report back when it's done before continuing.\n# 3. verify login, then refuse to snapshot if not logged in. Prefer the status command's EXIT CODE (most\n# agent CLIs exit non-zero when unauthenticated) over string-matching. If you must grep, fold stderr\n# first (`status 2>&1 | grep …` — many agents print the success line there) and match the agent's exact\n# success line; never `grep -qi 'logged in'`, which also matches \"not logged in\". Codex example: §7f.\n# 4. snapshot; parse new id\n# 5. merge { snapshotId:, authSourceSnapshotId: } into state; remove auth sandbox\n# print only the state JSON to stdout\n```\n\n### 7c. Create (`-create.sh`) — per workspace\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\n# read authenticated snapshotId/scope/project/port/repo*/project_root (env→state→fallback)\n# fail clearly if snapshotId is missing (point back to Phases 2–3)\n# name = orca-${ORCA_RECIPE_ID}-${ORCA_VM_INSTANCE_ID} (sanitized, length-capped)\n# 1. boot sandbox from snapshotId with a published port; capture the public URL → pairing address\n# (an externally reachable wss:// URL); trap: remove sandbox on error\n# 2. remote exec: ensure repo at desired commit; rebuild only if commit changed (cache marker)\n# 3. remote exec: start orca serve in the background and read the recipe JSON it writes (see below)\n# 4. print serve's JSON to stdout, optionally enriched with userData:\n# { schemaVersion:1, pairingCode, projectRoot, userData:{ provider, resourceId:name, snapshotId } }\n```\n\n**The exact `orca serve` invocation and its output (verified — do not improvise the flags).** Inside the\nVM, run:\n\n```bash\norca serve \\\n --port \"$PORT\" \\\n --project-root \"$ABS_REPO_PATH_ON_REMOTE\" \\\n --pairing-address \"$EXTERNAL_WSS_URL\" \\\n --recipe-json\n```\n\n**Binary name:** in a VM built from source (the Phase-2 flow), run it as `pnpm exec orca-dev serve …`\nfrom the repo root — `orca-dev` is the in-repo entrypoint and is what the §7f example uses. Plain\n`orca serve …` is the same command when the built CLI is installed on the VM's PATH. The flags/output\nare identical either way.\n\nThere is **no `--host` flag**. `--project-root` must be an absolute directory on the remote. With\n`--recipe-json` the server **stays running** and prints exactly this single object to **stdout**, then\nkeeps serving:\n\n```json\n{ \"schemaVersion\": 1, \"pairingCode\": \"\", \"projectRoot\": \"\" }\n```\n\n`pairingCode` is the pairing URL, already pointing at whatever you passed as `--pairing-address` — so set\n`--pairing-address` to the externally reachable address and **pass `pairingCode` through unchanged; never\nhand-rewrite it**. Because serve runs in the foreground and doesn't exit, redirect its stdout to a file\nand poll until that file parses as JSON (and bail if the process dies — dump its stderr log). Your\n`create` script then prints that JSON (optionally merging `userData`). Concrete pattern: §7f.\n\n### 7d. Suspend / resume / destroy — per workspace\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\npayload=\"$(cat)\" # Orca passes lifecycle JSON on stdin\nresource_id=\"$(node -e 'const d=JSON.parse(process.argv[1]); process.stdout.write(d.recipeResult?.userData?.resourceId ?? \"\")' \"$payload\")\"\n[ -n \"$resource_id\" ] || { echo \"No resource id in lifecycle payload\" >&2; exit 1; }\n# suspend: provider suspend \"$resource_id\"\n# resume: provider resume \"$resource_id\"; then RE-EMIT fresh recipe JSON (pairing may change)\n# destroy: provider remove \"$resource_id\" (or set destroy: none in orca.yaml)\n```\n\n### 7e. State file — scaffold with scope/project/repo filled in and snapshot ids empty (§6).\n\n### 7f. Worked example — Vercel Sandbox (all three phases)\n\nA real, working shape (the Vercel surface is a CLI: `vercel sandbox create|exec|snapshot|remove`). Adapt\nnames; verify flags against `vercel sandbox --help` for the user's CLI version before relying on them.\nThese ground §7a (base snapshot) and §7b (auth), which are otherwise generic skeletons.\n\n**Phase 2 — base snapshot (§7a):** provision → install tools + clone + headless build → snapshot.\n\n```bash\n# provision a fresh build sandbox (retain a couple of snapshots); trap-remove on error\nvercel sandbox create --name \"$base\" --runtime node24 --timeout 30m --vcpus 4 --publish-port \"$port\" \\\n --snapshot-expiration 30d --keep-last-snapshots 2 \"${vercel_args[@]}\" >&2\n# remote build (long timeout): install pkgs+gh+pnpm+agent CLI, clone with GIT_ASKPASS (write the helper\n# with LITERAL \\$1/\\$GH_TOKEN so they resolve at git-runtime, not write-time — see §5/§7f create — then\n# `rm -f /tmp/askpass.sh`), write the headless main-only build config (drop the renderer), dev setup,\n# build CLI + headless main, smoke-check\nvercel sandbox exec \"$base\" \"${vercel_args[@]}\" --timeout 25m --env \"GH_TOKEN=$gh_token\" … -- bash -lc '…build…' >&2\n# snapshot the STOPPED sandbox and parse the id from CLI output (fail if unparseable)\nout=\"$(vercel sandbox snapshot \"$base\" --stop --expiration 30d \"${vercel_args[@]}\" 2>&1)\"; printf '%s\\n' \"$out\" >&2\nsnapshot_id=\"$(printf '%s\\n' \"$out\" | sed -nE 's/.*(snap_[A-Za-z0-9]+).*/\\1/p' | tail -1)\"\n# merge { baseName, snapshotId, scope, project, port, repoUrl, repoRef, projectRoot } into state; print state JSON\n```\n\n**Phase 3 — agent-auth snapshot (§7b):** boot the base, log the agent in interactively, re-snapshot.\n(`codex` below is an example — substitute the user's chosen agent's login/status verbs, e.g. `claude`.)\n\n```bash\nvercel sandbox create --name \"$auth\" --snapshot \"$snapshot_id\" --timeout 30m --publish-port \"$port\" \"${vercel_args[@]}\" >&2\n# INTERACTIVE — the USER runs this in their own terminal (you have no interactive TTY) and completes the\n# URL/code on the HOST. --device-auth is MANDATORY on a headless VM: plain `codex login` binds a loopback\n# callback port the host browser can't reach and hangs. Ask the user to report back when login finishes.\nvercel sandbox exec --interactive --tty \"$auth\" \"${vercel_args[@]}\" -- bash -lc 'codex login --device-auth'\n# refuse to snapshot an unauthenticated VM — fold stderr, match codex's exact success line (§4)\nvercel sandbox exec \"$auth\" \"${vercel_args[@]}\" --timeout 30s -- bash -lc 'codex login status 2>&1' | grep -Eqi 'Logged in using ChatGPT|Logged in via device' \\\n || { echo \"agent not logged in; not snapshotting\" >&2; exit 1; }\nout=\"$(vercel sandbox snapshot \"$auth\" --stop --expiration 30d \"${vercel_args[@]}\" 2>&1)\"; printf '%s\\n' \"$out\" >&2\nnew_id=\"$(printf '%s\\n' \"$out\" | sed -nE 's/.*(snap_[A-Za-z0-9]+).*/\\1/p' | tail -1)\"\n# overwrite state.snapshotId = new_id, record authSourceSnapshotId = snapshot_id; remove the auth sandbox\n```\n\n**Per-workspace `create`** (the fast path):\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\n# resolve from env→state→fallback: snapshot_id, scope, project, port, repo_url, repo_ref, project_root\nvercel_args=(); [ -n \"$scope\" ] && vercel_args+=(--scope \"$scope\"); [ -n \"$project\" ] && vercel_args+=(--project \"$project\")\n[ -n \"$snapshot_id\" ] || { echo \"snapshotId missing — run Phases 2–3 first\" >&2; exit 1; }\ngh_token=\"${GH_TOKEN:-${GITHUB_TOKEN:-$(command -v gh >/dev/null 2>&1 && gh auth token 2>/dev/null || true)}}\"\nrecipe_id=\"${ORCA_RECIPE_ID:-vercel-sandbox}\"\nrecipe_id=\"${recipe_id//./-}\" # Vercel names forbid dots.\ninstance_id=\"${ORCA_VM_INSTANCE_ID:-$(date +%s)}\"\nmax_recipe_id_length=$((128 - ${#instance_id} - 6)) # Preserve the unique instance suffix.\n[ \"$max_recipe_id_length\" -gt 0 ] || { echo \"ORCA_VM_INSTANCE_ID is too long for a Vercel sandbox name\" >&2; exit 1; }\nname=\"orca-${recipe_id:0:max_recipe_id_length}-${instance_id}\"\n\n# Arm cleanup BEFORE create so a failing create can't leak a half-built paid sandbox.\ncleanup_on_error() { [ \"$?\" -ne 0 ] && vercel sandbox remove \"$name\" \"${vercel_args[@]}\" >/dev/null 2>&1 || true; }\ntrap cleanup_on_error EXIT\n\n# 1. boot from the authenticated snapshot, publish the serve port\ncreate_output=\"$(vercel sandbox create --name \"$name\" --snapshot \"$snapshot_id\" \\\n --timeout 30m --publish-port \"$port\" \"${vercel_args[@]}\" 2>&1)\"; printf '%s\\n' \"$create_output\" >&2\n# Vercel prints the published https URL; derive the external wss:// pairing address from it\npublic_url=\"$(printf '%s\\n' \"$create_output\" | sed -nE 's#.*(https://[^[:space:]]+\\.vercel\\.run).*#\\1#p' | head -1)\"\n[ -n \"$public_url\" ] || { echo \"no published URL in create output\" >&2; exit 1; }\npairing_ws=\"${public_url/https:\\/\\//wss://}\"\n\n# 2. (remote) ensure the repo is at the right commit; rebuild only if the commit changed (cache marker)\nvercel sandbox exec \"$name\" \"${vercel_args[@]}\" --timeout 20m \\\n --env \"GH_TOKEN=$gh_token\" --env \"ORCA_PROJECT_ROOT=$project_root\" \\\n --env \"ORCA_REPO_URL=$repo_url\" --env \"ORCA_REPO_REF=$repo_ref\" \\\n -- bash -lc 'set -euo pipefail; cd \"$ORCA_PROJECT_ROOT\"; \\\n # Re-establish git auth for the private-repo fetch (why + full rationale: §5); else it hangs on a prompt.\n # Load-bearing escaping: \\$1 and \\$GH_TOKEN must land LITERALLY and resolve at git-runtime. Test after\n # any edit here — reformatting the nested printf/node quoting silently breaks the fetch or leaks the token.\n if [ -n \"${GH_TOKEN:-}\" ]; then \\\n printf \"%s\\n\" \"#!/usr/bin/env bash\" \"case \\\"\\$1\\\" in *Username*) echo x-access-token;; *Password*) echo \\\"\\$GH_TOKEN\\\";; esac\" > /tmp/askpass.sh; \\\n chmod 700 /tmp/askpass.sh; export GIT_ASKPASS=/tmp/askpass.sh GIT_TERMINAL_PROMPT=0; fi; \\\n git fetch origin \"$ORCA_REPO_REF\"; \\\n git checkout -B \"$ORCA_REPO_REF\" FETCH_HEAD; \\\n rm -f /tmp/askpass.sh; \\\n c=\"$(git rev-parse HEAD)\"; [ -f .orca-built ] && [ \"$(cat .orca-built)\" = \"$c\" ] || { \\\n pnpm install --prefer-offline && pnpm run build:cli && \\\n node config/scripts/run-electron-vite-build.mjs --config config/electron-vite.vm-serve.config.ts && \\\n printf \"%s\" \"$c\" > .orca-built; }' >&2\n\n# 3. (remote) start orca serve in the background, writing recipe JSON to a file; poll until it parses\nrecipe_json=\"$(vercel sandbox exec \"$name\" \"${vercel_args[@]}\" --timeout 60s \\\n --env \"ORCA_PORT=$port\" --env \"ORCA_PROJECT_ROOT=$project_root\" --env \"ORCA_PAIRING_ADDRESS=$pairing_ws\" \\\n -- bash -lc 'set -euo pipefail; cd \"$ORCA_PROJECT_ROOT\"; rm -f /tmp/orca-recipe.json /tmp/orca-serve.log; \\\n nohup pnpm exec orca-dev serve --port \"$ORCA_PORT\" --project-root \"$ORCA_PROJECT_ROOT\" \\\n --pairing-address \"$ORCA_PAIRING_ADDRESS\" --recipe-json >/tmp/orca-recipe.json 2>/tmp/orca-serve.log /dev/null 2>&1 && { cat /tmp/orca-recipe.json; exit 0; }; \\\n kill -0 \"$pid\" 2>/dev/null || { cat /tmp/orca-serve.log >&2; exit 1; }; sleep 0.25; \\\n done; cat /tmp/orca-serve.log >&2; echo \"serve recipe JSON timed out\" >&2; exit 1')\"\n\n# 4. print serve's JSON enriched with userData (single object on stdout)\nnode -e 'const p=JSON.parse(process.argv[1]); console.log(JSON.stringify({...p, schemaVersion:1,\n userData:{...p.userData, provider:\"vercel-sandbox\", resourceId:process.argv[2], snapshotId:process.argv[3]}}))' \\\n \"$recipe_json\" \"$name\" \"$snapshot_id\"\ntrap - EXIT\n```\n\n`suspend`/`resume`/`destroy` use `vercel sandbox stop|...|remove \"$resource_id\"` reading\n`userData.resourceId` from stdin (§7d). This is the **Orca-server** connection mode (the recipe emits a\npairing URL). If the user chose **SSH** in the §1 interview, use §7g instead.\n\n### 7g. Worked example — existing SSH host (SSH connection mode)\n\nSSH mode is **fundamentally different from §7c/§7f**, not a relabeling of them:\n\n- **`create` does NOT run `orca serve` and does NOT emit a `pairingCode`.** Orca itself connects to the\n host over its SSH relay, brings up the git + filesystem providers, and imports the repo. The script's\n only job is to make the host ready and **print SSH connection details** Orca will dial.\n- The result uses a `connection` block with `type: \"ssh\"` and a `target`, **not** the flat\n `pairingCode`/`projectRoot` shape. Exact shape (Orca rejects anything else):\n\n```json\n{\n \"schemaVersion\": 1,\n \"connection\": {\n \"type\": \"ssh\",\n \"projectRoot\": \"/abs/path/to/repo/on/host\",\n \"target\": {\n \"label\": \"my-box\",\n \"host\": \"192.0.2.10\",\n \"port\": 22,\n \"username\": \"ubuntu\",\n \"identityFile\": \"~/.ssh/id_ed25519\",\n \"jumpHost\": \"bastion.example.com\",\n \"proxyCommand\": \"cloudflared access ssh --hostname %h\",\n \"relayGracePeriodSeconds\": 0,\n \"portForwards\": []\n }\n }\n}\n```\n\n`label`, `host`, `port`, `username` are required; the rest are optional — omit any you don't need.\n\nFor an explicitly requested one-VM-per-workspace checkout, the create script must read\n`ORCA_RECIPE_RESULT_SCHEMA_VERSION`, `ORCA_REPO_URL`, `ORCA_REPO_REF`, `ORCA_REPO_REF_HEAD`, and\n`ORCA_REPO_BRANCH`. Use `ORCA_REPO_REF` to fetch the selected source, but create\n`ORCA_REPO_BRANCH` at the exact `ORCA_REPO_REF_HEAD` commit; resolving the symbolic ref again can race\nwith an upstream update. `ORCA_REPO_URL` and `ORCA_REPO_REF` are a matched fetch pair, including when\nthe desktop source uses multiple remotes. Return that primary checkout at `projectRoot` and emit the\nsame SSH result with:\n\n```bash\n[ -n \"${ORCA_REPO_REF_HEAD:-}\" ] || { echo \"missing pinned source commit\" >&2; exit 1; }\ngit fetch origin \"$ORCA_REPO_REF\"\ngit cat-file -e \"${ORCA_REPO_REF_HEAD}^{commit}\"\ngit checkout -B \"$ORCA_REPO_BRANCH\" \"$ORCA_REPO_REF_HEAD\"\n```\n\n```json\n{\n \"schemaVersion\": 2,\n \"checkoutMode\": \"provisioned-root\",\n \"connection\": {\n \"type\": \"ssh\",\n \"projectRoot\": \"/abs/repo\",\n \"target\": { \"label\": \"my-box\", \"host\": \"192.0.2.10\", \"port\": 22, \"username\": \"ubuntu\" }\n }\n}\n```\n\nFail if the requested schema is not `2`; do not silently fall back to the ordinary recipe shape.\n\n**Networking → which `target` fields to set** (how *your desktop* reaches the box — there is no\n`orca serve` URL in SSH mode):\n\n- Public IP / DNS, or a Tailscale/VPN address → `host`; SSH port → `port` (usually 22).\n- Key auth → `identityFile` (add `identitiesOnly: true` if the agent has many keys).\n- Through a bastion → `jumpHost` (a `user@host` ProxyJump) **or** a full `proxyCommand` (e.g. an access\n proxy). Use one, not both.\n- A service port the workspace needs → add entries to `portForwards`.\n- `relayGracePeriodSeconds` (optional): how long Orca keeps the SSH relay alive after the workspace\n detaches before tearing it down; `0` = tear down immediately. Leave it off unless the user wants a\n reconnect grace window.\n\n**Toolchain & agent auth on a persistent (no-snapshot) host — do this ONCE, by hand, before wiring the\nrecipe** (there's no base image to bake; the host *is* the base). Run the §7f Phase-2 install steps and\nthe §7f Phase-3 ` login --device-auth` **directly over SSH on the host** (interactive, e.g.\n`ssh -t user@host ' login --device-auth'`). After that the host stays ready across workspaces.\n\n```bash\n#!/usr/bin/env bash\nset -euo pipefail\n# resolve from env→state→fallback (default unset optionals to \"\"): ssh_username, host,\n# ssh_port (default 22), identity_file, jump_host, proxy_command, project_root, repo_url, repo_ref\n: \"${identity_file:=}\"; : \"${jump_host:=}\"; : \"${proxy_command:=}\" # avoid set -u aborts on optionals\ngh_token=\"${GH_TOKEN:-${GITHUB_TOKEN:-$(command -v gh >/dev/null 2>&1 && gh auth token 2>/dev/null || true)}}\"\nssh_target=\"${ssh_username}@${host}\"\nssh_opts=(-p \"$ssh_port\"); [ -n \"$identity_file\" ] && ssh_opts+=(-i \"$identity_file\")\n# Why: a fresh host's key isn't in known_hosts; a StrictHostKeyChecking prompt would HANG a\n# non-interactive create. Pre-add the key (or set the option) so it can't block.\nssh-keyscan -p \"$ssh_port\" \"$host\" >> \"$HOME/.ssh/known_hosts\" 2>/dev/null || true\n\n# 1. ensure the repo is present and at the right commit on the host (NO orca serve here)\nssh \"${ssh_opts[@]}\" \"$ssh_target\" \\\n \"GH_TOKEN='$gh_token' GIT_TERMINAL_PROMPT=0 bash -lc '\n set -euo pipefail\n [ -d \\\"$project_root/.git\\\" ] || git clone \\\"$repo_url\\\" \\\"$project_root\\\"\n cd \\\"$project_root\\\" && git fetch origin \\\"$repo_ref\\\" && git checkout -B \\\"$repo_ref\\\" FETCH_HEAD\n '\" >&2\n\n# 2. print the SSH connection block (NO pairingCode, NO orca serve). host/port/username tell Orca's\n# relay how to dial in; identityFile/jumpHost/proxyCommand/portForwards are emitted when set.\nnode -e 'const [host,port,user,idf,jh,pc,root]=process.argv.slice(1);\n const target={ label:\"per-workspace-host\", host, port:Number(port), username:user };\n if(idf) target.identityFile=idf; if(jh) target.jumpHost=jh; if(pc) target.proxyCommand=pc;\n // add target.portForwards=[...] here if the workspace needs forwarded service ports\n console.log(JSON.stringify({ schemaVersion:1, connection:{ type:\"ssh\", projectRoot:root, target } }))' \\\n \"$host\" \"$ssh_port\" \"$ssh_username\" \"$identity_file\" \"$jump_host\" \"$proxy_command\" \"$project_root\"\n```\n\n`suspend`/`resume`/`destroy`: on a persistent host there's usually nothing to tear down — set\n`destroy: none` and omit suspend/resume. (Orca still disconnects/reconnects its own SSH relay on\nsleep/wake/delete — that's separate from these scripts.)\n\nIf the SSH host is instead an **ephemeral/snapshot-capable VM** (your hypervisor, or a cloud VM with\nimage support), keep the §7f Phase-2/3 base-image model for provisioning, but still emit the\n`connection.type:\"ssh\"` block above instead of starting `orca serve`.\n\n### 7h. Worked example — local Docker SSH (SSH connection mode)\n\nLocal Docker can model an ephemeral SSH VM without cloud cost: build a base image with `sshd`, tools,\nrepo prerequisites, and the agent CLI; run an **interactive auth container** once; then `docker commit`\nthat container as the authenticated image used by per-workspace `create`.\n\nKey points:\n\n- Publish container SSH to a random localhost port (`-p 127.0.0.1::22`) and emit\n `connection.type:\"ssh\"` with `host:\"127.0.0.1\"`, that port, `username`, `identityFile`, and\n `identitiesOnly:true`.\n- Generate a repo-local SSH key if needed, but gitignore the private/public key files.\n- **Bake SSH host keys into the base image** (`ssh-keygen -A` at **build** time; at runtime only generate\n if absent). Ephemeral containers all present the **same** host key, so `known_hosts` on `127.0.0.1`\n doesn't churn as the published port rotates across workspaces (otherwise every container's freshly\n generated key collides on `localhost` and trips host-key-changed warnings).\n- The auth image is the Docker equivalent of Phase 3: the **user** runs the agent login **inside** the\n container (you can't drive it — you have no interactive TTY), configures proxy env/config, approves\n hooks, and you commit once they report it's done. On a headless container use the **device-auth** flow\n (§4). Verify login before committing — exit code, or fold stderr and match the exact success line (§4).\n- Do not bind-mount or copy the host's full agent home into the image. Let each container have writable\n agent state; only the committed auth image should carry reusable authenticated state.\n- If committing from an interactive shell, force the runtime entrypoint back to `sshd`:\n `docker commit --change='ENTRYPOINT [\"/usr/local/bin/orca-docker-ssh-entrypoint\"]' …`.\n- `destroy` should read `recipeResult.userData.resourceId` and run `docker rm -f \"$resource_id\"`.\n\nValidation before wiring/live use:\n\n```bash\ndocker image inspect \"$auth_image\" --format '{{json .Config.Entrypoint}}'\ndocker run -d --name \"$name\" -p 127.0.0.1::22 -e \"ORCA_SSH_PUBLIC_KEY=$pubkey\" \"$auth_image\"\ndocker ps -a --filter \"name=$name\"\ndocker logs \"$name\"\nssh -i \"$key\" -p \"$port\" -o IdentitiesOnly=yes user@127.0.0.1 'codex --version'\n```\n\nIf the container exits immediately, inspect logs before the cleanup trap removes it; a committed\ninteractive image with `ENTRYPOINT [\"bash\"]` is a common cause.\n\nAlso confirm the **host key is stable** across containers: the SSH `ssh -i … 127.0.0.1` dial should not\ntrigger a host-key-changed warning when a second container reuses the port. If it does, the host keys\nweren't baked into the base image (see the `ssh-keygen -A` point above).\n\n### 7i. Windows local-side scripts\n\nThe local-side scripts run on the user's desktop. On **Windows**, a bare `.sh` won't execute. Either\nrequire WSL/Git-Bash (and point `orca.yaml` at e.g. `bash ./scripts/orca-vm/.sh` via a `.cmd`\nlauncher), or scaffold PowerShell equivalents. Minimal PowerShell shape:\n\n```powershell\n#requires -Version 5\n$ErrorActionPreference = 'Stop'\n# resolve env→state→fallback; run the provider CLI / ssh the same way;\n# capture provider output; build the result object for the chosen mode and write ONE line of JSON to stdout.\n# Orca-server mode: @{ schemaVersion=1; pairingCode=$pairingCode; projectRoot=$projectRoot; userData=@{...} }\n# SSH mode: @{ schemaVersion=1; connection=@{ type=\"ssh\"; projectRoot=$projectRoot;\n# target=@{ label=$label; host=$host; port=$port; username=$user } } } (see §7g/§7h)\n($result | ConvertTo-Json -Compress -Depth 6)\n# progress/errors → Write-Error / the error stream, never stdout.\n```\n\nThe remote-side commands you run *inside* the Linux VM stay bash regardless of the desktop OS.\n\n---\n\n## 8. Per-workspace recipe contract (the fast path)\n\nOnce the authenticated snapshot exists, this runs on every workspace create. Define recipes in\n`orca.yaml`:\n\n```yaml\nenvironmentRecipes:\n - id: cloud-sandbox\n name: Cloud Sandbox\n create: ./scripts/orca-vm/cloud-sandbox-create.sh\n suspend: ./scripts/orca-vm/cloud-sandbox-suspend.sh\n resume: ./scripts/orca-vm/cloud-sandbox-resume.sh\n destroy: ./scripts/orca-vm/cloud-sandbox-destroy.sh\n```\n\n`create` runs **locally from the repo root** and prints **one** JSON object to stdout. Its shape depends\non the connection mode chosen in §1:\n\n**Orca-server mode** — boot the env, start `orca serve` in it, and print serve's result:\n\n```json\n{\n \"schemaVersion\": 1,\n \"pairingCode\": \"orca-pairing-code-or-url\",\n \"projectRoot\": \"/absolute/path/to/repo/on/remote\",\n \"userData\": { \"provider\": \"example\", \"resourceId\": \"provider-resource-id\" }\n}\n```\n\nHere `pairingCode` (from `orca serve --recipe-json`) and `projectRoot` are required; `schemaVersion` (`1`)\nand `userData` are optional.\n\n**SSH mode** — do **not** run `orca serve`; print the `connection.type:\"ssh\"` block instead (full shape +\nworked script in §7g). `pairingCode` is **not** used in SSH mode.\n\n**Optional provisioned root** — only for direct SSH and only when explicitly requested. Add\n`checkoutMode: provisioned-root` to the recipe, require `ORCA_RECIPE_RESULT_SCHEMA_VERSION=2`, create\nthe requested `ORCA_REPO_BRANCH` at the pinned `ORCA_REPO_REF_HEAD` commit (use `ORCA_REPO_REF` only\nto fetch that commit) at the returned `projectRoot`, and emit schema version 2 with\n`checkoutMode: \"provisioned-root\"`. All recipes without this field retain the schema-v1 behavior above.\n\nLifecycle hooks (all run locally):\n\n- `create`: required. Prints recipe result JSON.\n- `suspend`: optional. Sleep; reads lifecycle payload on stdin.\n- `resume`: optional. Wake; reads payload on stdin and **prints fresh recipe JSON** (pairing may change).\n- `destroy`: optional unless `destroy: none`. Delete/cleanup; reads payload on stdin.\n\nStart Orca remotely with `orca serve --port \"$PORT\" --project-root \"$ABS_ROOT\" --pairing-address\n\"$EXTERNAL_WSS_URL\" --recipe-json` (exact flags + output in §7c). Set `--pairing-address` to the\nexternally reachable address so the emitted `pairingCode` is reachable; tunneling/port mapping is the\nscript's job.\n\nBackward compatibility: `command`→`create`, `cleanup`→`destroy`, `cleanup: none`→`destroy: none`.\nPrefer the lifecycle names.\n\n---\n\n## 9. Doctor and validation\n\nValidate in two stages — the cheap dry run first, then the live self-test.\n\n### Dry run (free, non-destructive) — always do this first\n\n`orca vm recipe doctor --repo-path --json` validates **static wiring only** — it does\n**not** boot anything. It checks: local-host execution (v1), repo path, recipe id exists,\ncreate/destroy/suspend/resume command paths resolve, suspend/resume are paired, and each script is\nexecutable (POSIX exec bit; skipped on Windows). Fix every failure here before spending any cloud money.\n\n### Live self-test (`--provision`) — diagnose and iterate yourself\n\n`orca vm recipe doctor --repo-path --provision --json` actually runs the recipe end\nto end: it executes `create`, validates the returned recipe JSON, then runs `destroy` to **tear the\nenvironment back down** (so the test leaves nothing running, as long as `destroy` works). It spends real\ncloud money, so get the user's OK **once** before starting — that one approval covers the whole loop\nbelow; do not re-ask before each run.\n\nOn failure, the JSON result includes a `provisionTranscript` with the **complete** captured output of\neach stage so you can self-diagnose without asking the user to relay logs:\n\n```json\n{\n \"ok\": false,\n \"checks\": [ { \"id\": \"recipe.provision\", \"status\": \"fail\", \"message\": \"…\" } ],\n \"provisionTranscript\": {\n \"provision\": { \"exitCode\": 0, \"signal\": null, \"stdout\": \"…\", \"stderr\": \"…\", \"parseError\": \"…\" },\n \"destroy\": { \"exitCode\": 0, \"signal\": null, \"stdout\": \"…\", \"stderr\": \"…\" }\n }\n}\n```\n\n**Run it as a loop:** read `provisionTranscript.provision.stderr` / `.stdout` / `.parseError` (and\n`destroy.*`), fix the script, and re-run `--provision` until `ok` is `true` — iterating on your own\nrather than waiting for the user to paste errors. Common reads: a non-empty `stderr` with `exitCode 0`\nplus a `parseError` means `create` ran but printed something other than the single recipe-result JSON on\nstdout (often a stray `echo` — route it to stderr, see §10); a non-zero `exitCode` is a provider/script\nfailure described in `stderr`. Each stream is redacted and capped (head+tail) — large logs keep both the\nsetup context and the failure.\n\nThe self-test cannot see provider-side truth beyond what the scripts print, so still confirm: state has a\npopulated **authenticated** `snapshotId` (Phases 2–3 done), and `destroy` is implemented/tested (or\nexplicitly `none` — in which case the self-test won't tear down, so clean up manually).\n\nFor SSH recipes, also smoke-test the exact emitted target before declaring success: dial the host/port\nwith the identity/proxy settings, run `pwd`, verify the repo path, check the agent binary, and confirm\n`destroy` removes the provider resource/container. For Docker, inspect the auth image entrypoint and do a\nstartup-only `docker run` before the full clone/install path.\n\n---\n\n## 10. Failure modes\n\n- **Build exceeds plan timeout (e.g. Hobby 45m).** Use enough vCPUs and a timeout covering the build;\n else split work or use a higher plan. The cap also limits per-workspace runtime — surface it.\n- **Build exceeds plan RAM.** Build the **headless main only** (drop the renderer) — the biggest fitter.\n- **Private-repo clone hangs/fails.** Wrong/missing token. Use `GIT_ASKPASS` + `GIT_TERMINAL_PROMPT=0`\n so it fails fast instead of prompting.\n- **`GIT_ASKPASS` helper aborts the clone with \"`$1: unbound variable`\".** The `printf`/heredoc that writes\n the helper inside `bash -lc` under `set -u` expanded `$1`/`$GH_TOKEN` at **write** time. Escape them\n (`\\$1`, `\\$GH_TOKEN`) so they land literally and resolve at git-runtime; this also keeps the real token\n out of the file. `rm -f` the helper afterward (§5, §7f).\n- **Agent verified as \"not logged in\" despite a good login.** `codex login status` (and similar) print\n \"Logged in …\" to **stderr**; an stdout-only `grep` misses it. Prefer the status **exit code**; if you\n grep, fold stderr first (`status 2>&1 | grep …`) and match the exact success line — not `grep -qi\n 'logged in'`, which also matches \"not logged in\".\n- **Headless agent login hangs.** Plain OAuth `login` starts a loopback callback server on a VM/container\n port the host browser can't reach. Use the **device-auth** flow (`login --device-auth`) — it prints a\n URL + code the user opens on the host.\n- **`known_hosts` host-key churn on local Docker.** Each ephemeral container regenerating its SSH host key\n collides on `127.0.0.1` as the published port rotates. Bake host keys into the base image at build time\n (`ssh-keygen -A`; runtime generates only if absent) so all containers share one stable key (§7h).\n- **Snapshot expired/evicted.** If `create` hits an unknown snapshot id, rerun Phases 2–3 and update\n `snapshotId`.\n- **Agent auth didn't persist.** Confirm `snapshotId` points at the **authenticated** snapshot; re-run\n Phase 3. Warn that short-lived tokens may need periodic re-auth.\n- **Agent auth copied from the host breaks.** Do not bind-mount/copy a full host agent home; sqlite\n files can be unwritable or host-specific, hooks may need approval again, and config may reference\n local-only env vars. Authenticate inside the runtime and snapshot/commit that layer.\n- **Docker auth image exits immediately.** Inspect `docker image inspect … .Config.Entrypoint` and\n `docker logs`. If the image was committed from an interactive shell, reset the entrypoint to the SSH\n entrypoint during `docker commit`.\n- **Leaked paid resource.** Every long script must trap errors and remove the sandbox it created.\n- **`create` emits non-JSON on stdout.** A stray `echo` corrupts the result — stdout is for the final\n JSON only; everything else to stderr. The `--provision` self-test surfaces this as `exitCode 0` + a\n `parseError` with the offending stdout in `provisionTranscript` (§9).\n\n---\n\n## 11. Boundaries\n\n- Don't create accounts, choose plans/regions, or invent scope/project/org/image/billing ids.\n- Don't invent or store credentials; no secrets in `userData`, state, comments, docs, or commits.\n- Don't run paid/long phases (base snapshot, auth, live test) without an explicit OK.\n- Don't hide provider errors behind generic messages — preserve actionable stderr.\n- Don't make Orca own provider lifecycle beyond invoking the configured scripts.\n- Don't commit or create an Orca workspace unless asked.\n" // oxfmt-ignore -const ORCHESTRATION_MARKDOWN = "---\nname: orchestration\ndescription: >-\n Use Orca orchestration for structured multi-agent coordination: threaded\n messages, blocking ask/reply flows, task dispatch, worker_done/escalation\n waits, task DAGs, decision gates, coordinator loops, or decomposing work\n across agents. Use `orca-cli` instead for full ownership handoffs, including\n requests phrased as \"hand off\", \"handoff\", \"handover\", \"give this to another\n agent\", or \"another worktree\" when the user did not explicitly ask to\n supervise, monitor, wait for results, or coordinate a DAG. Use `orca-cli` for\n ordinary terminal control, lightweight terminal prompts, shell commands, Orca\n worktree management, reading or waiting on terminals, and automation of the\n browser embedded inside Orca. Use Computer Use for browser windows, webviews,\n Orca app UI, or desktop UI outside Orca's embedded browser.\n---\n\n# Orca Inter-Agent Orchestration\n\nOrchestration is Orca's structured coordination layer for agent messages, task ownership, dispatch state, and worker completion tracking.\n\nUse this skill when coordination state matters. For lightweight terminal prompts or basic worktree/terminal/built-in-browser control, use `orca-cli`.\n\n## Tool Boundary\n\nIf a task says to use Orca orchestration, the coordinator must create or bind a Run, create the Task with `orca orchestration task-create`, then attach the worker with either the preferred `orca orchestration worker-start` composition or the low-level `orca orchestration dispatch --inject` path.\n\nDo not substitute non-Orca subagent tools, generic agent-spawn APIs, or chat-only parallel worker features. Those may create useful workers, but they do not create Orca task/dispatch provenance, injected lifecycle preambles, `worker_done` authority, or decision gates.\n\nBefore claiming a worker was orchestrated, verify the task/dispatch exists:\n\n```bash\norca orchestration task-list --json\norca orchestration dispatch-show --task --json\n```\n\nIf the work was accidentally run outside Orca orchestration, say so plainly. To repair provenance, rerun or revalidate the needed work through a fresh Orca terminal plus injected dispatch; do not retroactively describe the external worker as orchestrated.\n\n## When To Use\n\n- Send/reply/ask between agent terminals with persistent messages.\n- Dispatch structured tasks to workers and wait for `worker_done` or `escalation`.\n- Track task DAGs with dependencies.\n- Run coordinator loops or decision gates.\n\nDo not use orchestration merely because the user says \"hand off\", \"handoff\", \"handover\", \"give this to another agent\", or asks for another worktree/agent/model/effort. Those are full ownership transfers unless the user explicitly asks to supervise, monitor, wait for worker completion/results, coordinate a DAG, use decision gates, or keep a blocking ask/reply loop.\n\n## Preconditions\n\n- `orca status --json` should show a running runtime.\n- `orca` must be on PATH (`orca-ide` on Linux).\n- The orchestration experimental feature must be enabled in Settings > Experimental.\n- `orca orchestration` commands are RPC calls to the running Orca runtime.\n\n## Contract Migration\n\nOrca adopts a live pre-update orchestration assignment into an ordinary Run. Adoption preserves the existing agent process, PTY/session, terminal handle, tab/leaf/pane, worktree or folder workspace, Task, and Dispatch; it never restarts or replaces the worker. The retired scheduler is not revived, and a newly created attempt uses the current grammar.\n\nTreat the authority label on injected or formatted messages as definitive:\n\n- `[LEGACY COMPATIBILITY]` is live and attested. Run only the exact supported command printed with the message, using the same CLI executable and arguments that the original prompt supplied.\n- `[LEGACY RECOVERY REPLAY — MAY HAVE BEEN SEEN]` is one bounded, at-least-once cutover replay. Process it idempotently and acknowledge it only through the exact displayed guidance.\n- `[LEGACY READ-ONLY]` is inspection-only. It has no reply, acknowledgment, or lifecycle action.\n- An unlabeled current message uses the current guide and current grammar.\n\nAn explicitly selected current Run, attested current Run binding, current Dispatch, or federated attachment takes precedence over legacy fallback. A retained adoption record alone never turns a current command into a legacy call.\n\nDatabase provenance, an old-looking terminal, or a legacy Run ID does not prove mutation authority. If the runtime cannot prove liveness, principal ownership, capability, or the exact legacy contract, it degrades to read-only inspection and must not fall back to local execution. Exact recovery may restore the already-live PTY once in its original inactive background tab. It must not spawn, write, signal, stop, switch, focus, split, or inject a terminal. Loss of lifecycle authority does not invalidate the existing assignment, process, or filesystem work.\n\nCompatibility retries have narrow guarantees. A pending ask, a reply, a final Dispatch settlement, and a consuming check have durable recovery identities. A-era heartbeat and escalation calls remain at-least-once across a manual A-to-B retry because identical later signals may be intentional. If an A-era ask may already have been answered, run the exact non-consuming recovery check printed by the runtime first; after its answer is printed and acknowledged, a new invocation with the same question creates a new question. Never guess among multiple identical question threads.\n\nWhen a compatibility or recovery command returns structured next-step arguments, run those exact arguments with the same CLI executable. The arguments intentionally omit the executable name so the guidance works with `orca`, `orca-ide`, `orca-dev`, or another configured Orca CLI command. Do not translate the command from memory, broaden its recipient, or retry it as a current mutation unless the returned guidance explicitly says to.\n\nOn packaged Windows, a legacy ask uses a two-step commit/resume protocol. The initial command durably commits the question, prints its exact `ask --resume ` command, and exits with launcher status `75`; it does not wait for the answer. Run that exact resume command after the launcher or update boundary. Resume is idempotent and read-oriented: it waits for the already-committed question and does not create another one. For a WSL process that received compatibility proof at launch, use the printed executable `orca-ide` WSL resume command so the same distro and packaged launcher authority are preserved; do not substitute a PATH-resolved local CLI. Older WSL processes that never received the hidden launch token remain lifecycle read-only after the update, even while their terminal and filesystem work continue.\n\nLegacy inspection remains available without consuming mail:\n\n```bash\norca orchestration run-list --json\n# run_legacy_local is an empty audit tombstone after adoption.\norca orchestration run-show --id run_legacy_local --json\n# In run-list, find the ordinary Run whose objective is:\n# \"Recovered orchestration work from a contract update\"\norca orchestration run-show --id --json\norca orchestration task-list --run --json\norca orchestration inbox --full --json\norca orchestration check --terminal --peek --format --json\norca terminal read --terminal --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\n```\n\nIf the original coordinator is unavailable or cannot prove its retained authority, a current coordinator may explicitly take over the adopted Run from its own live agent terminal:\n\n```bash\norca orchestration run-use --id --takeover-legacy --json\norca orchestration check --run --json\n```\n\nTakeover fences only the old coordinator, binds the current one, and moves pending worker mail into current Run Delivery. It is bound to the authenticated invoking terminal; `--from` cannot name another coordinator. Live legacy workers keep their original Tasks, Dispatches, processes, filesystems, and old prompt commands; their later questions, escalations, and completion reports route to the current coordinator. Do not use takeover while the original coordinator is still actively coordinating, because its later lifecycle mutations are rejected.\n\nDo not launch a replacement editor merely because the desktop app or runtime was updated. If adoption cannot prove continuing authority, keep the original worker as the only editor until it reaches a stable handoff point, then use a new current Dispatch in a conflict-free placement for any remaining work.\n\n## Ownership\n\nNew orchestration messages and tasks belong to one explicitly bound Run. A Run is only a durable namespace and coordinator inbox; it never schedules or places workers. Lifecycle authority comes from the active Dispatch, and terminal handles remain routing metadata rather than durable identity. Send `worker_done` and `heartbeat` from the worker's own terminal; Orca routes them to that Dispatch's Run.\n\nClassify inherited context before sending lifecycle messages:\n\n- Coordinated subtask: a live coordinator owns the DAG and waits on this dispatch. Follow the preamble exactly, including `worker_done`, heartbeat/status, `ask`, and `escalation`.\n- Full handoff means ownership transfer, not supervised dispatch. The original actor is not monitoring a DAG, so do not create lifecycle obligations unless the user explicitly asks you to supervise.\n- Classify requests containing \"hand off\", \"handoff\", \"handover\", \"give this to another agent\", \"give this to another worktree\", \"another agent\", or \"another worktree\" as full handoffs by default, even when the user names a custom model or reasoning effort.\n- Use supervised orchestration only when the user explicitly asks you to \"supervise\", \"monitor\", \"wait\", \"track completion\", \"wait for worker_done\", return results, coordinate a DAG, use a decision gate, or manage ask/reply flow.\n- Do not use `orca orchestration dispatch --inject` for full handoffs. It injects a coordinator preamble that tells the worker to send `worker_done`, heartbeat, and `ask` messages, then end its turn under the original terminal's dispatch lifecycle.\n- Do not run `orca orchestration task-create`, `orca orchestration dispatch --inject`, or `orca orchestration check --wait` for full handoffs. Do not peek at terminal output after prompt delivery to monitor progress.\n- A review-only `worker_done` reports findings; it does not authorize coordinator file edits. After a review-only completion, synthesize findings, ask a decision gate if ownership is unclear, and dispatch or hand off fixes unless the user explicitly asked the coordinator to own fixes.\n- If the user's plan names a next owner agent (for example, \"then use opencode to create a PR\"), post-review corrections and PR prep belong to that named owner. The coordinator routes, synthesizes, asks decision gates when needed, and supervises; the named owner edits files and creates the PR.\n\nIf unclear, inspect orchestration state before sending lifecycle messages:\n\n```bash\norca orchestration task-list --json\norca terminal list --json\n# If inherited context includes a task id:\norca orchestration dispatch-show --task --json\n```\n\n## Messaging\n\n```bash\norca orchestration send --subject [--to ] [--from ] [--body ] [--type ] [--priority ] [--thread-id ] [--payload ] [--json]\norca orchestration check [--terminal ] [--ack ] [--peek|--all] [--types ] [--format] [--wait] [--timeout-ms ] [--json]\norca orchestration reply --id --body [--from ] [--json]\norca orchestration ask (--question |--resume ) [--options ] [--timeout-ms ] [--from ] [--json]\norca orchestration inbox [--limit ] [--json]\n```\n\nRules:\n\n- Omit `--from` unless impersonating another terminal; Orca auto-resolves it from the current terminal.\n- A coordinator `check` returns the bound Run's oldest FIFO Delivery (up to 50 messages) and replays that exact batch until `--ack `. Process every message before acknowledging; `check --ack --wait` acknowledges, checks, and waits in one operation.\n- Use `--peek` and `--all` only for read-only history/debugging. Type filters decide when a waiter wakes; the returned actionable Delivery is still the oldest full batch.\n- Use `dispatch:` for coordinator guidance to one supervised worker. Orca routes that stable address locally or through the connected-server relay; do not substitute a remote terminal handle.\n- Terminal handles remain appropriate for low-level pre-Dispatch messaging. Prefer `agentTerminalHandle` from the create response, fall back to `startupTerminal.handle` for older runtimes, then re-resolve with `orca terminal list --worktree ... --json` if missing or stale. Continue with the replacement handle only; never dual-send to old and new handles.\n- `terminal list --json` omits `visualLayouts` because handle recovery does not need topology. Add `--include-visual-layouts` only for explicit tab and pane inspection.\n- `orca orchestration check --peek --format --json` returns locally formatted unread mail without consuming it; it never writes to terminal input or remotely wakes another terminal. Use `orchestration dispatch --inject` to deliver a tracked task, or `terminal send` when an existing agent needs a free-form prompt.\n- While supervising workers manually, use `check --wait --types worker_done,escalation,question --timeout-ms ` instead of sleep/poll loops. Process the whole Delivery, reply to `question` messages with `orca orchestration reply --id --body --json`, then acknowledge and keep waiting.\n- Treat a `check --wait` timeout or `{count:0}` as a checkpoint, not a worker failure. Long coding tasks routinely run 15-60 minutes; keep using rolling waits unless you receive `worker_done`/`escalation`, the terminal exits or disappears, or the user explicitly asks you to stop.\n- Heartbeats and visible terminal activity mean the worker is alive, not done. Do not stop, close, kill, or restart a worker just because it has not produced a completion message yet.\n- Use `ask` when a worker needs a blocking answer from the coordinator; it defaults to the active Dispatch's Run. Timeout or disconnect leaves the question pending, so resume by its original message ID instead of asking again.\n- `check --wait` returns one bounded Delivery, not every future completion. Process every message, acknowledge it, then keep waiting until every expected Dispatch settles.\n- Group addresses include `@all`, `@idle`, `@claude`, `@codex`, `@opencode`, `@gemini`, `@droid`, `@grok`, `@cursor`, and `@worktree:`.\n- Message types include `status`, `dispatch`, `worker_done`, `merge_ready`, `escalation`, `handoff`, `question`, `decision_gate` (legacy/gates), and `heartbeat`.\n- Use group addresses only for messages that are genuinely useful to many terminals, such as `status` broadcasts or intentional fan-out questions. Do not send dispatch lifecycle messages to groups.\n- `worker_done` belongs to the active Dispatch and defaults to its Run mailbox; never target a group.\n- A valid `worker_done` for the active `taskId` + `dispatchId` marks the task and dispatch completed automatically. Do not follow it with `task-update --status completed`; reserve manual updates for explicit recovery or overrides.\n- `heartbeat` is also Dispatch-scoped. Include both IDs and omit `--to` so Orca uses the owning Run; use `status` for broad progress updates.\n\n## Tasks And Dispatch\n\nA Run is the namespace/inbox, a Task is the work item, and a Dispatch assigns one Task attempt to a terminal. Create or bind a Run once before the common loop.\n\n```bash\norca orchestration run-create --objective --json\norca orchestration task-create --spec [--deps ] [--parent ] [--json]\norca orchestration task-list [--status ] [--ready] [--brief] [--json]\norca orchestration task-update --id --status [--result ] [--json]\norca orchestration dispatch --task --to [--from ] [--inject] [--json]\norca orchestration dispatch-show --task [--json]\n```\n\nTask statuses: `pending`, `ready`, `dispatched`, `completed`, `failed`, `blocked`.\n\nDispatch rules:\n\n- `--inject` sends the task spec plus preamble into a recognized agent CLI so it can report `worker_done`.\n- If the target is a bare shell, omit `--inject`, dispatch for tracking if needed, then send the prompt manually with `orca terminal send --terminal --text --enter --json`.\n- After 3 consecutive failures on one task, the dispatch context circuit-breaks and the task is marked failed.\n- Use `task-list --brief --json` for coordinator sweeps; it collapses whitespace and caps each echoed spec at 160 characters (`spec_truncated` marks shortened rows). Omit `--brief` when the full spec is required, or when an older CLI rejects it as an unknown flag.\n\n## Preferred Supervised Worker Loop\n\nUse `worker-start` for the normal supervised path. It composes the existing worktree, terminal, readiness, and dispatch primitives while returning exact created/reused effects. Agents still choose placement and concurrency; Orca does not schedule workers or infer conflicts.\n\nCreate the Run and every independent Task first, then start all independent workers before waiting:\n\n```bash\norca orchestration run-create --objective \"\" --json\norca orchestration task-create --spec \"\" --json\norca orchestration task-create --spec \"\" --json\norca orchestration worker-start --task --worktree current --agent codex --json\norca orchestration worker-start --task --worktree current --agent claude --json\n```\n\n`current` and exact existing worktrees create a fresh agent terminal and do not rerun setup. Reuse an existing agent only with `--terminal `.\n\nFor a per-invocation Claude, Codex, or Cursor launch, pass an opaque provider model id with `--model`; add `--effort` only when that agent/model supports the level. These options apply only to fresh agent terminals, override general agent default arguments, and are reported under `launch.requested` and `launch.effective` in the receipt:\n\n```bash\norca orchestration worker-start --task --worktree current --agent claude --model opus --effort high --json\n```\n\n`--effort` requires `--model`, and neither option can combine with `--terminal`. A connected worker server must advertise launch-preference support before Orca forwards either option.\n\nFor a new worktree, setup runs by default and agent-first creation reuses the returned startup agent terminal:\n\n```bash\norca orchestration worker-start --task --worktree new-child --name --agent codex --setup run --json\n# Independent/top-level:\norca orchestration worker-start --task --worktree new-top-level --name --agent codex --setup run --json\n```\n\nSetup normally starts alongside the agent. Only a repository explicitly configured with `wait-for-setup` delays agent launch until setup succeeds. Use `--setup skip` or `--setup inherit` only for a concrete reason.\n\nRead the returned receipt before continuing: `ready` plus setup `running` is normal for start-immediately, while wait-for-setup returns setup `succeeded` before accepting task input. A failed or unknown start exits nonzero; inspect its `stage`, `effects`, and `residualResources` instead of guessing or automatically retrying. A wait-for-setup timeout can honestly leave setup `running`, which is not proof of failure.\n\nTo run the worker on another connected Orca server, add `--on `. The Run and Tasks remain authoritative on the current server; later commands route by Dispatch ID, so never repeat `--on`:\n\n```bash\n# Mac Run home -> Windows worker (the reverse is identical from a Windows Run home)\norca orchestration worker-start --task --on windows --worktree new-top-level --repo --name --agent codex --setup run --json\norca orchestration worker-show --dispatch --json\norca orchestration worker-read --dispatch --limit 50 --json\norca orchestration send --to dispatch: --subject \"Follow-up\" --body \"\" --json\n```\n\nRemote `current` and `new-child` are intentionally invalid because those words are ambiguous across servers. Use an exact discovered remote worktree selector or `new-top-level` with an explicit remote repo selector.\n\nThe follow-up is structured inbox mail, not prompt injection. The worker's next\n`orchestration check` receives it even when the Dispatch is on another connected Orca server.\n\n`worker-read` defaults to `--source auto`: Orca returns the exact hook-reported Codex, Claude, OpenClaude, or Grok transcript when it can prove the worker session, otherwise it returns bounded terminal output with `source: \"terminal\"` and a typed `fallbackReason`. Continue with the returned top-level `cursor`; it stays pinned to that exact source. If Orca reports `source_changed`, start a fresh read without the old cursor. Never supply or guess a provider session ID or transcript path.\n\nWait until every expected Dispatch settles, not for a fixed number of batches:\n\n```bash\norca orchestration check --wait --types worker_done,escalation,question --timeout-ms 900000 --json\n# Process every message. For each accepted worker_done that is not immediately reused:\norca orchestration worker-release --dispatch --json\n# Acknowledge only after every message and required release decision is handled:\norca orchestration check --ack --wait --types worker_done,escalation,question --timeout-ms 900000 --json\n```\n\nAfter processing each accepted `worker_done`, choose the terminal's next owner before you acknowledge the Delivery or wait again. If the same exact agent has an immediate follow-up Task, read the `worker.agent_terminal_handle` field of `worker-show --dispatch --json`, then run `orca orchestration worker-start --task --terminal --json` so Orca transfers cleanup ownership to the new Dispatch. Otherwise run `orca orchestration worker-release --dispatch --json`.\n\nRun `worker-release` after both succeeded and failed `worker_done` reports unless the user explicitly asked to keep that worker live. Release is post-completion cleanup, not cancellation: Orca first preserves inspectable output, then closes only the exact agent terminal owned by that settled Dispatch. Reused or pre-existing terminals, setup terminals, coordinators, active workers, user-taken-over terminals, and identities Orca cannot prove are retained. If the user explicitly asks to keep the live terminal for debugging, record that exception with `orca orchestration worker-retain --dispatch --json` instead of silently skipping cleanup. When the user is finished, the same Dispatch can be passed to `worker-release`, which clears the requested retention and releases the terminal.\n\nDo not release a worker because of a timeout, TUI idle state, heartbeat, status, question, escalation, or rejected/stale `worker_done`. If release returns `release_pending` or `release_unknown`, do not substitute `terminal close`; follow the exact recovery action in the receipt. A replayed Delivery may repeat `worker-release` safely.\n\nWorkers report exactly once using the IDs and capability injected by Orca; they do not supply Run/server/terminal identity:\n\n```bash\norca orchestration send --type worker_done --subject \"\" --body \"\" --task-id --dispatch-id --outcome succeeded --files-modified \"path/a,path/b\" --json\n# On failure, use --outcome failed; never encode failure only in prose.\n```\n\nA worker question defaults to its owning Run. Timeout leaves it pending:\n\n```bash\norca orchestration ask --question \"\" --options \"yes,no\" --timeout-ms 600000 --json\norca orchestration ask --resume --timeout-ms 600000 --json\n# Coordinator:\norca orchestration reply --id --body \"\" --json\n```\n\nRecovery is conditional, never a fixed destructive sequence:\n\n- `worker-show --dispatch ` says `ready`: keep waiting or read bounded output.\n- It proves `failed` or `stopped`: start a replacement with `worker-start --task --retry-of ` plus an explicit `--on`/`--worktree` and `--agent`/`--terminal` choice. Retry does not silently inherit placement.\n- It remains `outcome_unknown`: either `worker-stop --dispatch ` and inspect again, or explicitly `worker-abandon --dispatch ` while accepting that resources may still be live. Abandon performs no remote, process, or filesystem action.\n- `worker-stop` closes only the exact supervised agent terminal. It never deletes the worktree, setup terminal, configured tabs, or unrelated processes.\n\nLow-level `worktree create`, `terminal create`, and `dispatch --inject` remain valid recipes for custom argv or topology that `worker-start` does not express.\n\n`dispatch --inject` deliberately keeps an operator-started terminal unsupervised: it never creates a `worker_dispatches` row and `worker-stop`/`worker-abandon` never close that process. The dispatch context is still authoritative, so `worker-show`, `worker-read`, and `worker-list` report it as `unsupervised`; settled `worker-retain` and `worker-release` report `retained` with `no_owned_resource` and take no process action. Use `worker-start --terminal ` when supervision and worker lifecycle state are required.\n\n## Gates And Legacy Inspection\n\n```bash\norca orchestration gate-create --task --question [--options ] [--json]\norca orchestration gate-resolve --id --resolution [--json]\norca orchestration gate-list [--task ] [--status ] [--json]\n```\n\nUse `ask` for worker-to-coordinator questions; it creates a `question` message that the coordinator answers with `reply`. Use `gate-create` only for coordinator-managed task DAG decisions, not for answering a worker's `ask`.\n\n`coordinator-start`, `coordinator-stop`, `run`, and `run-stop` are retired scheduler commands. They perform no effects and return the current-skill recovery action. They are not aliases for lightweight Run creation or binding.\n\nRecovery only: `orca orchestration reset --tasks|--messages|--all --json` clears the selected local orchestration database state. Do not run it during active coordination unless explicitly abandoning that state.\n\n## Full Handoffs\n\nFor full ownership transfer, use non-lifecycle terminal/worktree commands and then stop monitoring unless the user asks for supervision.\n\nTreat these as full handoff requests by default: \"hand off\", \"handoff\", \"handover\", \"give this to another agent\", \"give this to another worktree\", \"send this to another agent\", \"another agent\", \"another worktree\", or \"launch another agent to own this.\" Custom model or reasoning effort words such as `gpt-5.5`, `high`, or `xhigh` do not make the handoff supervised.\n\nSupervised orchestration remains available only when the user explicitly asks for supervision or coordination: \"supervise\", \"monitor\", \"wait for worker_done\", \"wait for results\", \"track completion\", \"DAG\", \"decision gate\", \"ask/reply\", or \"coordinate workers.\"\n\nDo not run `orca orchestration task-create`, `orca orchestration dispatch --inject`, or `orca orchestration check --wait` for full handoffs. `task-create` is also forbidden because it records coordinator-owned tracking state; if a task row is needed, the user asked for supervised orchestration. Do not create a `taskId`/`dispatchId`, inject a lifecycle preamble, wait for completion, or read the worker terminal after prompt delivery except to avoid losing the initial prompt.\n\nNew top-level worktree handoff:\n\n```bash\norca worktree create --name --no-parent --agent codex --prompt \"\" --setup run --json\n```\n\nBefore creating a new worktree from an active feature branch, decide and state whether the desired Orca lineage is child or top-level. Use child worktree lineage only when the new work is conceptually stacked under or dependent on the active worktree. For independent repo-wide fixes, standalone feature work, or unrelated follow-up tasks, create a top-level worktree with `--no-parent`.\n\nExisting terminal handoff:\n\n```bash\norca terminal send --terminal --text \"\" --enter --json\n```\n\nCustom Codex model/effort handoff:\n\n`orca worktree create --agent codex --prompt ...` launches the known Codex agent but does not accept Codex-specific `--model` or `-c model_reasoning_effort=...` arguments. When the user asks for a specific Codex model or effort, create the independent worktree first, launch Codex with the requested command in that worktree, wait only for TUI readiness if prompt delivery would otherwise race startup, send the prompt, and stop.\n\nThe two-step custom-argv path cannot enforce a repository's explicit `wait-for-setup` startup policy because the later `terminal create` is not the startup owned by `worktree create`. Use it only when the repository starts agents immediately. If the repository requires `wait-for-setup`, use an agent-first configured launcher that can preserve sequencing, or stop and ask rather than silently bypassing the policy.\n\nNote: when no repo default-terminal configuration supplies a primary terminal, bare create opens a fallback shell before `terminal create` adds the agent. Configured default tabs are materialized instead and may run real commands. Prefer `--agent` whenever custom argv is not required. With the two-step path, target only the agent handle; close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell.\n\nUse the exact full `::` worktree id returned by `orca worktree create --json`; a bare repo id cannot target the new worktree.\n\n```bash\norca worktree create --name --no-parent --setup run --json\norca terminal create --worktree id: --title --command 'codex --model gpt-5.5 -c model_reasoning_effort=\"xhigh\"' --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca terminal send --terminal --text \"\" --enter --json\n```\n\nWait only for `tui-idle` when needed to avoid losing the prompt. Do not monitor task completion.\n\n`--no-parent` only controls Orca lineage; it does not choose the Git base. If the work should start from the repo default base, omit `--base-branch` so Orca uses that default, or explicitly pass the repo default base (`origin/main`, `origin/master`, or the `orca repo show --repo --json` value); never base it on the current feature branch unless the user explicitly asks for stacked work or \"branch from current\". Put current-branch context in the prompt instead.\n\n## Worker Terminals\n\nChoose the worker location before creating a terminal. `Fresh worker` means a fresh agent session, not a new git worktree. For parallel work, create one fresh agent terminal per worker in the same required worktree, falling back to the active worktree when none is named. If the task says current worktree only, depends on uncommitted files/artifacts, or must validate/PR the current branch, keep every worker in the active worktree:\n\n```bash\norca terminal create --worktree active --title --command \"codex\" --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca orchestration dispatch --task --to --inject --json\n```\n\nReuse an idle agent in the required worktree only if the prompt allows reuse; otherwise create a fresh terminal there. Create a new worktree only when the user explicitly requests one or a concrete checkout or filesystem conflict makes sharing unsafe or impossible; if the user did not request it, state that conflict before running `worktree create`. Independent tasks, parallel execution, convenience, or a preference for separate checkouts are not isolation requirements.\n\nWhen a new worktree is allowed, use child lineage for isolated work that is stacked under or dependent on the active worktree, and use `--no-parent` when it is not stacked. Decide the Git base separately: `--no-parent` makes the worktree top-level in Orca, while omitted `--base-branch` uses the repo default base.\n\nFor every new worktree, pass `--setup run` so any configured repository setup hook runs. This does not mean waiting for setup before agent launch: preserve the repository's startup policy, whose default starts setup and the agent side by side. Use `--setup skip` or `--setup inherit` only when there is a concrete task-specific reason, and state that reason before creating the worktree. This rule does not rerun setup for current or existing worktrees.\n\n```bash\norca worktree create --name --agent codex --setup run --json\n# or: --agent claude | omp | pi | grok | ...\n# Read from agentTerminalHandle, falling back to startupTerminal.handle.\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca orchestration dispatch --task --to --inject --json\n```\n\nFor new-worktree workers, read the id and `agentTerminalHandle` from `worktree create`, falling back to `startupTerminal.handle` for older runtimes. Use that as the sole worker handle when present; otherwise use `terminal list` to resolve the agent handle. Omit `--repo` only inside an Orca-managed worktree; otherwise pass `--repo `.\n\n**For an allowed new worktree, use agent-first:** `--agent` reveals the new worktree and launches the selected agent **in its first terminal**, without adding a separate fallback shell for that worker. Pass `--setup run`; repo setup and default-terminal settings may add intentional tabs or splits. Do **not** run bare `worktree create` and then `terminal create --command ` for the same worker when agent-first create is available: without configured default tabs, that two-step path leaves a fallback shell + agent pair. Only use it when custom agent argv is required (for example Codex model/effort flags) or when an older CLI rejects `--agent`; if you must, message only the agent handle. Configured default tabs are intentional surfaces, so close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell. Do not run `worktree create` when the task must stay in the current worktree.\n\nUse `orca worktree create --prompt ...` or `orca terminal send ...` for full handoffs or untracked/lightweight prompts. Those paths do not attach `taskId`/`dispatchId`; the worker should not send lifecycle messages unless the prompt supplies a live orchestration preamble.\n\nSidebar lineage and orchestration lifecycle are related but not identical. A same-worktree worker may appear as a peer under that worktree in the sidebar while remaining a child dispatch in orchestration state; only an actual child worktree creates visible parent/child worktree lineage.\n\nOther terminal commands coordinators often need:\n\n```bash\norca terminal list [--worktree ] [--include-visual-layouts] [--json]\norca terminal create [--worktree ] [--title ] [--command ] [--json]\norca terminal split --terminal [--direction horizontal|vertical] [--command ] [--json]\norca terminal wait --terminal --for tui-idle --timeout-ms --json\norca terminal read --terminal --json\norca terminal send --terminal --text --enter --json\n```\n\nIf an older CLI rejects `worktree create --agent`, create the worktree normally, then run `orca terminal create --worktree --command \"codex\" --json` or `--command \"claude\"`.\n\nWait for `tui-idle` before dispatching. Always pass `--timeout-ms`; real coding tasks can take 15-60 minutes. During supervision, use rolling `check --wait` windows. If a window returns no matching message, inspect `task-list`, `terminal read`, or `terminal wait --for tui-idle` as a liveness checkpoint; if the terminal is still working or producing activity, keep waiting instead of retrying the task.\n\n## Agent Guidance\n\n- Workers with a valid live preamble must send `worker_done` exactly once from their own terminal with an explicit `--outcome succeeded` or `--outcome failed`:\n `orca orchestration send --type worker_done --subject \"\" --body \"<3-sentence summary: what you did, what you found, what's left>\" --task-id --dispatch-id --outcome succeeded --files-modified \"path/a\" --report-path \"\" --json`\n- A failed outcome is still a terminal report, but Orca records both the Dispatch and Task as failed. Never encode failure only in the subject/body.\n- After sending `worker_done`, end that dispatched turn and idle at the agent prompt. Do not autonomously start more work, poll, or attempt to close the terminal yourself. A direct user instruction takes precedence and starts ordinary user-owned work: follow it without coordinator approval or a fresh Dispatch, never refuse it because of worker/coordinator roles, and do not reuse the settled Dispatch's lifecycle IDs. A coordinator-supervised follow-up still arrives with a fresh preamble + TASK block.\n- For long tasks, send heartbeat/status only when the preamble asks for it, including both IDs:\n `orca orchestration send --type heartbeat --subject \"alive\" --payload '{\"taskId\":\"\",\"dispatchId\":\"\",\"phase\":\"implementing\"}' --json`\n- If blocked before completion, use `ask`; use `escalation` only when ownership is valid and the coordinator must intervene.\n- Treat preambles inherited through terminal history or full handoffs as stale unless the current prompt explicitly keeps that coordinator in the loop.\n- Coordinators must account for every settled worker terminal before waiting again or ending the turn: immediately reuse the exact worker for a new Dispatch, explicitly retain it at the user's request with `worker-retain`, or run `worker-release`. Do not leave a completed worker live merely to inspect output; released workers remain readable through `worker-read`.\n- Coordinators should use `task-list --ready` as external memory, dispatch parallel waves, and avoid dependency chains deeper than 3-4 steps.\n\n## Example\n\n```bash\norca terminal create --worktree active --title login-css-worker --command \"claude\" --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca orchestration task-create --spec \"Fix the login button CSS\" --json\norca orchestration dispatch --task --to --inject --json\norca orchestration check --wait --types worker_done,escalation,question --timeout-ms 900000 --json\n```\n\n## Next Action\n\nCoordinator: confirm `orca status --json`, create or bind a Run, inspect `task-list`/`dispatch-show` if inheriting state, then use the explicit supervised loop (`task-create` -> `worker-start` -> `check --wait`). Use low-level terminal creation plus `dispatch --inject` only when the composed start does not express the needed topology. After every accepted `worker_done`, either transfer the exact terminal to an immediate follow-up Dispatch or run `worker-release` before the next wait.\n\nWorker: if the current prompt contains a live dispatch preamble, do the task, use `ask` for blocking questions, and send `worker_done` once with the required payload. If the preamble is stale or absent, do not send lifecycle messages; inspect state or treat the prompt as an ordinary handoff.\n" +const ORCHESTRATION_MARKDOWN = "---\nname: orchestration\ndescription: >-\n Use Orca orchestration for structured multi-agent coordination: threaded\n messages, blocking ask/reply flows, task dispatch, worker_done/escalation\n waits, task DAGs, decision gates, coordinator loops, or decomposing work\n across agents. Use `orca-cli` instead for full ownership handoffs, including\n requests phrased as \"hand off\", \"handoff\", \"handover\", \"give this to another\n agent\", or \"another worktree\" when the user did not explicitly ask to\n supervise, monitor, wait for results, or coordinate a DAG. Use `orca-cli` for\n ordinary terminal control, lightweight terminal prompts, shell commands, Orca\n worktree management, reading or waiting on terminals, and automation of the\n browser embedded inside Orca. Use Computer Use for browser windows, webviews,\n Orca app UI, or desktop UI outside Orca's embedded browser.\n---\n\n# Orca Inter-Agent Orchestration\n\nOrchestration is Orca's structured coordination layer for agent messages, task ownership, dispatch state, and worker completion tracking.\n\nUse this skill when coordination state matters. For lightweight terminal prompts or basic worktree/terminal/built-in-browser control, use `orca-cli`.\n\n## Tool Boundary\n\nIf a task says to use Orca orchestration, the coordinator must create or bind a Run, create the Task with `orca orchestration task-create`, then attach the worker with either the preferred `orca orchestration worker-start` composition or the low-level `orca orchestration dispatch --inject` path.\n\nDo not substitute non-Orca subagent tools, generic agent-spawn APIs, or chat-only parallel worker features. Those may create useful workers, but they do not create Orca task/dispatch provenance, injected lifecycle preambles, `worker_done` authority, or decision gates.\n\nBefore claiming a worker was orchestrated, verify the task/dispatch exists:\n\n```bash\norca orchestration task-list --json\norca orchestration dispatch-show --task --json\n```\n\nIf the work was accidentally run outside Orca orchestration, say so plainly. To repair provenance, rerun or revalidate the needed work through a fresh Orca terminal plus injected dispatch; do not retroactively describe the external worker as orchestrated.\n\n## When To Use\n\n- Send/reply/ask between agent terminals with persistent messages.\n- Dispatch structured tasks to workers and wait for `worker_done` or `escalation`.\n- Track task DAGs with dependencies.\n- Run coordinator loops or decision gates.\n\nDo not use orchestration merely because the user says \"hand off\", \"handoff\", \"handover\", \"give this to another agent\", or asks for another worktree/agent/model/effort. Those are full ownership transfers unless the user explicitly asks to supervise, monitor, wait for worker completion/results, coordinate a DAG, use decision gates, or keep a blocking ask/reply loop.\n\n## Preconditions\n\n- `orca status --json` should show a running runtime.\n- `orca` must be on PATH (`orca-ide` on Linux).\n- The orchestration experimental feature must be enabled in Settings > Experimental.\n- `orca orchestration` commands are RPC calls to the running Orca runtime.\n\n## Contract Migration\n\nOrca adopts a live pre-update orchestration assignment into an ordinary Run. Adoption preserves the existing agent process, PTY/session, terminal handle, tab/leaf/pane, worktree or folder workspace, Task, and Dispatch; it never restarts or replaces the worker. The retired scheduler is not revived, and a newly created attempt uses the current grammar.\n\nTreat the authority label on injected or formatted messages as definitive:\n\n- `[LEGACY COMPATIBILITY]` is live and attested. Run only the exact supported command printed with the message, using the same CLI executable and arguments that the original prompt supplied.\n- `[LEGACY RECOVERY REPLAY — MAY HAVE BEEN SEEN]` is one bounded, at-least-once cutover replay. Process it idempotently and acknowledge it only through the exact displayed guidance.\n- `[LEGACY READ-ONLY]` is inspection-only. It has no reply, acknowledgment, or lifecycle action.\n- An unlabeled current message uses the current guide and current grammar.\n\nAn explicitly selected current Run, attested current Run binding, current Dispatch, or federated attachment takes precedence over legacy fallback. A retained adoption record alone never turns a current command into a legacy call.\n\nDatabase provenance, an old-looking terminal, or a legacy Run ID does not prove mutation authority. If the runtime cannot prove liveness, principal ownership, capability, or the exact legacy contract, it degrades to read-only inspection and must not fall back to local execution. Exact recovery may restore the already-live PTY once in its original inactive background tab. It must not spawn, write, signal, stop, switch, focus, split, or inject a terminal. Loss of lifecycle authority does not invalidate the existing assignment, process, or filesystem work.\n\nCompatibility retries have narrow guarantees. A pending ask, a reply, a final Dispatch settlement, and a consuming check have durable recovery identities. A-era heartbeat and escalation calls remain at-least-once across a manual A-to-B retry because identical later signals may be intentional. If an A-era ask may already have been answered, run the exact non-consuming recovery check printed by the runtime first; after its answer is printed and acknowledged, a new invocation with the same question creates a new question. Never guess among multiple identical question threads.\n\nWhen a compatibility or recovery command returns structured next-step arguments, run those exact arguments with the same CLI executable. The arguments intentionally omit the executable name so the guidance works with `orca`, `orca-ide`, `orca-dev`, or another configured Orca CLI command. Do not translate the command from memory, broaden its recipient, or retry it as a current mutation unless the returned guidance explicitly says to.\n\nOn packaged Windows, a legacy ask uses a two-step commit/resume protocol. The initial command durably commits the question, prints its exact `ask --resume ` command, and exits with launcher status `75`; it does not wait for the answer. Run that exact resume command after the launcher or update boundary. Resume is idempotent and read-oriented: it waits for the already-committed question and does not create another one. For a WSL process that received compatibility proof at launch, use the printed executable `orca-ide` WSL resume command so the same distro and packaged launcher authority are preserved; do not substitute a PATH-resolved local CLI. Older WSL processes that never received the hidden launch token remain lifecycle read-only after the update, even while their terminal and filesystem work continue.\n\nLegacy inspection remains available without consuming mail:\n\n```bash\norca orchestration run-list --json\n# run_legacy_local is an empty audit tombstone after adoption.\norca orchestration run-show --id run_legacy_local --json\n# In run-list, find the ordinary Run whose objective is:\n# \"Recovered orchestration work from a contract update\"\norca orchestration run-show --id --json\norca orchestration task-list --run --json\norca orchestration inbox --full --json\norca orchestration check --terminal --peek --format --json\norca terminal read --terminal --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\n```\n\nIf the original coordinator is unavailable or cannot prove its retained authority, a current coordinator may explicitly take over the adopted Run from its own live agent terminal:\n\n```bash\norca orchestration run-use --id --takeover-legacy --json\norca orchestration check --run --json\n```\n\nTakeover fences only the old coordinator, binds the current one, and moves pending worker mail into current Run Delivery. It is bound to the authenticated invoking terminal; `--from` cannot name another coordinator. Live legacy workers keep their original Tasks, Dispatches, processes, filesystems, and old prompt commands; their later questions, escalations, and completion reports route to the current coordinator. Do not use takeover while the original coordinator is still actively coordinating, because its later lifecycle mutations are rejected.\n\nDo not launch a replacement editor merely because the desktop app or runtime was updated. If adoption cannot prove continuing authority, keep the original worker as the only editor until it reaches a stable handoff point, then use a new current Dispatch in a conflict-free placement for any remaining work.\n\n## Ownership\n\nNew orchestration messages and tasks belong to one explicitly bound Run. A Run is only a durable namespace and coordinator inbox; it never schedules or places workers. Lifecycle authority comes from the active Dispatch, and terminal handles remain routing metadata rather than durable identity. Send `worker_done` and `heartbeat` from the worker's own terminal; Orca routes them to that Dispatch's Run.\n\nClassify inherited context before sending lifecycle messages:\n\n- Coordinated subtask: a live coordinator owns the DAG and waits on this dispatch. Follow the preamble exactly, including `worker_done`, heartbeat/status, `ask`, and `escalation`.\n- Full handoff means ownership transfer, not supervised dispatch. The original actor is not monitoring a DAG, so do not create lifecycle obligations unless the user explicitly asks you to supervise.\n- Classify requests containing \"hand off\", \"handoff\", \"handover\", \"give this to another agent\", \"give this to another worktree\", \"another agent\", or \"another worktree\" as full handoffs by default, even when the user names a custom model or reasoning effort.\n- Use supervised orchestration only when the user explicitly asks you to \"supervise\", \"monitor\", \"wait\", \"track completion\", \"wait for worker_done\", return results, coordinate a DAG, use a decision gate, or manage ask/reply flow.\n- Do not use `orca orchestration dispatch --inject` for full handoffs. It injects a coordinator preamble that tells the worker to send `worker_done`, heartbeat, and `ask` messages, then end its turn under the original terminal's dispatch lifecycle.\n- Do not run `orca orchestration task-create`, `orca orchestration dispatch --inject`, or `orca orchestration check --wait` for full handoffs. Do not peek at terminal output after prompt delivery to monitor progress.\n- A review-only `worker_done` reports findings; it does not authorize coordinator file edits. After a review-only completion, synthesize findings, ask a decision gate if ownership is unclear, and dispatch or hand off fixes unless the user explicitly asked the coordinator to own fixes.\n- If the user's plan names a next owner agent (for example, \"then use opencode to create a PR\"), post-review corrections and PR prep belong to that named owner. The coordinator routes, synthesizes, asks decision gates when needed, and supervises; the named owner edits files and creates the PR.\n\nIf unclear, inspect orchestration state before sending lifecycle messages:\n\n```bash\norca orchestration task-list --json\norca terminal list --json\n# If inherited context includes a task id:\norca orchestration dispatch-show --task --json\n```\n\n## Messaging\n\n```bash\norca orchestration send --subject [--to ] [--from ] [--body ] [--type ] [--priority ] [--thread-id ] [--payload ] [--json]\norca orchestration check [--terminal ] [--ack ] [--peek|--all] [--types ] [--format] [--wait] [--timeout-ms ] [--json]\norca orchestration reply --id --body [--from ] [--json]\norca orchestration ask (--question |--resume ) [--options ] [--timeout-ms ] [--from ] [--json]\norca orchestration inbox [--limit ] [--json]\n```\n\nRules:\n\n- Omit `--from` unless impersonating another terminal; Orca auto-resolves it from the current terminal.\n- A coordinator `check` returns the bound Run's oldest FIFO Delivery (up to 50 messages) and replays that exact batch until `--ack `. Process every message before acknowledging; `check --ack --wait` acknowledges, checks, and waits in one operation.\n- Use `--peek` and `--all` only for read-only history/debugging. Type filters decide when a waiter wakes; the returned actionable Delivery is still the oldest full batch.\n- Use `dispatch:` for coordinator guidance to one supervised worker. Orca routes that stable address locally or through the connected-server relay; do not substitute a remote terminal handle.\n- Terminal handles remain appropriate for low-level pre-Dispatch messaging. Prefer `agentTerminalHandle` from the create response, fall back to `startupTerminal.handle` for older runtimes, then re-resolve with `orca terminal list --worktree ... --json` if missing or stale. Continue with the replacement handle only; never dual-send to old and new handles.\n- `terminal list --json` omits `visualLayouts` because handle recovery does not need topology. Add `--include-visual-layouts` only for explicit tab and pane inspection.\n- `orca orchestration check --peek --format --json` returns locally formatted unread mail without consuming it; it never writes to terminal input or remotely wakes another terminal. Use `orchestration dispatch --inject` to deliver a tracked task, or `terminal send` when an existing agent needs a free-form prompt.\n- While supervising workers manually, use `check --wait --types worker_done,escalation,question --timeout-ms ` instead of sleep/poll loops. Process the whole Delivery, reply to `question` messages with `orca orchestration reply --id --body --json`, then acknowledge and keep waiting.\n- Treat a `check --wait` timeout or `{count:0}` as a checkpoint, not a worker failure. Long coding tasks routinely run 15-60 minutes; keep using rolling waits unless you receive `worker_done`/`escalation`, the terminal exits or disappears, or the user explicitly asks you to stop.\n- Heartbeats and visible terminal activity mean the worker is alive, not done. Do not stop, close, kill, or restart a worker just because it has not produced a completion message yet.\n- Use `ask` when a worker needs a blocking answer from the coordinator; it defaults to the active Dispatch's Run. Timeout or disconnect leaves the question pending, so resume by its original message ID instead of asking again.\n- `check --wait` returns one bounded Delivery, not every future completion. Process every message, acknowledge it, then keep waiting until every expected Dispatch settles.\n- Group addresses include `@all`, `@idle`, `@claude`, `@codex`, `@opencode`, `@gemini`, `@droid`, `@grok`, `@cursor`, and `@worktree:`.\n- Message types include `status`, `dispatch`, `worker_done`, `merge_ready`, `escalation`, `handoff`, `question`, `decision_gate` (legacy/gates), and `heartbeat`.\n- Use group addresses only for messages that are genuinely useful to many terminals, such as `status` broadcasts or intentional fan-out questions. Do not send dispatch lifecycle messages to groups.\n- `worker_done` belongs to the active Dispatch and defaults to its Run mailbox; never target a group.\n- A valid `worker_done` for the active `taskId` + `dispatchId` marks the task and dispatch completed automatically. Do not follow it with `task-update --status completed`; reserve manual updates for explicit recovery or overrides.\n- `heartbeat` is also Dispatch-scoped. Include both IDs and omit `--to` so Orca uses the owning Run; use `status` for broad progress updates.\n\n## Tasks And Dispatch\n\nA Run is the namespace/inbox, a Task is the work item, and a Dispatch assigns one Task attempt to a terminal. Create or bind a Run once before the common loop.\n\n```bash\norca orchestration run-create --objective --json\norca orchestration task-create --spec [--deps ] [--parent ] [--json]\norca orchestration task-list [--status ] [--ready] [--brief] [--json]\norca orchestration task-update --id --status [--result ] [--json]\norca orchestration dispatch --task --to [--from ] [--inject] [--json]\norca orchestration dispatch-show --task [--json]\n```\n\nTask statuses: `pending`, `ready`, `dispatched`, `completed`, `failed`, `blocked`.\n\nDispatch rules:\n\n- `--inject` sends the task spec plus preamble into a recognized agent CLI so it can report `worker_done`.\n- If the target is a bare shell, omit `--inject`, dispatch for tracking if needed, then send the prompt manually with `orca terminal send --terminal --text --enter --json`.\n- After 3 consecutive failures on one task, the dispatch context circuit-breaks and the task is marked failed.\n- Use `task-list --brief --json` for coordinator sweeps; it collapses whitespace and caps each echoed spec at 160 characters (`spec_truncated` marks shortened rows). Omit `--brief` when the full spec is required, or when an older CLI rejects it as an unknown flag.\n\n## How deep workers can nest\n\nA dispatched worker normally cannot dispatch sub-workers. Attempting it fails with\n`nested_worker_depth_exceeded` and a message telling the worker to complete the task\nitself. Do that — do not try to route around it.\n\nThe limit is a number, not an on/off switch. `Settings -> Agents -> Nested worker depth`\nsets how many generations are allowed:\n\n- `1` (default): a coordinator dispatches workers; those workers do not dispatch.\n- `2`: workers may dispatch one further generation.\n\nDepth is counted from the terminal that issues the command, not from the Run. Creating a\nnew Run does not reset it — a worker that runs `run-create` then `worker-start` is still a\nworker, and still counted. This is the part that changed: the old behaviour rejected\nsub-dispatch only because a worker's terminal was not bound to a Run, so creating a Run was\nenough to slip past it.\n\nTwo limits worth knowing:\n\n- **It is a guardrail, not a security boundary.** A caller that declares another terminal's\n handle while its own launch evidence is unverifiable (an ordinary restored terminal, for\n example) can be counted as that terminal instead. Orca does not treat workers as hostile.\n- **It applies while a Dispatch is active.** After `worker_done`, or after a coordinator\n settles the task, the terminal is no longer a worker and is counted as a root again. The\n process may still be alive; that is the documented boundary, not an accident.\n\n## Preferred Supervised Worker Loop\n\nUse `worker-start` for the normal supervised path. It composes the existing worktree, terminal, readiness, and dispatch primitives while returning exact created/reused effects. Agents still choose placement and concurrency; Orca does not schedule workers or infer conflicts.\n\nCreate the Run and every independent Task first, then start all independent workers before waiting:\n\n```bash\norca orchestration run-create --objective \"\" --json\norca orchestration task-create --spec \"\" --json\norca orchestration task-create --spec \"\" --json\norca orchestration worker-start --task --worktree current --agent codex --json\norca orchestration worker-start --task --worktree current --agent claude --json\n```\n\n`current` and exact existing worktrees create a fresh agent terminal and do not rerun setup. Reuse an existing agent only with `--terminal `.\n\nFor a per-invocation Claude, Codex, or Cursor launch, pass an opaque provider model id with `--model`; add `--effort` only when that agent/model supports the level. These options apply only to fresh agent terminals, override general agent default arguments, and are reported under `launch.requested` and `launch.effective` in the receipt:\n\n```bash\norca orchestration worker-start --task --worktree current --agent claude --model opus --effort high --json\n```\n\n`--effort` requires `--model`, and neither option can combine with `--terminal`. A connected worker server must advertise launch-preference support before Orca forwards either option.\n\nFor a new worktree, setup runs by default and agent-first creation reuses the returned startup agent terminal:\n\n```bash\norca orchestration worker-start --task --worktree new-child --name --agent codex --setup run --json\n# Independent/top-level:\norca orchestration worker-start --task --worktree new-top-level --name --agent codex --setup run --json\n```\n\nSetup normally starts alongside the agent. Only a repository explicitly configured with `wait-for-setup` delays agent launch until setup succeeds. Use `--setup skip` or `--setup inherit` only for a concrete reason.\n\nRead the returned receipt before continuing: `ready` plus setup `running` is normal for start-immediately, while wait-for-setup returns setup `succeeded` before accepting task input. A failed or unknown start exits nonzero; inspect its `stage`, `effects`, and `residualResources` instead of guessing or automatically retrying. A wait-for-setup timeout can honestly leave setup `running`, which is not proof of failure.\n\nTo run the worker on another connected Orca server, add `--on `. The Run and Tasks remain authoritative on the current server; later commands route by Dispatch ID, so never repeat `--on`:\n\n```bash\n# Mac Run home -> Windows worker (the reverse is identical from a Windows Run home)\norca orchestration worker-start --task --on windows --worktree new-top-level --repo --name --agent codex --setup run --json\norca orchestration worker-show --dispatch --json\norca orchestration worker-read --dispatch --limit 50 --json\norca orchestration send --to dispatch: --subject \"Follow-up\" --body \"\" --json\n```\n\nRemote `current` and `new-child` are intentionally invalid because those words are ambiguous across servers. Use an exact discovered remote worktree selector or `new-top-level` with an explicit remote repo selector.\n\nThe follow-up is structured inbox mail, not prompt injection. The worker's next\n`orchestration check` receives it even when the Dispatch is on another connected Orca server.\n\n`worker-read` defaults to `--source auto`: Orca returns the exact hook-reported Codex, Claude, OpenClaude, or Grok transcript when it can prove the worker session, otherwise it returns bounded terminal output with `source: \"terminal\"` and a typed `fallbackReason`. Continue with the returned top-level `cursor`; it stays pinned to that exact source. If Orca reports `source_changed`, start a fresh read without the old cursor. Never supply or guess a provider session ID or transcript path.\n\nWait until every expected Dispatch settles, not for a fixed number of batches:\n\n```bash\norca orchestration check --wait --types worker_done,escalation,question --timeout-ms 900000 --json\n# Process every message. For each accepted worker_done that is not immediately reused:\norca orchestration worker-release --dispatch --json\n# Acknowledge only after every message and required release decision is handled:\norca orchestration check --ack --wait --types worker_done,escalation,question --timeout-ms 900000 --json\n```\n\nAfter processing each accepted `worker_done`, choose the terminal's next owner before you acknowledge the Delivery or wait again. If the same exact agent has an immediate follow-up Task, read the `worker.agent_terminal_handle` field of `worker-show --dispatch --json`, then run `orca orchestration worker-start --task --terminal --json` so Orca transfers cleanup ownership to the new Dispatch. Otherwise run `orca orchestration worker-release --dispatch --json`.\n\nRun `worker-release` after both succeeded and failed `worker_done` reports unless the user explicitly asked to keep that worker live. Release is post-completion cleanup, not cancellation: Orca first preserves inspectable output, then closes only the exact agent terminal owned by that settled Dispatch. Reused or pre-existing terminals, setup terminals, coordinators, active workers, user-taken-over terminals, and identities Orca cannot prove are retained. If the user explicitly asks to keep the live terminal for debugging, record that exception with `orca orchestration worker-retain --dispatch --json` instead of silently skipping cleanup. When the user is finished, the same Dispatch can be passed to `worker-release`, which clears the requested retention and releases the terminal.\n\nDo not release a worker because of a timeout, TUI idle state, heartbeat, status, question, escalation, or rejected/stale `worker_done`. If release returns `release_pending` or `release_unknown`, do not substitute `terminal close`; follow the exact recovery action in the receipt. A replayed Delivery may repeat `worker-release` safely.\n\nWorkers report exactly once using the IDs and capability injected by Orca; they do not supply Run/server/terminal identity:\n\n```bash\norca orchestration send --type worker_done --subject \"\" --body \"\" --task-id --dispatch-id --outcome succeeded --files-modified \"path/a,path/b\" --json\n# On failure, use --outcome failed; never encode failure only in prose.\n```\n\nA worker question defaults to its owning Run. Timeout leaves it pending:\n\n```bash\norca orchestration ask --question \"\" --options \"yes,no\" --timeout-ms 600000 --json\norca orchestration ask --resume --timeout-ms 600000 --json\n# Coordinator:\norca orchestration reply --id --body \"\" --json\n```\n\nRecovery is conditional, never a fixed destructive sequence:\n\n- `worker-show --dispatch ` says `ready`: keep waiting or read bounded output.\n- It proves `failed` or `stopped`: start a replacement with `worker-start --task --retry-of ` plus an explicit `--on`/`--worktree` and `--agent`/`--terminal` choice. Retry does not silently inherit placement.\n- It remains `outcome_unknown`: either `worker-stop --dispatch ` and inspect again, or explicitly `worker-abandon --dispatch ` while accepting that resources may still be live. Abandon performs no remote, process, or filesystem action.\n- `worker-stop` closes only the exact supervised agent terminal. It never deletes the worktree, setup terminal, configured tabs, or unrelated processes.\n\nLow-level `worktree create`, `terminal create`, and `dispatch --inject` remain valid recipes for custom argv or topology that `worker-start` does not express.\n\n`dispatch --inject` deliberately keeps an operator-started terminal unsupervised: it never creates a `worker_dispatches` row and `worker-stop`/`worker-abandon` never close that process. The dispatch context is still authoritative, so `worker-show`, `worker-read`, and `worker-list` report it as `unsupervised`; settled `worker-retain` and `worker-release` report `retained` with `no_owned_resource` and take no process action. Use `worker-start --terminal ` when supervision and worker lifecycle state are required.\n\n## Gates And Legacy Inspection\n\n```bash\norca orchestration gate-create --task --question [--options ] [--json]\norca orchestration gate-resolve --id --resolution [--json]\norca orchestration gate-list [--task ] [--status ] [--json]\n```\n\nUse `ask` for worker-to-coordinator questions; it creates a `question` message that the coordinator answers with `reply`. Use `gate-create` only for coordinator-managed task DAG decisions, not for answering a worker's `ask`.\n\n`coordinator-start`, `coordinator-stop`, `run`, and `run-stop` are retired scheduler commands. They perform no effects and return the current-skill recovery action. They are not aliases for lightweight Run creation or binding.\n\nRecovery only: `orca orchestration reset --tasks|--messages|--all --json` clears the selected local orchestration database state. Do not run it during active coordination unless explicitly abandoning that state.\n\n## Full Handoffs\n\nFor full ownership transfer, use non-lifecycle terminal/worktree commands and then stop monitoring unless the user asks for supervision.\n\nTreat these as full handoff requests by default: \"hand off\", \"handoff\", \"handover\", \"give this to another agent\", \"give this to another worktree\", \"send this to another agent\", \"another agent\", \"another worktree\", or \"launch another agent to own this.\" Custom model or reasoning effort words such as `gpt-5.5`, `high`, or `xhigh` do not make the handoff supervised.\n\nSupervised orchestration remains available only when the user explicitly asks for supervision or coordination: \"supervise\", \"monitor\", \"wait for worker_done\", \"wait for results\", \"track completion\", \"DAG\", \"decision gate\", \"ask/reply\", or \"coordinate workers.\"\n\nDo not run `orca orchestration task-create`, `orca orchestration dispatch --inject`, or `orca orchestration check --wait` for full handoffs. `task-create` is also forbidden because it records coordinator-owned tracking state; if a task row is needed, the user asked for supervised orchestration. Do not create a `taskId`/`dispatchId`, inject a lifecycle preamble, wait for completion, or read the worker terminal after prompt delivery except to avoid losing the initial prompt.\n\nNew top-level worktree handoff:\n\n```bash\norca worktree create --name --no-parent --agent codex --prompt \"\" --setup run --json\n```\n\nBefore creating a new worktree from an active feature branch, decide and state whether the desired Orca lineage is child or top-level. Use child worktree lineage only when the new work is conceptually stacked under or dependent on the active worktree. For independent repo-wide fixes, standalone feature work, or unrelated follow-up tasks, create a top-level worktree with `--no-parent`.\n\nExisting terminal handoff:\n\n```bash\norca terminal send --terminal --text \"\" --enter --json\n```\n\nCustom Codex model/effort handoff:\n\n`orca worktree create --agent codex --prompt ...` launches the known Codex agent but does not accept Codex-specific `--model` or `-c model_reasoning_effort=...` arguments. When the user asks for a specific Codex model or effort, create the independent worktree first, launch Codex with the requested command in that worktree, wait only for TUI readiness if prompt delivery would otherwise race startup, send the prompt, and stop.\n\nThe two-step custom-argv path cannot enforce a repository's explicit `wait-for-setup` startup policy because the later `terminal create` is not the startup owned by `worktree create`. Use it only when the repository starts agents immediately. If the repository requires `wait-for-setup`, use an agent-first configured launcher that can preserve sequencing, or stop and ask rather than silently bypassing the policy.\n\nNote: when no repo default-terminal configuration supplies a primary terminal, bare create opens a fallback shell before `terminal create` adds the agent. Configured default tabs are materialized instead and may run real commands. Prefer `--agent` whenever custom argv is not required. With the two-step path, target only the agent handle; close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell.\n\nUse the exact full `::` worktree id returned by `orca worktree create --json`; a bare repo id cannot target the new worktree.\n\n```bash\norca worktree create --name --no-parent --setup run --json\norca terminal create --worktree id: --title --command 'codex --model gpt-5.5 -c model_reasoning_effort=\"xhigh\"' --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca terminal send --terminal --text \"\" --enter --json\n```\n\nWait only for `tui-idle` when needed to avoid losing the prompt. Do not monitor task completion.\n\n`--no-parent` only controls Orca lineage; it does not choose the Git base. If the work should start from the repo default base, omit `--base-branch` so Orca uses that default, or explicitly pass the repo default base (`origin/main`, `origin/master`, or the `orca repo show --repo --json` value); never base it on the current feature branch unless the user explicitly asks for stacked work or \"branch from current\". Put current-branch context in the prompt instead.\n\n## Worker Terminals\n\nChoose the worker location before creating a terminal. `Fresh worker` means a fresh agent session, not a new git worktree. For parallel work, create one fresh agent terminal per worker in the same required worktree, falling back to the active worktree when none is named. If the task says current worktree only, depends on uncommitted files/artifacts, or must validate/PR the current branch, keep every worker in the active worktree:\n\n```bash\norca terminal create --worktree active --title --command \"codex\" --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca orchestration dispatch --task --to --inject --json\n```\n\nReuse an idle agent in the required worktree only if the prompt allows reuse; otherwise create a fresh terminal there. Create a new worktree only when the user explicitly requests one or a concrete checkout or filesystem conflict makes sharing unsafe or impossible; if the user did not request it, state that conflict before running `worktree create`. Independent tasks, parallel execution, convenience, or a preference for separate checkouts are not isolation requirements.\n\nWhen a new worktree is allowed, use child lineage for isolated work that is stacked under or dependent on the active worktree, and use `--no-parent` when it is not stacked. Decide the Git base separately: `--no-parent` makes the worktree top-level in Orca, while omitted `--base-branch` uses the repo default base.\n\nFor every new worktree, pass `--setup run` so any configured repository setup hook runs. This does not mean waiting for setup before agent launch: preserve the repository's startup policy, whose default starts setup and the agent side by side. Use `--setup skip` or `--setup inherit` only when there is a concrete task-specific reason, and state that reason before creating the worktree. This rule does not rerun setup for current or existing worktrees.\n\n```bash\norca worktree create --name --agent codex --setup run --json\n# or: --agent claude | omp | pi | grok | ...\n# Read from agentTerminalHandle, falling back to startupTerminal.handle.\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca orchestration dispatch --task --to --inject --json\n```\n\nFor new-worktree workers, read the id and `agentTerminalHandle` from `worktree create`, falling back to `startupTerminal.handle` for older runtimes. Use that as the sole worker handle when present; otherwise use `terminal list` to resolve the agent handle. Omit `--repo` only inside an Orca-managed worktree; otherwise pass `--repo `.\n\n**For an allowed new worktree, use agent-first:** `--agent` reveals the new worktree and launches the selected agent **in its first terminal**, without adding a separate fallback shell for that worker. Pass `--setup run`; repo setup and default-terminal settings may add intentional tabs or splits. Do **not** run bare `worktree create` and then `terminal create --command ` for the same worker when agent-first create is available: without configured default tabs, that two-step path leaves a fallback shell + agent pair. Only use it when custom agent argv is required (for example Codex model/effort flags) or when an older CLI rejects `--agent`; if you must, message only the agent handle. Configured default tabs are intentional surfaces, so close a prior terminal only after `terminal list` or `terminal show` confirms it is an unused shell. Do not run `worktree create` when the task must stay in the current worktree.\n\nUse `orca worktree create --prompt ...` or `orca terminal send ...` for full handoffs or untracked/lightweight prompts. Those paths do not attach `taskId`/`dispatchId`; the worker should not send lifecycle messages unless the prompt supplies a live orchestration preamble.\n\nSidebar lineage and orchestration lifecycle are related but not identical. A same-worktree worker may appear as a peer under that worktree in the sidebar while remaining a child dispatch in orchestration state; only an actual child worktree creates visible parent/child worktree lineage.\n\nOther terminal commands coordinators often need:\n\n```bash\norca terminal list [--worktree ] [--include-visual-layouts] [--json]\norca terminal create [--worktree ] [--title ] [--command ] [--json]\norca terminal split --terminal [--direction horizontal|vertical] [--command ] [--json]\norca terminal wait --terminal --for tui-idle --timeout-ms --json\norca terminal read --terminal --json\norca terminal send --terminal --text --enter --json\n```\n\nIf an older CLI rejects `worktree create --agent`, create the worktree normally, then run `orca terminal create --worktree --command \"codex\" --json` or `--command \"claude\"`.\n\nWait for `tui-idle` before dispatching. Always pass `--timeout-ms`; real coding tasks can take 15-60 minutes. During supervision, use rolling `check --wait` windows. If a window returns no matching message, inspect `task-list`, `terminal read`, or `terminal wait --for tui-idle` as a liveness checkpoint; if the terminal is still working or producing activity, keep waiting instead of retrying the task.\n\n## Agent Guidance\n\n- Workers with a valid live preamble must send `worker_done` exactly once from their own terminal with an explicit `--outcome succeeded` or `--outcome failed`:\n `orca orchestration send --type worker_done --subject \"\" --body \"<3-sentence summary: what you did, what you found, what's left>\" --task-id --dispatch-id --outcome succeeded --files-modified \"path/a\" --report-path \"\" --json`\n- A failed outcome is still a terminal report, but Orca records both the Dispatch and Task as failed. Never encode failure only in the subject/body.\n- After sending `worker_done`, end that dispatched turn and idle at the agent prompt. Do not autonomously start more work, poll, or attempt to close the terminal yourself. A direct user instruction takes precedence and starts ordinary user-owned work: follow it without coordinator approval or a fresh Dispatch, never refuse it because of worker/coordinator roles, and do not reuse the settled Dispatch's lifecycle IDs. A coordinator-supervised follow-up still arrives with a fresh preamble + TASK block.\n- For long tasks, send heartbeat/status only when the preamble asks for it, including both IDs:\n `orca orchestration send --type heartbeat --subject \"alive\" --payload '{\"taskId\":\"\",\"dispatchId\":\"\",\"phase\":\"implementing\"}' --json`\n- If blocked before completion, use `ask`; use `escalation` only when ownership is valid and the coordinator must intervene.\n- Treat preambles inherited through terminal history or full handoffs as stale unless the current prompt explicitly keeps that coordinator in the loop.\n- Coordinators must account for every settled worker terminal before waiting again or ending the turn: immediately reuse the exact worker for a new Dispatch, explicitly retain it at the user's request with `worker-retain`, or run `worker-release`. Do not leave a completed worker live merely to inspect output; released workers remain readable through `worker-read`.\n- Coordinators should use `task-list --ready` as external memory, dispatch parallel waves, and avoid dependency chains deeper than 3-4 steps.\n\n## Example\n\n```bash\norca terminal create --worktree active --title login-css-worker --command \"claude\" --json\norca terminal wait --terminal --for tui-idle --timeout-ms 60000 --json\norca orchestration task-create --spec \"Fix the login button CSS\" --json\norca orchestration dispatch --task --to --inject --json\norca orchestration check --wait --types worker_done,escalation,question --timeout-ms 900000 --json\n```\n\n## Next Action\n\nCoordinator: confirm `orca status --json`, create or bind a Run, inspect `task-list`/`dispatch-show` if inheriting state, then use the explicit supervised loop (`task-create` -> `worker-start` -> `check --wait`). Use low-level terminal creation plus `dispatch --inject` only when the composed start does not express the needed topology. After every accepted `worker_done`, either transfer the exact terminal to an immediate follow-up Dispatch or run `worker-release` before the next wait.\n\nWorker: if the current prompt contains a live dispatch preamble, do the task, use `ask` for blocking questions, and send `worker_done` once with the required payload. If the preamble is stale or absent, do not send lifecycle messages; inspect state or treat the prompt as an ordinary handoff.\n" // Why: no current guide has bundled reference documents, so --full is byte-identical for now. // oxfmt-ignore diff --git a/src/main/persistence/applying-settings/settings-update.ts b/src/main/persistence/applying-settings/settings-update.ts index ff4c55b09a8..c4ded8da753 100644 --- a/src/main/persistence/applying-settings/settings-update.ts +++ b/src/main/persistence/applying-settings/settings-update.ts @@ -1,5 +1,6 @@ import type { GlobalSettings } from '../../../shared/global-settings-types' import { normalizeDisabledTuiAgents } from '../../../shared/tui-agent-selection' +import { resolveNestedWorkerMaxDepth } from '../../../shared/nested-worker-depth' import { normalizeTuiAgentArgsRecord, normalizeTuiAgentEnvRecord @@ -73,6 +74,11 @@ export function updateSettings( if ('agentSkillSharingEnabled' in updates) { sanitizedUpdates.agentSkillSharingEnabled = updates.agentSkillSharingEnabled === true } + if ('nestedWorkerMaxDepth' in updates) { + sanitizedUpdates.nestedWorkerMaxDepth = resolveNestedWorkerMaxDepth({ + nestedWorkerMaxDepth: updates.nestedWorkerMaxDepth + }) + } if ('disabledTuiAgents' in updates) { sanitizedUpdates.disabledTuiAgents = normalizeDisabledTuiAgents(updates.disabledTuiAgents) } diff --git a/src/main/runtime/exit-provenance-audit.test.ts b/src/main/runtime/exit-provenance-audit.test.ts index ee6b713dd61..38865c2269d 100644 --- a/src/main/runtime/exit-provenance-audit.test.ts +++ b/src/main/runtime/exit-provenance-audit.test.ts @@ -7,6 +7,7 @@ import { join } from 'node:path' import { OrcaRuntimeService } from './orca-runtime' import { OrchestrationDb } from './orchestration/db' import type { DispatchContextRow } from './orchestration/types' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' const TAB_ID = '11111111-1111-4111-8111-111111111111' const LEAF_ID = '22222222-2222-4222-8222-222222222222' @@ -71,7 +72,10 @@ function dispatchOnHandle( coordinatorPaneKey: '99999999-9999-4999-8999-999999999999:88888888-8888-4888-8888-888888888888' }) const task = db.createTask({ spec, runId: run.id }) - return { ...db.createDispatchContext(task.id, HANDLE, PANE_KEY), runId: run.id } + return { + ...createRootDispatch(db, task.id, HANDLE, PANE_KEY), + runId: run.id + } } /** Where a lightweight Run's coordinator actually reads its mail (STA-4604). */ diff --git a/src/main/runtime/orca-runtime.test.ts b/src/main/runtime/orca-runtime.test.ts index 381ed183516..1f9d810e977 100644 --- a/src/main/runtime/orca-runtime.test.ts +++ b/src/main/runtime/orca-runtime.test.ts @@ -144,6 +144,7 @@ import { } from './terminal-view-attribute-store' import { clearConfiguredWorktreeSharedDirectoriesCacheForTests } from '../git/worktree-shared-directories' import { setWorktreeWatcherRemoval } from '../ipc/worktree-watcher-removal' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' const ORIGINAL_PLATFORM = process.platform const ORIGINAL_PLATFORM_DESCRIPTOR = Object.getOwnPropertyDescriptor(process, 'platform') @@ -22004,6 +22005,8 @@ describe('OrcaRuntimeService', () => { try { const task = db.createTask({ spec: 'continue after missing worker recovery' }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: { topology: 'current', agent: 'codex' } }) @@ -22109,6 +22112,8 @@ describe('OrcaRuntimeService', () => { try { const task = db.createTask({ spec: 'retry missing worker recovery' }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: { topology: 'current', agent: 'codex' } }) @@ -43844,11 +43849,12 @@ describe('OrcaRuntimeService', () => { ['worker-folder', runB.id] ].map(([name, runId]) => { const task = db.createTask({ spec: name, runId }) - return [name, db.createDispatchContext(task.id, handles[name], paneKey(name))] + return [name, createRootDispatch(db, task.id, handles[name], paneKey(name))] }) ) const legacyTask = db.createTask({ spec: 'legacy worker' }) - const legacyDispatch = db.createDispatchContext( + const legacyDispatch = createRootDispatch( + db, legacyTask.id, handles['legacy-worker'], paneKey('legacy-worker') @@ -43976,7 +43982,8 @@ describe('OrcaRuntimeService', () => { expect(creatorAuthority?.processIncarnation).toBeTruthy() expect(coordinatorAuthority?.processIncarnation).toBeTruthy() const creatorTask = db.createTask({ spec: 'create nested work', runId: runA.id }) - db.createDispatchContext( + createRootDispatch( + db, creatorTask.id, handles.creator, paneKey('creator'), @@ -43991,7 +43998,8 @@ describe('OrcaRuntimeService', () => { createdByProcessIncarnation: creatorAuthority?.processIncarnation ?? undefined, createdByRunGeneration: runA.consumer_generation }) - const workerDispatch = db.createDispatchContext( + const workerDispatch = createRootDispatch( + db, workerTask.id, handles.worker, paneKey('worker') @@ -44004,7 +44012,8 @@ describe('OrcaRuntimeService', () => { createdByProcessIncarnation: coordinatorAuthority?.processIncarnation ?? undefined, createdByRunGeneration: runA.consumer_generation }) - const coordinatorCreatedDispatch = db.createDispatchContext( + const coordinatorCreatedDispatch = createRootDispatch( + db, coordinatorCreatedTask.id, handles['coordinator-created-worker'], paneKey('coordinator-created-worker') @@ -44092,7 +44101,8 @@ describe('OrcaRuntimeService', () => { coordinatorPaneKey: makePaneKey(terminals[99].tabId, terminals[99].leafId) }) const task = db.createTask({ spec: 'one dispatched terminal', runId: run.id }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, handles[0], makePaneKey(terminals[0].tabId, terminals[0].leafId) diff --git a/src/main/runtime/orca-runtime.ts b/src/main/runtime/orca-runtime.ts index b86135d76a3..a50cef258e6 100644 --- a/src/main/runtime/orca-runtime.ts +++ b/src/main/runtime/orca-runtime.ts @@ -18,6 +18,7 @@ import { assertAgentSkillSharingAllowed, isAgentSkillSharingEnabled } from '../../shared/agent-skill-sharing-gate' +import { resolveNestedWorkerMaxDepth } from '../../shared/nested-worker-depth' import { sortDirEntries } from '../../shared/file-name-sort' import { isServerDriveListRequest, listWindowsDrives } from './windows-drive-listing' import { extractLastOsc7Uri, extractOscScanTail } from '../daemon/osc7-uri-extraction' @@ -1407,6 +1408,7 @@ type RuntimeStore = { prBotAuthorOverrides?: GlobalSettings['prBotAuthorOverrides'] artifactSharingEnabled?: GlobalSettings['artifactSharingEnabled'] agentSkillSharingEnabled?: GlobalSettings['agentSkillSharingEnabled'] + nestedWorkerMaxDepth?: GlobalSettings['nestedWorkerMaxDepth'] terminalQuickCommands?: GlobalSettings['terminalQuickCommands'] gitlabProjects?: GlobalSettings['gitlabProjects'] mobileAutoRestoreFitMs?: number | null @@ -5283,6 +5285,11 @@ export class OrcaRuntimeService { assertAgentSkillSharingAllowed(() => isAgentSkillSharingEnabled(this.store?.getSettings())) } + /** Renderer-owned; read here because dispatch enforcement lives in main. */ + getNestedWorkerMaxDepth(): number { + return resolveNestedWorkerMaxDepth(this.store?.getSettings()) + } + async publishDiscoveredSkillsFromAgent( request: AgentSkillShareRequest, discoveredSkills: readonly DiscoveredSkill[], diff --git a/src/main/runtime/orchestration-mailbox-detached-routing.test.ts b/src/main/runtime/orchestration-mailbox-detached-routing.test.ts index 654f01acef1..e1d484d3b14 100644 --- a/src/main/runtime/orchestration-mailbox-detached-routing.test.ts +++ b/src/main/runtime/orchestration-mailbox-detached-routing.test.ts @@ -16,6 +16,7 @@ import { temporaryDirectories, TERMINAL_HANDLE } from './orchestration-mailbox-notification-test-harness' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' vi.mock('electron', () => ({ app: { getPath: vi.fn(() => tmpdir()), isPackaged: false }, @@ -43,7 +44,7 @@ describe('orchestration detached mailbox routing', () => { '33333333-3333-4333-8333-333333333333:44444444-4444-4444-8444-444444444444' }) const task = db.createTask({ spec: 'Worker task', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) await driveToLiveIdle(harness.runtime) const message = db.insertMessage({ from: 'term_coordinator', @@ -118,7 +119,8 @@ describe('orchestration detached mailbox routing', () => { '55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666' }) const task = db.createTask({ spec: 'Worker task', runId: workerRun.id }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, 'term_mailbox_before_remint', `99999999-9999-4999-8999-999999999999:${LEAF_ID}` @@ -356,7 +358,7 @@ describe('orchestration detached mailbox routing', () => { '55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666' }) const task = db.createTask({ spec: 'Reminted worker', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_before_remint', PANE_KEY) + const dispatch = createRootDispatch(db, task.id, 'term_before_remint', PANE_KEY) const waiting = harness.runtime.waitForMessage(`dispatch:${dispatch.id}`, { typeFilter: ['dispatch'], timeoutMs: 5_000 @@ -430,7 +432,7 @@ describe('orchestration detached mailbox routing', () => { coordinatorPaneKey: PANE_KEY }) const task = db.createTask({ spec: 'Same-handle worker', runId: run.id }) - db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) const message = db.insertMessage({ from: 'term_sender', to: TERMINAL_HANDLE, @@ -459,7 +461,7 @@ describe('orchestration detached mailbox routing', () => { coordinatorPaneKey: PANE_KEY }) const task = db.createTask({ spec: 'Same-handle worker', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) const message = db.insertMessage({ from: 'term_sender', to: TERMINAL_HANDLE, @@ -501,7 +503,7 @@ describe('orchestration detached mailbox routing', () => { .prepare('UPDATE messages SET to_handle = ? WHERE id = ?') .run(TERMINAL_HANDLE, directMessage.id) const task = db.createTask({ spec: 'Dispatch migration', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_dispatch', PANE_KEY) + const dispatch = createRootDispatch(db, task.id, 'term_dispatch', PANE_KEY) db.insertMessage({ from: 'term_sender', to: `dispatch:${dispatch.id}`, diff --git a/src/main/runtime/orchestration-mailbox-notification-consistency.test.ts b/src/main/runtime/orchestration-mailbox-notification-consistency.test.ts index 38f574d9491..bbcbc42e564 100644 --- a/src/main/runtime/orchestration-mailbox-notification-consistency.test.ts +++ b/src/main/runtime/orchestration-mailbox-notification-consistency.test.ts @@ -28,6 +28,7 @@ import { } from './orchestration-mailbox-notification-test-harness' import { RpcDispatcher } from './rpc/dispatcher' import { ORCHESTRATION_METHODS } from './rpc/methods/orchestration' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' vi.mock('electron', () => ({ app: { getPath: vi.fn(() => tmpdir()), isPackaged: false }, @@ -750,7 +751,7 @@ describe('orchestration notification mailbox consistency', () => { '55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666' }) const task = db.createTask({ spec: 'Worker task', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) for (let index = 0; index < 50; index += 1) { insertDirectRunMessage(db, run.id, `Worker status ${index}`) } diff --git a/src/main/runtime/orchestration-mailbox-routing-races.test.ts b/src/main/runtime/orchestration-mailbox-routing-races.test.ts index 935f05c2877..ff79fabc065 100644 --- a/src/main/runtime/orchestration-mailbox-routing-races.test.ts +++ b/src/main/runtime/orchestration-mailbox-routing-races.test.ts @@ -19,6 +19,7 @@ import { temporaryDirectories, TERMINAL_HANDLE } from './orchestration-mailbox-notification-test-harness' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' vi.mock('electron', () => ({ app: { getPath: vi.fn(() => tmpdir()), isPackaged: false }, @@ -69,7 +70,7 @@ describe('orchestration mailbox routing races', () => { coordinatorPaneKey: SECOND_PANE_KEY }) const task = db.createTask({ spec: 'Worker task', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) for (let index = 0; index < 151; index += 1) { insertDirectRunMessage(db, run.id, `Before completion ${index}`) } @@ -152,7 +153,7 @@ describe('orchestration mailbox routing races', () => { coordinatorPaneKey: SECOND_PANE_KEY }) const task = db.createTask({ spec: 'Waiting worker', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) const status = insertDirectRunMessage(db, run.id, 'Filtered-out status') const controller = new AbortController() const waiting = dispatchMailboxCheck(harness.runtime, { @@ -201,7 +202,7 @@ describe('orchestration mailbox routing races', () => { coordinatorPaneKey: SECOND_PANE_KEY }) const task = db.createTask({ spec: 'Cancelled worker check', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, TERMINAL_HANDLE, PANE_KEY) + const dispatch = createRootDispatch(db, task.id, TERMINAL_HANDLE, PANE_KEY) for (let index = 0; index < 151; index += 1) { insertDirectRunMessage(db, run.id, `Before cancelled migration ${index}`) } @@ -296,7 +297,7 @@ describe('orchestration mailbox routing races', () => { '55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666' }) const task = db.createTask({ spec: 'Worker task', runId: run.id }) - db.createDispatchContext(task.id, 'term_previous_worker', PANE_KEY) + createRootDispatch(db, task.id, 'term_previous_worker', PANE_KEY) const current = insertDirectRunMessage(db, run.id, 'Current worker handle') const previous = db.insertMessage({ from: 'term_coordinator', @@ -421,9 +422,9 @@ describe('orchestration mailbox routing races', () => { '55555555-5555-4555-8555-555555555555:66666666-6666-4666-8666-666666666666' }) const task = db.createTask({ spec: 'Valid worker', runId: run.id }) - const valid = db.createDispatchContext(task.id, 'term_old', PANE_KEY) + const valid = createRootDispatch(db, task.id, 'term_old', PANE_KEY) const collisionTask = db.createTask({ spec: 'Malformed collision', runId: run.id }) - db.createDispatchContext(collisionTask.id, 'term_collision', `:${LEAF_ID}`) + createRootDispatch(db, collisionTask.id, 'term_collision', `:${LEAF_ID}`) expect(db.getActiveDispatchForIdentity('term_reminted', PANE_KEY)?.id).toBe(valid.id) const plan = sqliteFor(db) diff --git a/src/main/runtime/orchestration/coordinator-decision-gates.test.ts b/src/main/runtime/orchestration/coordinator-decision-gates.test.ts index e5a76ab6a22..15e0cd2c7b0 100644 --- a/src/main/runtime/orchestration/coordinator-decision-gates.test.ts +++ b/src/main/runtime/orchestration/coordinator-decision-gates.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it } from 'vitest' import { openDecisionGateFromMessage } from './coordinator-decision-gates' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('coordinator decision-gate authority', () => { let db: OrchestrationDb @@ -12,7 +13,7 @@ describe('coordinator decision-gate authority', () => { it('opens a gate only for the sender-owned active Dispatch', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'owned gate target' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', 'tab_owner:leaf_owner') + const dispatch = createRootDispatch(db, task.id, 'term_owner', 'tab_owner:leaf_owner') const logs: string[] = [] openDecisionGateFromMessage( @@ -41,13 +42,14 @@ describe('coordinator decision-gate authority', () => { it('rejects a gate targeting another active Dispatch without mutating either Task', () => { db = new OrchestrationDb(':memory:') const attackerTask = db.createTask({ spec: 'attacker assignment' }) - const attacker = db.createDispatchContext( + const attacker = createRootDispatch( + db, attackerTask.id, 'term_attacker', 'tab_attacker:leaf_attacker' ) const victimTask = db.createTask({ spec: 'victim assignment' }) - const victim = db.createDispatchContext(victimTask.id, 'term_victim', 'tab_victim:leaf_victim') + const victim = createRootDispatch(db, victimTask.id, 'term_victim', 'tab_victim:leaf_victim') const logs: string[] = [] openDecisionGateFromMessage( @@ -78,7 +80,7 @@ describe('coordinator decision-gate authority', () => { it('accepts the canonical sender of an imported federated Dispatch', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'remote gate target' }) - const dispatch = db.createDispatchContext(task.id, 'remote-worker') + const dispatch = createRootDispatch(db, task.id, 'remote-worker') openDecisionGateFromMessage( db, diff --git a/src/main/runtime/orchestration/coordinator-escalation-triage.test.ts b/src/main/runtime/orchestration/coordinator-escalation-triage.test.ts index af973100336..c020e62dca0 100644 --- a/src/main/runtime/orchestration/coordinator-escalation-triage.test.ts +++ b/src/main/runtime/orchestration/coordinator-escalation-triage.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it } from 'vitest' import { applyEscalationToDispatch } from './coordinator-escalation-triage' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('coordinator escalation authority', () => { let db: OrchestrationDb @@ -12,13 +13,14 @@ describe('coordinator escalation authority', () => { it('rejects an escalation targeting another active Dispatch', () => { db = new OrchestrationDb(':memory:') const attackerTask = db.createTask({ spec: 'attacker assignment' }) - const attacker = db.createDispatchContext( + const attacker = createRootDispatch( + db, attackerTask.id, 'term_attacker', 'tab_attacker:leaf_attacker' ) const victimTask = db.createTask({ spec: 'victim assignment' }) - const victim = db.createDispatchContext(victimTask.id, 'term_victim') + const victim = createRootDispatch(db, victimTask.id, 'term_victim') const logs: string[] = [] applyEscalationToDispatch( @@ -42,7 +44,7 @@ describe('coordinator escalation authority', () => { it('accepts the canonical sender of an imported federated Dispatch', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'remote escalation target' }) - const dispatch = db.createDispatchContext(task.id, 'remote-worker') + const dispatch = createRootDispatch(db, task.id, 'remote-worker') applyEscalationToDispatch( db, diff --git a/src/main/runtime/orchestration/coordinator-runtime-contract.ts b/src/main/runtime/orchestration/coordinator-runtime-contract.ts index f01f4188ea7..742434b1b80 100644 --- a/src/main/runtime/orchestration/coordinator-runtime-contract.ts +++ b/src/main/runtime/orchestration/coordinator-runtime-contract.ts @@ -26,6 +26,8 @@ export type CoordinatorRuntime = { probeWorktreeDrift(worktreeSelector: string): Promise // Why: pane-only fallback preserves reservation identity for lightweight runtime fakes. getTerminalPaneKey?(handle: string): string | null + // Why optional: lightweight fakes omit it and get the fail-closed default. + getNestedWorkerMaxDepth?(): number // Why: automatic dispatch persists the same authenticated pane/process tuple as manual dispatch. getOrchestrationDispatchAuthority?(handle: string): { paneKey: string | null diff --git a/src/main/runtime/orchestration/coordinator-task-dispatch.ts b/src/main/runtime/orchestration/coordinator-task-dispatch.ts index 51e2d834df5..1a882ca3b73 100644 --- a/src/main/runtime/orchestration/coordinator-task-dispatch.ts +++ b/src/main/runtime/orchestration/coordinator-task-dispatch.ts @@ -68,6 +68,7 @@ export async function dispatchTaskToWorker(params: { onLog: (msg: string) => void // Why: the coordinator owns the failed-task list, so a circuit break is reported back instead of mutated here. onCircuitBroken: (taskId: string) => void + nestedWorkerMaxDepth: number }): Promise { const { db, runtime, task, targetHandle, baseDrift, onLog } = params // Why (§3.1): drift check runs before createDispatchContext so a refusal doesn't bump failure_count (carried forward as MAX in db.ts:301-306) and burn the circuit-breaker budget; the task stays `ready` and retries next tick. @@ -95,13 +96,17 @@ export async function dispatchTaskToWorker(params: { dispatchAuthority?.paneKey && dispatchAuthority.processIncarnation ? dispatchAuthority.processIncarnation : undefined - const dispatch = db.createDispatchContext( - task.id, - targetHandle, + const dispatch = db.createDispatchContext({ + taskId: task.id, + assigneeHandle: targetHandle, assigneePaneKey, - dispatchAuthority?.launchTokenHash ?? undefined, - processIncarnation - ) + launchTokenHash: dispatchAuthority?.launchTokenHash ?? undefined, + processIncarnation, + // Why system: the automatic loop is host-local Orca code driven by + // coordinator_runs, not a CLI caller, so it is a root by construction. + creator: { kind: 'system' }, + maxDepth: params.nestedWorkerMaxDepth + }) // Why: dispatched agents use orca-dev in dev mode to reach the dev runtime's socket, not production (Section 6.4). const preamble = buildDispatchPreamble({ diff --git a/src/main/runtime/orchestration/coordinator.test.ts b/src/main/runtime/orchestration/coordinator.test.ts index 7fcc3483be1..38701a9d7dd 100644 --- a/src/main/runtime/orchestration/coordinator.test.ts +++ b/src/main/runtime/orchestration/coordinator.test.ts @@ -7,6 +7,7 @@ import { DISPATCH_STALE_THRESHOLD, parseAllowStaleBaseFromSpec } from './coordinator-stale-base-flag' +import { createRootDispatch } from './db/root-dispatch-test-fixture' type DriftResult = { base: string @@ -223,7 +224,7 @@ describe('Coordinator', () => { const runtime = createMockRuntime() const task = db.createTask({ spec: 'send-driven completion' }) - const dispatch = db.createDispatchContext(task.id, 'term_a') + const dispatch = createRootDispatch(db, task.id, 'term_a') const msg = db.insertMessage({ from: 'term_a', to: 'coord', @@ -250,7 +251,7 @@ describe('Coordinator', () => { const runtime = createMockRuntime() const task = db.createTask({ spec: 'duplicate completion' }) - const dispatch = db.createDispatchContext(task.id, 'term_a') + const dispatch = createRootDispatch(db, task.id, 'term_a') const payload = JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, @@ -530,7 +531,7 @@ describe('Coordinator', () => { // No terminals available so dispatchReadyTasks creates one and we can // drive the stale-scan deterministically via SQL backdating. const task = db.createTask({ spec: 'work' }) - const ctx = db.createDispatchContext(task.id, 'term_stale') + const ctx = createRootDispatch(db, task.id, 'term_stale') // Backdate dispatched_at and last_heartbeat_at beyond the 10-min threshold // so getStaleDispatches returns this row on the first tick. @@ -569,7 +570,7 @@ describe('Coordinator', () => { runtime.terminals = [{ handle: 'term_a', worktreeId: 'wt1', connected: true, writable: true }] const task = db.createTask({ spec: 'work' }) - const ctx = db.createDispatchContext(task.id, 'term_a') + const ctx = createRootDispatch(db, task.id, 'term_a') const coordinator = new Coordinator(db, runtime, { spec: 'go', @@ -606,9 +607,9 @@ describe('Coordinator', () => { const logs: string[] = [] const task = db.createTask({ spec: 'retry-sensitive work' }) - const staleCtx = db.createDispatchContext(task.id, 'term_old') + const staleCtx = createRootDispatch(db, task.id, 'term_old') db.failDispatch(staleCtx.id, 'retry elsewhere') - const activeCtx = db.createDispatchContext(task.id, 'term_current') + const activeCtx = createRootDispatch(db, task.id, 'term_current') db.insertMessage({ from: 'term_old', @@ -664,7 +665,7 @@ describe('Coordinator', () => { const task = db.createTask({ spec: 'owned work' }) const leafId = '11111111-1111-4111-8111-111111111111' - const ctx = db.createDispatchContext(task.id, 'term_owner', `tab_before:${leafId}`) + const ctx = createRootDispatch(db, task.id, 'term_owner', `tab_before:${leafId}`) db.insertMessage({ from: 'term_reminted', diff --git a/src/main/runtime/orchestration/coordinator.ts b/src/main/runtime/orchestration/coordinator.ts index 85250770dee..252c9f89ea9 100644 --- a/src/main/runtime/orchestration/coordinator.ts +++ b/src/main/runtime/orchestration/coordinator.ts @@ -13,6 +13,7 @@ import { listAvailableWorkerTerminals, warnStaleDispatches } from './coordinator-task-dispatch' +import { NESTED_WORKER_MAX_DEPTH_DEFAULT } from '../../../shared/nested-worker-depth' export type CoordinatorOptions = { spec: string @@ -283,6 +284,8 @@ export class Coordinator { baseDrift, coordinatorHandle: this.opts.coordinatorHandle, worktree: this.opts.worktree, + nestedWorkerMaxDepth: + this.runtime.getNestedWorkerMaxDepth?.() ?? NESTED_WORKER_MAX_DEPTH_DEFAULT, onLog: this.opts.onLog, onCircuitBroken: (taskId) => this.state.failedTasks.push(taskId) }) diff --git a/src/main/runtime/orchestration/db-empty-dispatch-shortcircuit.benchmark.test.ts b/src/main/runtime/orchestration/db-empty-dispatch-shortcircuit.benchmark.test.ts index 991d9df4358..c729a8b1dd7 100644 --- a/src/main/runtime/orchestration/db-empty-dispatch-shortcircuit.benchmark.test.ts +++ b/src/main/runtime/orchestration/db-empty-dispatch-shortcircuit.benchmark.test.ts @@ -1,5 +1,6 @@ import { describe, expect, it } from 'vitest' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' // Why: buildAgentOrchestrationByPaneKey issues 2 dispatch lookups per terminal // on EVERY 16ms graph publish. For users who never orchestrate, every one of @@ -64,7 +65,7 @@ describe('orchestration empty-dispatch short-circuit (benchmark)', () => { it('still runs the fan-out once a dispatch exists (correctness preserved)', () => { const db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - db.createDispatchContext(task.id, 'term_5') + createRootDispatch(db, task.id, 'term_5') const handles = Array.from({ length: 10 }, (_, i) => `term_${i}`) const contexts = simulateGraphPublish(db, handles) @@ -75,7 +76,7 @@ describe('orchestration empty-dispatch short-circuit (benchmark)', () => { it('predicate lifecycle: false when empty, true after dispatch (even completed), false after reset', () => { const db = new OrchestrationDb(':memory:') expect(db.hasAnyDispatchContexts()).toBe(false) - const ctx = db.createDispatchContext(db.createTask({ spec: 'work' }).id, 'term_worker') + const ctx = createRootDispatch(db, db.createTask({ spec: 'work' }).id, 'term_worker') expect(db.hasAnyDispatchContexts()).toBe(true) // Completed rows still count — recent-completed lookups must stay valid. db.completeDispatch(ctx.id) diff --git a/src/main/runtime/orchestration/db-heartbeat-straggler-guard.test.ts b/src/main/runtime/orchestration/db-heartbeat-straggler-guard.test.ts index 7c2fd3e44db..793c218e162 100644 --- a/src/main/runtime/orchestration/db-heartbeat-straggler-guard.test.ts +++ b/src/main/runtime/orchestration/db-heartbeat-straggler-guard.test.ts @@ -1,5 +1,6 @@ import { afterEach, describe, expect, it } from 'vitest' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' let db: OrchestrationDb | undefined @@ -13,7 +14,7 @@ function seedHeartbeatedDispatch(): { d: OrchestrationDb; dispatchId: string } { const d = new OrchestrationDb(':memory:') db = d const task = d.createTask({ spec: 'work' }) - const dispatch = d.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(d, task.id, 'term_worker') d.recordHeartbeat(dispatch.id, '2026-05-03T00:00:00.000Z') return { d, dispatchId: dispatch.id } } diff --git a/src/main/runtime/orchestration/db-task-dispatch-invariant.test.ts b/src/main/runtime/orchestration/db-task-dispatch-invariant.test.ts index a4ebe6b66a8..80b4f8cf53e 100644 --- a/src/main/runtime/orchestration/db-task-dispatch-invariant.test.ts +++ b/src/main/runtime/orchestration/db-task-dispatch-invariant.test.ts @@ -4,6 +4,7 @@ import { join } from 'node:path' import { afterEach, describe, expect, it, vi } from 'vitest' import type Database from '../../sqlite/sync-database' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' type DatabaseHarness = { db: OrchestrationDb @@ -31,7 +32,7 @@ describe('Task/Dispatch invariant transactions', () => { const { db } = createDatabase() const task = db.createTask({ spec: 'atomic work' }) const dependent = db.createTask({ spec: 'dependent work', deps: [task.id] }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const capability = db.mintDispatchCapability({ dispatchId: dispatch.id, paneKey: 'tab_worker:leaf_worker', @@ -74,7 +75,7 @@ describe('Task/Dispatch invariant transactions', () => { it('does not commit a caller-owned transaction', () => { const { db } = createDatabase() const task = db.createTask({ spec: 'outer transaction work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const sqlite = sqliteFor(db) sqlite.exec('BEGIN IMMEDIATE') @@ -101,7 +102,7 @@ describe('Task/Dispatch invariant transactions', () => { const sqlite = sqliteFor(db) sqlite.exec('BEGIN IMMEDIATE') - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') expect(db.getTask(task.id)?.status).toBe('dispatched') expect(db.getDispatchContextById(dispatch.id)?.status).toBe('dispatched') sqlite.exec('ROLLBACK') @@ -115,9 +116,9 @@ describe('Task/Dispatch invariant transactions', () => { (status) => { const { db } = createDatabase() const task = db.createTask({ spec: 'legacy split work' }) - const first = db.createDispatchContext(task.id, 'term_first') + const first = createRootDispatch(db, task.id, 'term_first') sqliteFor(db).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const second = db.createDispatchContext(task.id, 'term_second') + const second = createRootDispatch(db, task.id, 'term_second') db.updateTaskStatus(task.id, status, 'terminal result') @@ -131,10 +132,10 @@ describe('Task/Dispatch invariant transactions', () => { expect(db.getActiveDispatchForTerminal('term_first')).toBeUndefined() expect(db.getActiveDispatchForTerminal('term_second')).toBeUndefined() expect(() => - db.createDispatchContext(db.createTask({ spec: 'first later work' }).id, 'term_first') + createRootDispatch(db, db.createTask({ spec: 'first later work' }).id, 'term_first') ).not.toThrow() expect(() => - db.createDispatchContext(db.createTask({ spec: 'second later work' }).id, 'term_second') + createRootDispatch(db, db.createTask({ spec: 'second later work' }).id, 'term_second') ).not.toThrow() } ) @@ -142,9 +143,9 @@ describe('Task/Dispatch invariant transactions', () => { it('does not requeue a legacy split Task while another Dispatch remains active', () => { const { db } = createDatabase() const task = db.createTask({ spec: 'legacy split retry' }) - const first = db.createDispatchContext(task.id, 'term_first') + const first = createRootDispatch(db, task.id, 'term_first') sqliteFor(db).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const second = db.createDispatchContext(task.id, 'term_second') + const second = createRootDispatch(db, task.id, 'term_second') db.failDispatch(second.id, 'targeted failure') @@ -156,9 +157,9 @@ describe('Task/Dispatch invariant transactions', () => { it('does not block a legacy split Task while another Dispatch remains active', () => { const { db } = createDatabase() const task = db.createTask({ spec: 'legacy split release' }) - const first = db.createDispatchContext(task.id, 'term_first') + const first = createRootDispatch(db, task.id, 'term_first') sqliteFor(db).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const second = db.createDispatchContext(task.id, 'term_second') + const second = createRootDispatch(db, task.id, 'term_second') expect(db.beginWorkerStop(second.id, 'runtime_test')).toMatchObject({ disposition: 'context_only', @@ -174,7 +175,7 @@ describe('Task/Dispatch invariant transactions', () => { (status) => { const { db } = createDatabase() const task = db.createTask({ spec: 'guarded work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') expect(() => db.updateTaskStatus(task.id, status, 'must not persist')).toThrowError( expect.objectContaining({ @@ -215,7 +216,7 @@ describe('Task/Dispatch invariant transactions', () => { return prepare(sql) }) - expect(() => first.db.createDispatchContext(task.id, 'term_worker')).toThrow( + expect(() => createRootDispatch(first.db, task.id, 'term_worker')).toThrow( `Task ${task.id} is failed; only ready tasks can be dispatched` ) expect(injected).toBe(true) @@ -233,7 +234,8 @@ describe('Task/Dispatch invariant transactions', () => { let winnerId: string | undefined vi.spyOn(sqlite, 'prepare').mockImplementation((sql) => { if (!winnerId && sql.includes('INSERT INTO dispatch_contexts')) { - winnerId = concurrent.db.createDispatchContext( + winnerId = createRootDispatch( + concurrent.db, secondTask.id, 'term_reminted', 'tab_new:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb' @@ -243,7 +245,8 @@ describe('Task/Dispatch invariant transactions', () => { }) expect(() => - first.db.createDispatchContext( + createRootDispatch( + first.db, firstTask.id, 'term_worker', 'tab_old:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb' @@ -264,13 +267,16 @@ describe('Task/Dispatch invariant transactions', () => { it('rejects worker authority when another Dispatch owns the pane', () => { const { db } = createDatabase() const ownerTask = db.createTask({ spec: 'current pane owner' }) - const owner = db.createDispatchContext( + const owner = createRootDispatch( + db, ownerTask.id, 'term_owner', 'tab_old:cccccccc-cccc-4ccc-8ccc-cccccccccccc' ) const workerTask = db.createTask({ spec: 'competing supervised worker' }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: workerTask.id, startOptions: {} }) @@ -304,7 +310,12 @@ describe('Task/Dispatch invariant transactions', () => { (status) => { const { db } = createDatabase() const task = db.createTask({ spec: 'supervised lifecycle' }) - const started = db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) const capability = db.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: 'term_worker', @@ -348,6 +359,8 @@ describe('Task/Dispatch invariant transactions', () => { const { db } = createDatabase() const task = db.createTask({ spec: 'federated lifecycle' }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, federation: { diff --git a/src/main/runtime/orchestration/db-task-dispatch-lifecycle-guards.test.ts b/src/main/runtime/orchestration/db-task-dispatch-lifecycle-guards.test.ts index 2a2e0b8d692..e1e9da0ee41 100644 --- a/src/main/runtime/orchestration/db-task-dispatch-lifecycle-guards.test.ts +++ b/src/main/runtime/orchestration/db-task-dispatch-lifecycle-guards.test.ts @@ -4,6 +4,7 @@ import { join } from 'node:path' import { afterEach, describe, expect, it } from 'vitest' import type Database from '../../sqlite/sync-database' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' type WorkerFixture = { dispatchId: string @@ -55,7 +56,7 @@ describe('Task/Dispatch lifecycle guards', () => { (outcome) => { const database = createDatabase() const task = database.createTask({ spec: 'legacy mixed split' }) - const contextOnly = database.createDispatchContext(task.id, 'term_context') + const contextOnly = createRootDispatch(database, task.id, 'term_context') sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) const worker = startWorker(database, task.id, 'reporter') @@ -75,7 +76,8 @@ describe('Task/Dispatch lifecycle guards', () => { }) expect(database.getActiveDispatchForTerminal('term_context')).toBeUndefined() expect(() => - database.createDispatchContext( + createRootDispatch( + database, database.createTask({ spec: 'later context work' }).id, 'term_context' ) @@ -88,7 +90,7 @@ describe('Task/Dispatch lifecycle guards', () => { const task = database.createTask({ spec: 'reversed legacy mixed split' }) const worker = startWorker(database, task.id, 'reversed_reporter') sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const contextOnly = database.createDispatchContext(task.id, 'term_reversed_context') + const contextOnly = createRootDispatch(database, task.id, 'term_reversed_context') expect( database.settleWorkerReport({ @@ -175,6 +177,8 @@ describe('Task/Dispatch lifecycle guards', () => { const database = createDatabase() const task = database.createTask({ spec: `${kind} split start failure` }) const failed = database.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, ...(kind === 'federated' @@ -219,9 +223,14 @@ describe('Task/Dispatch lifecycle guards', () => { (operation) => { const database = createDatabase() const task = database.createTask({ spec: `${operation} historical sibling` }) - const contextOnly = database.createDispatchContext(task.id, `term_${operation}`) + const contextOnly = createRootDispatch(database, task.id, `term_${operation}`) sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const failed = database.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const failed = database.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) database.failWorkerStart(failed.dispatch.id, 'start_failed', 'worker failed to start') expect(database.getTask(task.id)?.status).toBe('dispatched') @@ -239,7 +248,8 @@ describe('Task/Dispatch lifecycle guards', () => { expect(database.getDispatchContextById(contextOnly.id)?.status).toBe('failed') expect(database.getActiveDispatchForTerminal(`term_${operation}`)).toBeUndefined() expect(() => - database.createDispatchContext( + createRootDispatch( + database, database.createTask({ spec: `${operation} later work` }).id, `term_${operation}` ) @@ -305,7 +315,12 @@ describe('Task/Dispatch lifecycle guards', () => { const task = database.createTask({ spec: 'uncertain legacy worker split' }) const live = startWorker(database, task.id, 'uncertain_live') sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const uncertain = database.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const uncertain = database.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) database.markWorkerStartUnknown(uncertain.dispatch.id, 'agent_readiness', 'outcome unknown') expect(database.getTask(task.id)?.status).toBe('blocked') @@ -331,7 +346,12 @@ describe('Task/Dispatch lifecycle guards', () => { const task = database.createTask({ spec: `${recovery} uncertain sibling` }) const live = startWorker(database, task.id, `${recovery}_live`) sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const uncertain = database.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const uncertain = database.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) database.markWorkerStartUnknown(uncertain.dispatch.id, 'agent_readiness', 'outcome unknown') if (recovery === 'federated-reconcile') { @@ -381,7 +401,7 @@ describe('Task/Dispatch lifecycle guards', () => { const task = database.createTask({ spec: 'corrupt gated task' }) const gate = database.createGate({ taskId: task.id, question: 'Proceed?' }) sqliteFor(database).prepare("UPDATE tasks SET status = 'ready' WHERE id = ?").run(task.id) - const dispatch = database.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(database, task.id, 'term_worker') sqliteFor(database).prepare("UPDATE tasks SET status = 'blocked' WHERE id = ?").run(task.id) expect(() => database.resolveGate(gate.id, 'yes')).toThrowError( @@ -404,7 +424,12 @@ function createDatabase(): OrchestrationDb { } function startWorker(database: OrchestrationDb, taskId: string, name: string): WorkerFixture { - const started = database.createStartingWorkerDispatch({ taskId, startOptions: {} }) + const started = database.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId, + startOptions: {} + }) const paneSuffix = name.length.toString(16).padStart(12, '0') const paneKey = `tab_${name}:aaaaaaaa-aaaa-4aaa-8aaa-${paneSuffix}` const processIncarnation = `${name}:1` diff --git a/src/main/runtime/orchestration/db-task-dispatch-races.test.ts b/src/main/runtime/orchestration/db-task-dispatch-races.test.ts index c9c765f9757..fabb859acb6 100644 --- a/src/main/runtime/orchestration/db-task-dispatch-races.test.ts +++ b/src/main/runtime/orchestration/db-task-dispatch-races.test.ts @@ -4,6 +4,7 @@ import { join } from 'node:path' import { afterEach, describe, expect, it, vi } from 'vitest' import type Database from '../../sqlite/sync-database' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' type DatabaseHarness = { db: OrchestrationDb @@ -28,7 +29,7 @@ describe('Task/Dispatch concurrency', () => { it('rolls back Dispatch failure when Task requeue fails', () => { const { db } = createDatabase() const task = db.createTask({ spec: 'atomic retry failure' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') sqliteFor(db).exec(` CREATE TRIGGER reject_task_requeue BEFORE UPDATE OF status ON tasks @@ -55,7 +56,12 @@ describe('Task/Dispatch concurrency', () => { const first = createDatabase() const concurrent = createDatabase(first.path) const task = first.db.createTask({ spec: 'worker completion wins' }) - const started = first.db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = first.db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) const capability = first.db.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: 'term_worker', @@ -117,10 +123,14 @@ describe('Task/Dispatch concurrency', () => { const losingTask = first.db.createTask({ spec: 'losing worker' }) const winningTask = first.db.createTask({ spec: 'winning worker' }) const loser = first.db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: losingTask.id, startOptions: {} }) const winner = concurrent.db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: winningTask.id, startOptions: {} }) diff --git a/src/main/runtime/orchestration/db.test.ts b/src/main/runtime/orchestration/db.test.ts index 0894ca7c6a0..4825246586a 100644 --- a/src/main/runtime/orchestration/db.test.ts +++ b/src/main/runtime/orchestration/db.test.ts @@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest' import Database from '../../sqlite/sync-database' import { LEGACY_RUN_ID, OrchestrationDb } from './db' import type { MessageType } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' // Overwrites the datetime('now')-seeded timestamps with explicit fixture values // so stale-detection assertions stay deterministic (no wall clock). @@ -255,7 +256,7 @@ describe('OrchestrationDb', () => { it('completing a task frees its active dispatch context', () => { const d = createDb() const task = d.createTask({ spec: 'do it' }) - d.createDispatchContext(task.id, 'term_a') + createRootDispatch(d, task.id, 'term_a') d.updateTaskStatus(task.id, 'completed') @@ -285,7 +286,7 @@ describe('OrchestrationDb', () => { const d = createDb() const ready = d.createTask({ spec: 'ready task' }) const dispatched = d.createTask({ spec: 'active task' }) - const ctx = d.createDispatchContext(dispatched.id, 'term_worker') + const ctx = createRootDispatch(d, dispatched.id, 'term_worker') const rows = d.listTasksWithDispatch() const readyRow = rows.find((r) => r.id === ready.id) @@ -300,7 +301,7 @@ describe('OrchestrationDb', () => { it('listTasksWithDispatch does not surface completed dispatches', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - d.createDispatchContext(task.id, 'term_worker') + createRootDispatch(d, task.id, 'term_worker') d.updateTaskStatus(task.id, 'completed') const rows = d.listTasksWithDispatch() @@ -323,7 +324,7 @@ describe('OrchestrationDb', () => { it('creates a dispatch context and marks task as dispatched', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - const ctx = d.createDispatchContext(task.id, 'term_worker') + const ctx = createRootDispatch(d, task.id, 'term_worker') expect(ctx.id).toMatch(/^ctx_/) expect(ctx.task_id).toBe(task.id) @@ -337,7 +338,7 @@ describe('OrchestrationDb', () => { const parent = d.createTask({ spec: 'parent' }) const child = d.createTask({ spec: 'child', deps: [parent.id] }) - expect(() => d.createDispatchContext(child.id, 'term_worker')).toThrow( + expect(() => createRootDispatch(d, child.id, 'term_worker')).toThrow( /only ready tasks can be dispatched/ ) }) @@ -346,9 +347,9 @@ describe('OrchestrationDb', () => { const d = createDb() const t1 = d.createTask({ spec: 'first' }) const t2 = d.createTask({ spec: 'second' }) - d.createDispatchContext(t1.id, 'term_worker') + createRootDispatch(d, t1.id, 'term_worker') - expect(() => d.createDispatchContext(t2.id, 'term_worker')).toThrow( + expect(() => createRootDispatch(d, t2.id, 'term_worker')).toThrow( /already has an active dispatch/ ) }) @@ -362,9 +363,9 @@ describe('OrchestrationDb', () => { const d = createDb() const t1 = d.createTask({ spec: 'first' }) const t2 = d.createTask({ spec: 'second' }) - d.createDispatchContext(t1.id, 'term_old', `tab_1:${LEAF_A}`) + createRootDispatch(d, t1.id, 'term_old', `tab_1:${LEAF_A}`) - expect(() => d.createDispatchContext(t2.id, 'term_new', `tab_1:${LEAF_A}`)).toThrow( + expect(() => createRootDispatch(d, t2.id, 'term_new', `tab_1:${LEAF_A}`)).toThrow( /already has an active dispatch/ ) }) @@ -373,9 +374,9 @@ describe('OrchestrationDb', () => { const d = createDb() const t1 = d.createTask({ spec: 'first' }) const t2 = d.createTask({ spec: 'second' }) - d.createDispatchContext(t1.id, 'term_old', `tab_1:${LEAF_A}`) + createRootDispatch(d, t1.id, 'term_old', `tab_1:${LEAF_A}`) - expect(() => d.createDispatchContext(t2.id, 'term_new', `tab_2:${LEAF_A}`)).toThrow( + expect(() => createRootDispatch(d, t2.id, 'term_new', `tab_2:${LEAF_A}`)).toThrow( /already has an active dispatch/ ) }) @@ -384,37 +385,37 @@ describe('OrchestrationDb', () => { const d = createDb() const t1 = d.createTask({ spec: 'first' }) const t2 = d.createTask({ spec: 'second' }) - d.createDispatchContext(t1.id, 'term_a', `tab_1:${LEAF_A}`) + createRootDispatch(d, t1.id, 'term_a', `tab_1:${LEAF_A}`) - expect(() => d.createDispatchContext(t2.id, 'term_b', `tab_1:${LEAF_B}`)).not.toThrow() + expect(() => createRootDispatch(d, t2.id, 'term_b', `tab_1:${LEAF_B}`)).not.toThrow() }) it('falls back to handle lock when pane keys are missing', () => { const d = createDb() const t1 = d.createTask({ spec: 'first' }) const t2 = d.createTask({ spec: 'second' }) - d.createDispatchContext(t1.id, 'term_worker') + createRootDispatch(d, t1.id, 'term_worker') // New dispatch has a pane key but the active row is legacy (no pane key): // only handle identity can lock; a different handle is free. - expect(() => d.createDispatchContext(t2.id, 'term_other', `tab_1:${LEAF_A}`)).not.toThrow() + expect(() => createRootDispatch(d, t2.id, 'term_other', `tab_1:${LEAF_A}`)).not.toThrow() }) it('allows dispatch to a terminal after previous dispatch completes', () => { const d = createDb() const t1 = d.createTask({ spec: 'first' }) const t2 = d.createTask({ spec: 'second' }) - const ctx1 = d.createDispatchContext(t1.id, 'term_worker') + const ctx1 = createRootDispatch(d, t1.id, 'term_worker') d.completeDispatch(ctx1.id) - expect(() => d.createDispatchContext(t2.id, 'term_worker')).not.toThrow() + expect(() => createRootDispatch(d, t2.id, 'term_worker')).not.toThrow() }) it('getDispatchContext returns latest for a task', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - const ctx = d.createDispatchContext(task.id, 'term_a') + const ctx = createRootDispatch(d, task.id, 'term_a') const found = d.getDispatchContext(task.id) expect(found?.id).toBe(ctx.id) }) @@ -422,9 +423,9 @@ describe('OrchestrationDb', () => { it('getDispatchContext uses insertion order when timestamps tie', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - const ctx1 = d.createDispatchContext(task.id, 'term_a') + const ctx1 = createRootDispatch(d, task.id, 'term_a') d.failDispatch(ctx1.id, 'retry') - const ctx2 = d.createDispatchContext(task.id, 'term_a') + const ctx2 = createRootDispatch(d, task.id, 'term_a') expect(d.getDispatchContext(task.id)?.id).toBe(ctx2.id) }) @@ -432,7 +433,7 @@ describe('OrchestrationDb', () => { it('getActiveDispatchForTerminal returns active dispatch', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - d.createDispatchContext(task.id, 'term_a') + createRootDispatch(d, task.id, 'term_a') const active = d.getActiveDispatchForTerminal('term_a') expect(active?.task_id).toBe(task.id) @@ -442,10 +443,10 @@ describe('OrchestrationDb', () => { it('getLatestDispatchForTerminal returns the most recent completed dispatch', () => { const d = createDb() const firstTask = d.createTask({ spec: 'first' }) - const first = d.createDispatchContext(firstTask.id, 'term_a') + const first = createRootDispatch(d, firstTask.id, 'term_a') d.completeDispatch(first.id) const secondTask = d.createTask({ spec: 'second' }) - const second = d.createDispatchContext(secondTask.id, 'term_a') + const second = createRootDispatch(d, secondTask.id, 'term_a') d.completeDispatch(second.id) const latest = d.getLatestDispatchForTerminal('term_a') @@ -457,19 +458,19 @@ describe('OrchestrationDb', () => { it('circuit breaker trips after 3 failures', () => { const d = createDb() const task = d.createTask({ spec: 'flaky' }) - const ctx = d.createDispatchContext(task.id, 'term_a') + const ctx = createRootDispatch(d, task.id, 'term_a') const after1 = d.failDispatch(ctx.id, 'timeout') expect(after1?.failure_count).toBe(1) expect(after1?.status).toBe('failed') expect(d.getTask(task.id)?.status).toBe('ready') - const ctx2 = d.createDispatchContext(task.id, 'term_a') + const ctx2 = createRootDispatch(d, task.id, 'term_a') const after2 = d.failDispatch(ctx2.id, 'timeout') expect(after2?.failure_count).toBe(2) expect(after2?.status).toBe('failed') - const ctx3 = d.createDispatchContext(task.id, 'term_a') + const ctx3 = createRootDispatch(d, task.id, 'term_a') const after3 = d.failDispatch(ctx3.id, 'timeout') expect(after3?.failure_count).toBe(3) expect(after3?.status).toBe('circuit_broken') @@ -480,7 +481,7 @@ describe('OrchestrationDb', () => { it('completeDispatch sets completed_at', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - const ctx = d.createDispatchContext(task.id, 'term_a') + const ctx = createRootDispatch(d, task.id, 'term_a') d.completeDispatch(ctx.id) const updated = d.getDispatchContext(task.id) @@ -493,7 +494,7 @@ describe('OrchestrationDb', () => { it('creates a gate and blocks the task', () => { const d = createDb() const task = d.createTask({ spec: 'needs approval' }) - d.createDispatchContext(task.id, 'term_a') + createRootDispatch(d, task.id, 'term_a') const gate = d.createGate({ taskId: task.id, question: 'Proceed?', @@ -622,7 +623,7 @@ describe('OrchestrationDb', () => { const d = createDb() d.insertMessage({ from: 'a', to: 'b', subject: 'test' }) const task = d.createTask({ spec: 'work' }) - d.createDispatchContext(task.id, 'term_a') + createRootDispatch(d, task.id, 'term_a') d.resetTasks() @@ -647,7 +648,7 @@ describe('OrchestrationDb', () => { it('recordHeartbeat updates last_heartbeat_at on dispatched rows', () => { const d = createDb() const task = d.createTask({ spec: 'work' }) - const ctx = d.createDispatchContext(task.id, 'term_a') + const ctx = createRootDispatch(d, task.id, 'term_a') d.recordHeartbeat(ctx.id, '2026-05-04T00:00:00.000Z') const after = d.getDispatchContext(task.id) @@ -665,10 +666,10 @@ describe('OrchestrationDb', () => { const taskB = d.createTask({ spec: 'b' }) const taskC = d.createTask({ spec: 'c' }) const taskD = d.createTask({ spec: 'd' }) - const ctxA = d.createDispatchContext(taskA.id, 'term_a') - const ctxB = d.createDispatchContext(taskB.id, 'term_b') - const ctxC = d.createDispatchContext(taskC.id, 'term_c') - const ctxD = d.createDispatchContext(taskD.id, 'term_d') + const ctxA = createRootDispatch(d, taskA.id, 'term_a') + const ctxB = createRootDispatch(d, taskB.id, 'term_b') + const ctxC = createRootDispatch(d, taskC.id, 'term_c') + const ctxD = createRootDispatch(d, taskD.id, 'term_d') d.completeDispatch(ctxD.id) const now = Date.now() @@ -709,15 +710,15 @@ describe('OrchestrationDb', () => { // Fresh worker: dispatched 12:00, heartbeat 12:05 (space-format), both // after the 11:55 threshold → NOT stale. - const fresh = d.createDispatchContext(d.createTask({ spec: 'fresh' }).id, 'term_fresh') + const fresh = createRootDispatch(d, d.createTask({ spec: 'fresh' }).id, 'term_fresh') setDispatchTimes(d, fresh.id, '2026-07-12 12:00:00', '2026-07-12 12:05:00') // Legacy ISO-format fresh row (mixed-format table) stays fresh too. - const legacy = d.createDispatchContext(d.createTask({ spec: 'legacy' }).id, 'term_legacy') + const legacy = createRootDispatch(d, d.createTask({ spec: 'legacy' }).id, 'term_legacy') setDispatchTimes(d, legacy.id, '2026-07-12T12:00:00.000Z', '2026-07-12T12:05:00.000Z') // Genuinely hung: dispatched + heartbeated at 10:00, ~2h before threshold. - const hung = d.createDispatchContext(d.createTask({ spec: 'hung' }).id, 'term_hung') + const hung = createRootDispatch(d, d.createTask({ spec: 'hung' }).id, 'term_hung') setDispatchTimes(d, hung.id, '2026-07-12 10:00:00', '2026-07-12 10:00:00') const stale = d.getStaleDispatches('2026-07-12T11:55:00.000Z') @@ -729,7 +730,7 @@ describe('OrchestrationDb', () => { // Space-format dispatched_at one minute after the threshold, no heartbeat // yet → still inside the grace window, must not be flagged. - const ctx = d.createDispatchContext(d.createTask({ spec: 'x' }).id, 'term_x') + const ctx = createRootDispatch(d, d.createTask({ spec: 'x' }).id, 'term_x') setDispatchTimes(d, ctx.id, '2026-07-12 12:00:00') const stale = d.getStaleDispatches('2026-07-12T11:59:00.000Z') @@ -741,7 +742,7 @@ describe('OrchestrationDb', () => { it('getStaleDispatches keeps a fresh row just after a UTC-midnight threshold (#8452)', () => { const d = createDb() - const ctx = d.createDispatchContext(d.createTask({ spec: 'midnight' }).id, 'term_midnight') + const ctx = createRootDispatch(d, d.createTask({ spec: 'midnight' }).id, 'term_midnight') setDispatchTimes(d, ctx.id, '2026-05-04 00:04:00') const stale = d.getStaleDispatches('2026-05-04T00:00:00.000Z') @@ -754,7 +755,7 @@ describe('OrchestrationDb', () => { it('getStaleDispatches keeps a live worker with a fresh space-format heartbeat (#8452)', () => { const d = createDb() - const ctx = d.createDispatchContext(d.createTask({ spec: 'live' }).id, 'term_live') + const ctx = createRootDispatch(d, d.createTask({ spec: 'live' }).id, 'term_live') setDispatchTimes(d, ctx.id, '2026-07-12 10:00:00', '2026-07-12 11:59:00') const stale = d.getStaleDispatches('2026-07-12T11:55:00.000Z') @@ -911,7 +912,7 @@ describe('OrchestrationDb', () => { // (b) last_heartbeat_at column exists on dispatch_contexts const task = d.createTask({ spec: 'work' }) - const ctx = d.createDispatchContext(task.id, 'term_a') + const ctx = createRootDispatch(d, task.id, 'term_a') d.recordHeartbeat(ctx.id, '2026-05-04T00:00:00.000Z') expect(d.getDispatchContext(task.id)?.last_heartbeat_at).toBe('2026-05-04T00:00:00.000Z') expect(d.getTask(task.id)?.task_title).toBe('work') @@ -942,7 +943,7 @@ describe('OrchestrationDb', () => { db = d const task = d.createTask({ spec: 'work' }) - const ctx = d.createDispatchContext(task.id, 'term_a', 'tab_1:leaf_1') + const ctx = createRootDispatch(d, task.id, 'term_a', 'tab_1:leaf_1') expect(d.getDispatchContextById(ctx.id)?.assignee_pane_key).toBe('tab_1:leaf_1') const msg = d.insertMessage({ diff --git a/src/main/runtime/orchestration/db.ts b/src/main/runtime/orchestration/db.ts index c6cab061700..7b650f970ba 100644 --- a/src/main/runtime/orchestration/db.ts +++ b/src/main/runtime/orchestration/db.ts @@ -6,7 +6,7 @@ export { } from './db/contract-constants' export type { RunListPage, TaskRuntimeLineageRow } from './db/run-list-page' export { ORCHESTRATION_DELIVERY_BATCH_LIMIT } from './db/messages/mailbox-routing-page' -export { DISPATCH_CONTEXT_CLAIM_SQL } from './db/dispatch-context/dispatch-context-store' +export { DISPATCH_CONTEXT_CLAIM_SQL } from './db/dispatch-row-writer' export type { ForeignDirectMailboxRoutingPage, MailboxRoutingPage diff --git a/src/main/runtime/orchestration/db/attach-orchestration-db-methods.ts b/src/main/runtime/orchestration/db/attach-orchestration-db-methods.ts index 087ec7e1e53..69f12d9bbd6 100644 --- a/src/main/runtime/orchestration/db/attach-orchestration-db-methods.ts +++ b/src/main/runtime/orchestration/db/attach-orchestration-db-methods.ts @@ -4,6 +4,7 @@ import { attachDispatchCapability } from './dispatch-context/dispatch-capability import { attachDispatchCompletion } from './dispatch-context/dispatch-completion' import { attachDispatchContextStore } from './dispatch-context/dispatch-context-store' import { attachDispatchLookup } from './dispatch-context/dispatch-lookup' +import { attachDispatchDepth } from './dispatch-depth' import { attachWorkerReportSettlement } from './dispatch-context/worker-report-settlement' import { attachFederatedDispatchStore } from './federation/federated-dispatch-store' import { attachFederationRelayAck } from './federation/federation-relay-ack' @@ -115,6 +116,7 @@ export function attachOrchestrationDbMethods(ctor: { prototype: object }): void attachDispatchContextStore(ctor) attachDispatchCapability(ctor) attachDispatchLookup(ctor) + attachDispatchDepth(ctor) attachDispatchCompletion(ctor) attachWorkerReportSettlement(ctor) attachDecisionGateStore(ctor) diff --git a/src/main/runtime/orchestration/db/contract-constants.ts b/src/main/runtime/orchestration/db/contract-constants.ts index 6d175498bc4..56390138f0a 100644 --- a/src/main/runtime/orchestration/db/contract-constants.ts +++ b/src/main/runtime/orchestration/db/contract-constants.ts @@ -7,4 +7,4 @@ export const LEGACY_CONTRACT_VERSION = 0 export const CURRENT_CONTRACT_VERSION = ORCHESTRATION_CONTRACT_VERSION // Schema versions: v2 'heartbeat'+last_heartbeat_at, v3 delivered_at, v4 task-creator terminal, v5 task_title/display_name, v6 pane identity, v7 lightweight Runs, v8 crash-safe Run deliveries, v9 durable question threads, v10 Dispatch capabilities, v11 durable mutation receipts, v12 composed worker state, v18 post-v6 version-skew repair, v19 adopted legacy Runs and compatibility receipts, v20 legacy question backfill, v21 legacy scheduler-loss provenance, v22 dispatch assignee lookup, v23 worker terminal resource ownership, v24 creator-incarnation authority, v25 active Dispatch handle lookup, v26 indexed mutation receipt capacity, v27 durable federation acknowledgments, v28 durable local mutation caller identity. -export const SCHEMA_VERSION = 29 +export const SCHEMA_VERSION = 30 diff --git a/src/main/runtime/orchestration/db/dispatch-context/dispatch-context-store.ts b/src/main/runtime/orchestration/db/dispatch-context/dispatch-context-store.ts index c8415a0214c..cd4083acc0a 100644 --- a/src/main/runtime/orchestration/db/dispatch-context/dispatch-context-store.ts +++ b/src/main/runtime/orchestration/db/dispatch-context/dispatch-context-store.ts @@ -3,48 +3,27 @@ import { OrchestrationError } from '../../orchestration-error' import { parsePaneKey } from '../../../../../shared/stable-pane-id' import { CURRENT_CONTRACT_VERSION } from '../contract-constants' import { generateId } from '../generated-id' -import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL, paneKeyMatchSuffix } from '../pane-key-match' +import { paneKeyMatchSuffix } from '../pane-key-match' +import { claimDispatchContextRow } from '../dispatch-row-writer' +import type { DispatchCreator } from '../dispatch-depth' import type { OrchestrationDb } from '../orchestration-db' -export const DISPATCH_CONTEXT_CLAIM_SQL = `INSERT INTO dispatch_contexts ( - id, run_id, task_id, contract_version, launch_token_hash, - assignee_handle, assignee_pane_key, process_incarnation, - status, failure_count, dispatched_at -) -SELECT ?, run_id, id, ?, ?, ?, ?, ?, 'dispatched', ?, datetime('now') -FROM tasks -WHERE id = ? AND status = 'ready' - AND NOT EXISTS ( - SELECT 1 FROM dispatch_contexts active - WHERE active.assignee_handle = ? - AND active.status IN ('pending', 'dispatched') - ) - AND ( - ? IS NULL OR NOT EXISTS ( - SELECT 1 FROM dispatch_contexts active - WHERE active.assignee_pane_key = ? - AND active.status IN ('pending', 'dispatched') - ) - ) - AND ( - ? IS NULL OR NOT EXISTS ( - SELECT 1 FROM dispatch_contexts active - WHERE active.assignee_pane_key IS NOT NULL - AND active.status IN ('pending', 'dispatched') - AND instr(active.assignee_pane_key, ':') > 1 - AND ${DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL} = ? - ) - )` - export function createDispatchContext( this: OrchestrationDb, - taskId: string, - assigneeHandle: string, - // Why: pane key is the remint-stable identity behind the handle — lets worker_done ownership survive handle reissue. - assigneePaneKey?: string, - launchTokenHash?: string, - processIncarnation?: string + params: { + taskId: string + assigneeHandle: string + // Why: pane key is the remint-stable identity behind the handle — lets worker_done ownership survive handle reissue. + assigneePaneKey?: string + launchTokenHash?: string + processIncarnation?: string + /** Who is dispatching, for nesting depth. Required so a new caller must decide. */ + creator: DispatchCreator + maxDepth: number + } ): DispatchContextRow { + const { taskId, assigneeHandle, assigneePaneKey, launchTokenHash, processIncarnation } = params + const depth = this.resolveChildDispatchDepth(params.creator, params.maxDepth) const task = this.getTask(taskId) if (!task) { throw new Error(`Task not found: ${taskId}`) @@ -73,23 +52,18 @@ export function createDispatchContext( const id = generateId('ctx') this.db.exec('SAVEPOINT create_dispatch_context') try { - const inserted = this.db - .prepare(DISPATCH_CONTEXT_CLAIM_SQL) - .run( - id, - CURRENT_CONTRACT_VERSION, - launchTokenHash ?? null, - assigneeHandle, - assigneePaneKey ?? null, - processIncarnation ?? null, - priorFailures, - taskId, - assigneeHandle, - assigneePaneKey ?? null, - assigneePaneKey ?? null, - paneSuffix, - paneSuffix - ) + const inserted = claimDispatchContextRow(this.db, { + id, + contractVersion: CURRENT_CONTRACT_VERSION, + launchTokenHash: launchTokenHash ?? null, + assigneeHandle, + assigneePaneKey: assigneePaneKey ?? null, + processIncarnation: processIncarnation ?? null, + priorFailures, + depth, + taskId, + paneSuffix + }) if (inserted.changes !== 1) { const current = this.getTask(taskId) const occupied = this.findActiveDispatchForAssignee(assigneeHandle, assigneePaneKey) diff --git a/src/main/runtime/orchestration/db/dispatch-depth.test.ts b/src/main/runtime/orchestration/db/dispatch-depth.test.ts new file mode 100644 index 00000000000..97df3f01c51 --- /dev/null +++ b/src/main/runtime/orchestration/db/dispatch-depth.test.ts @@ -0,0 +1,271 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { OrchestrationDb } from '../db' +import { AmbiguousDispatchParentError } from './dispatch-depth' + +/** + * These pin the fence Orca documented but never enforced: before this feature a + * dispatched worker could create its own Run and dispatch sub-workers freely. + * Every rejection case here passes on the pre-change tree. + */ +describe('nested worker depth', () => { + let db: OrchestrationDb + const SYSTEM = { kind: 'system' } as const + const UNCAPPED = Number.MAX_SAFE_INTEGER + + afterEach(() => db?.close()) + + function coordinatorDispatchesWorker(maxDepth = UNCAPPED) { + db = new OrchestrationDb(':memory:') + const task = db.createTask({ spec: 'root task' }) + const worker = db.createDispatchContext({ + taskId: task.id, + assigneeHandle: 'term_worker', + assigneePaneKey: 'tab_worker:leaf_worker', + creator: SYSTEM, + maxDepth + }) + return worker + } + + it('stamps a root coordinator dispatch at depth 1', () => { + expect(coordinatorDispatchesWorker().depth).toBe(1) + }) + + it('refuses a worker dispatching a sub-worker at the default cap', () => { + coordinatorDispatchesWorker() + const nested = db.createTask({ spec: 'nested task' }) + expect(() => + db.createDispatchContext({ + taskId: nested.id, + assigneeHandle: 'term_sub', + assigneePaneKey: 'tab_sub:leaf_sub', + creator: { + kind: 'terminal', + handle: 'term_worker', + paneKey: 'tab_worker:leaf_worker' + }, + maxDepth: 1 + }) + ).toThrow(/depth 2 \(max 1\)/) + }) + + it('tells the refused worker to complete the task itself', () => { + coordinatorDispatchesWorker() + const nested = db.createTask({ spec: 'nested task' }) + expect(() => + db.createDispatchContext({ + taskId: nested.id, + assigneeHandle: 'term_sub', + creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' }, + maxDepth: 1 + }) + ).toThrow(/Complete this task yourself/) + }) + + it('permits one more generation when the cap is raised, and records depth 2', () => { + coordinatorDispatchesWorker() + const nested = db.createTask({ spec: 'nested task' }) + const sub = db.createDispatchContext({ + taskId: nested.id, + assigneeHandle: 'term_sub', + assigneePaneKey: 'tab_sub:leaf_sub', + creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' }, + maxDepth: 2 + }) + expect(sub.depth).toBe(2) + }) + + it('closes the run-create bypass: a fresh Run does not reset the creator depth', () => { + // The old fence keyed off Run binding, so a worker that created its own Run + // walked straight through. Depth comes from the creator's dispatch instead. + coordinatorDispatchesWorker() + const ownRun = db.createRun({ + objective: 'worker-owned run', + coordinatorHandle: 'term_worker', + coordinatorPaneKey: 'tab_worker:leaf_worker' + }) + const nested = db.createTask({ spec: 'nested task', runId: ownRun.id }) + expect(() => + db.createDispatchContext({ + taskId: nested.id, + assigneeHandle: 'term_sub', + creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' }, + maxDepth: 1 + }) + ).toThrow(/depth 2 \(max 1\)/) + }) + + it('treats the in-process coordinator loop as a root even from a worker pane', () => { + coordinatorDispatchesWorker() + expect(db.resolveCreatorDepth({ kind: 'system' })).toBe(0) + }) + + it('resolves an unknown terminal to root depth', () => { + db = new OrchestrationDb(':memory:') + expect(db.resolveCreatorDepth({ kind: 'terminal', handle: 'term_nobody' })).toBe(0) + }) + + describe('remote attachments as parents', () => { + const PANE = 'tab_remote:leaf_remote' + const INCARNATION = 'inc-1' + + function attachRemoteWorker(state: string, depth: number, paneKey = PANE, inc = INCARNATION) { + db.db + .prepare( + `INSERT INTO remote_dispatch_attachments + (dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch, + pane_key, process_incarnation, state, depth) + VALUES (?, ?, 'peer', 1, 'epoch', ?, ?, ?, ?)` + ) + .run(`ctx_${state}_${depth}_${paneKey}_${inc}`, 'task_remote', paneKey, inc, state, depth) + } + + // Loss of contact is never evidence of process death: an unverifiable remote + // worker must still block nesting. See docs/reference/ssh-execution-boundary.md. + for (const state of ['starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown']) { + it(`counts a '${state}' attachment as a live parent`, () => { + db = new OrchestrationDb(':memory:') + attachRemoteWorker(state, 1) + expect( + db.resolveCreatorDepth({ + kind: 'terminal', + handle: 'term_remote', + paneKey: PANE, + processIncarnation: INCARNATION + }) + ).toBe(1) + }) + } + + for (const state of ['succeeded', 'failed', 'stopped', 'abandoned']) { + it(`does not count a settled '${state}' attachment`, () => { + db = new OrchestrationDb(':memory:') + attachRemoteWorker(state, 1) + expect( + db.resolveCreatorDepth({ + kind: 'terminal', + handle: 'term_remote', + paneKey: PANE, + processIncarnation: INCARNATION + }) + ).toBe(0) + }) + } + + it('ignores an attachment whose pane was reused by a new process', () => { + db = new OrchestrationDb(':memory:') + attachRemoteWorker('ready', 2) + expect( + db.resolveCreatorDepth({ + kind: 'terminal', + handle: 'term_remote', + paneKey: PANE, + processIncarnation: 'inc-2' + }) + ).toBe(0) + }) + + it('fails closed when one identity matches two live attachments', () => { + db = new OrchestrationDb(':memory:') + attachRemoteWorker('ready', 1) + attachRemoteWorker('starting', 2) + expect(() => + db.resolveCreatorDepth({ + kind: 'terminal', + handle: 'term_remote', + paneKey: PANE, + processIncarnation: INCARNATION + }) + ).toThrow(AmbiguousDispatchParentError) + }) + + it('takes the maximum when a process holds both a local and a remote role', () => { + // Query order must not decide the answer: the deeper role governs. + db = new OrchestrationDb(':memory:') + const task = db.createTask({ spec: 'local role' }) + db.createDispatchContext({ + taskId: task.id, + assigneeHandle: 'term_both', + assigneePaneKey: PANE, + creator: { kind: 'system' }, + maxDepth: UNCAPPED + }) + attachRemoteWorker('ready', 3) + expect( + db.resolveCreatorDepth({ + kind: 'terminal', + handle: 'term_both', + paneKey: PANE, + processIncarnation: INCARNATION + }) + ).toBe(3) + }) + }) + + describe('the supervised worker-start path', () => { + // r2 put enforcement in createDispatchContext and missed this entirely: + // worker-start has its own insert, and so does every retry through it. + function startWorker( + taskId: string, + creator: Parameters[0], + maxDepth: number + ) { + return db.createStartingWorkerDispatch({ + taskId, + startOptions: {}, + creator, + maxDepth + }) + } + + it('stamps depth 1 for a root coordinator', () => { + db = new OrchestrationDb(':memory:') + const task = db.createTask({ spec: 'root work' }) + expect(startWorker(task.id, SYSTEM, UNCAPPED).dispatch.depth).toBe(1) + }) + + it('refuses a worker starting a sub-worker at the default cap', () => { + coordinatorDispatchesWorker() + const nested = db.createTask({ spec: 'nested work' }) + expect(() => + startWorker( + nested.id, + { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' }, + 1 + ) + ).toThrow(/depth 2 \(max 1\)/) + }) + + it('refuses a worker retrying into a sub-worker at the default cap', () => { + coordinatorDispatchesWorker() + const nested = db.createTask({ spec: 'nested retry work' }) + const first = startWorker(nested.id, SYSTEM, UNCAPPED) + db.failWorkerStart(first.dispatch.id, 'accepted', 'first attempt failed') + expect(() => + db.createStartingWorkerDispatch({ + taskId: nested.id, + startOptions: {}, + retryOf: first.dispatch.id, + creator: { kind: 'terminal', handle: 'term_worker', paneKey: 'tab_worker:leaf_worker' }, + maxDepth: 1 + }) + ).toThrow(/depth 2 \(max 1\)/) + }) + }) + + it('keeps a local row with a null process incarnation eligible as a parent', () => { + // Context-only dispatch stores null on purpose; requiring an incarnation + // locally would silently drop real parents and fail open. + db = new OrchestrationDb(':memory:') + const task = db.createTask({ spec: 'context only' }) + const row = db.createDispatchContext({ + taskId: task.id, + assigneeHandle: 'term_ctx', + assigneePaneKey: 'tab_ctx:leaf_ctx', + creator: { kind: 'system' }, + maxDepth: UNCAPPED + }) + expect(row.process_incarnation).toBeNull() + expect(db.resolveCreatorDepth({ kind: 'terminal', handle: 'term_ctx' })).toBe(1) + }) +}) diff --git a/src/main/runtime/orchestration/db/dispatch-depth.ts b/src/main/runtime/orchestration/db/dispatch-depth.ts new file mode 100644 index 00000000000..c52ab27ba67 --- /dev/null +++ b/src/main/runtime/orchestration/db/dispatch-depth.ts @@ -0,0 +1,156 @@ +import { + NESTED_WORKER_DEPTH_EXCEEDED_CODE, + NESTED_WORKER_DEPTH_EXCEEDED_NEXT_STEPS, + ROOT_DISPATCH_DEPTH, + nestedWorkerDepthExceededMessage +} from '../../../../shared/nested-worker-depth' +import { OrchestrationError } from '../orchestration-error' +import { isEquivalentPaneKey } from './pane-key-match' +import type { OrchestrationDb } from './orchestration-db' +import type { DispatchContextRow, RemoteDispatchAttachmentRow } from '../types' + +/** + * Who is creating a dispatch row, for nesting-depth purposes. + * + * `system` is Orca's own in-process coordinator loop, which is host-local code + * rather than a CLI caller and is a root by construction. It is an internal + * discriminated branch on purpose — never a caller-supplied value, or a worker + * could claim to be the loop. + */ +export type DispatchCreator = + | { kind: 'system' } + | { + kind: 'terminal' + handle: string + paneKey?: string + /** Remote attachment matching requires the exact incarnation; local rows do not. */ + processIncarnation?: string + } + +/** + * Attachment states in which the worker may still be running. + * + * `start_unknown` means prompt delivery may have succeeded; `stopping` and + * `stop_unknown` do not establish that the process exited. Loss of contact is + * never evidence of process death — see docs/reference/ssh-execution-boundary.md. + * An `unverifiable` worker must still count as a nesting parent. + */ +const POTENTIALLY_LIVE_ATTACHMENT_STATES = [ + 'starting', + 'ready', + 'start_unknown', + 'stopping', + 'stop_unknown' +] as const + +export class AmbiguousDispatchParentError extends Error { + constructor(message: string) { + super(message) + this.name = 'AmbiguousDispatchParentError' + } +} + +/** + * Depth of the deepest live role this caller currently holds. + * + * Why the maximum rather than the first match: one terminal process can hold a + * local dispatch and a remote attachment at the same time, and the command + * cannot say which role motivated it. Taking the maximum cannot undercount, so + * it cannot let a deep worker pass as a shallow one. + */ +export function resolveCreatorDepth(this: OrchestrationDb, creator: DispatchCreator): number { + if (creator.kind === 'system') { + return ROOT_DISPATCH_DEPTH + } + + const depths: number[] = [] + + // Local rows match on handle/pane as they always have. process_incarnation is + // nullable here and context-only dispatch stores null deliberately, so + // requiring it would drop real parents. + const local = this.findActiveDispatchForAssignee(creator.handle, creator.paneKey) as + | DispatchContextRow + | undefined + if (local) { + depths.push(local.depth) + } + + for (const attachment of findPotentiallyLiveAttachmentsForCreator.call(this, creator)) { + depths.push(attachment.depth) + } + + return depths.length > 0 ? Math.max(...depths) : ROOT_DISPATCH_DEPTH +} + +/** + * Remote attachments matching this caller's pane AND exact process incarnation. + * + * Handle is deliberately not compared: identity survives handle remint and + * nothing updates `remote_dispatch_attachments.terminal_handle` when it happens, + * so a stored-handle predicate would reject a live federated parent. Pane + * equivalence plus exact incarnation is what remote authority already uses. + */ +function findPotentiallyLiveAttachmentsForCreator( + this: OrchestrationDb, + creator: Extract +): RemoteDispatchAttachmentRow[] { + if (!creator.paneKey || !creator.processIncarnation) { + return [] + } + const placeholders = POTENTIALLY_LIVE_ATTACHMENT_STATES.map(() => '?').join(', ') + const rows = this.db + .prepare( + `SELECT * FROM remote_dispatch_attachments + WHERE process_incarnation = ? + AND pane_key IS NOT NULL + AND state IN (${placeholders})` + ) + .all( + creator.processIncarnation, + ...POTENTIALLY_LIVE_ATTACHMENT_STATES + ) as RemoteDispatchAttachmentRow[] + + const matches = rows.filter( + (row) => row.pane_key !== null && isEquivalentPaneKey(row.pane_key, creator.paneKey as string) + ) + + // Two live attachments for one identity is an anomaly, not a depth question. + // Surface it rather than silently picking one. + if (matches.length > 1) { + throw new AmbiguousDispatchParentError( + `Terminal ${creator.handle} matches ${matches.length} live remote attachments; cannot establish nesting depth.` + ) + } + return matches +} + +/** + * Depth to stamp on a row this creator is about to make, rejecting over-cap. + * + * Every path that mints a live worker goes through here, so the cap cannot be + * skipped by adding a new spawn verb. + */ +export function resolveChildDispatchDepth( + this: OrchestrationDb, + creator: DispatchCreator, + maxDepth: number +): number { + const childDepth = this.resolveCreatorDepth(creator) + 1 + if (childDepth > maxDepth) { + throw new OrchestrationError( + NESTED_WORKER_DEPTH_EXCEEDED_CODE, + nestedWorkerDepthExceededMessage(childDepth, maxDepth), + { effectsApplied: false, nextSteps: [...NESTED_WORKER_DEPTH_EXCEEDED_NEXT_STEPS] } + ) + } + return childDepth +} + +export type DispatchDepthMethods = { + resolveCreatorDepth: typeof resolveCreatorDepth + resolveChildDispatchDepth: typeof resolveChildDispatchDepth +} + +export function attachDispatchDepth(ctor: { prototype: object }): void { + Object.assign(ctor.prototype, { resolveCreatorDepth, resolveChildDispatchDepth }) +} diff --git a/src/main/runtime/orchestration/db/dispatch-row-writer-boundary.test.ts b/src/main/runtime/orchestration/db/dispatch-row-writer-boundary.test.ts new file mode 100644 index 00000000000..9b7581d6acd --- /dev/null +++ b/src/main/runtime/orchestration/db/dispatch-row-writer-boundary.test.ts @@ -0,0 +1,139 @@ +import { readFileSync, readdirSync, statSync } from 'node:fs' +import { join, relative, resolve } from 'node:path' +import { describe, expect, it } from 'vitest' + +/** + * Guard the live-worker row chokepoint at the tree level rather than per call site. + * + * Nesting depth has to be stamped on every row that represents a live supervised + * worker. Three separate modules used to own their own INSERT, and three review + * rounds each found one more spawn path than the previous round believed existed. + * `dispatch-row-writer.ts` owns the statements once; this test is what stops the + * fourth path from owning one again. + * + * Known limit, recorded rather than assumed away: this scans SQL string literals. + * SQL assembled from a shared table-name constant, split template fragments, or a + * query builder would evade it — see the detector cases below. + */ +const WRITER_MODULE = 'src/main/runtime/orchestration/db/dispatch-row-writer.ts' + +const GUARDED_TABLES = ['dispatch_contexts', 'remote_dispatch_attachments'] as const + +/** `INSERT ... INTO `, tolerating OR-clauses and newlines between the words. */ +const insertPattern = (table: string): RegExp => + new RegExp(String.raw`INSERT\b[\s\S]{0,40}?\bINTO\s+${table}\b`, 'i') + +/** + * Schema DDL, migrations, and reset all legitimately name these tables. They + * create, alter, and delete rows — they never mint a live worker. + */ +const EXEMPT_PATH_FRAGMENTS = ['/db/schema/', '/db/reset/', '/orchestration-schema-version-skew'] + +const SCANNED_EXTENSIONS = ['.ts', '.tsx'] +const IGNORED_DIRECTORIES = new Set(['node_modules', 'dist', 'out', 'build', '.git']) + +function isTestFile(path: string): boolean { + return ( + /\.(?:test|spec)\.tsx?$/.test(path) || + /(?:test-harness|test-utils|test-setup|test-fixture)/.test(path) || + path.includes('/__tests__/') || + path.includes('/__fixtures__/') + ) +} + +function collectSourceFiles(root: string): string[] { + const found: string[] = [] + let entries: string[] + try { + entries = readdirSync(root) + } catch { + return found + } + for (const entry of entries) { + if (IGNORED_DIRECTORIES.has(entry)) { + continue + } + const full = join(root, entry) + if (statSync(full).isDirectory()) { + found.push(...collectSourceFiles(full)) + } else if (SCANNED_EXTENSIONS.some((ext) => entry.endsWith(ext))) { + found.push(full) + } + } + return found +} + +describe('live-worker row insert boundary', () => { + const repoRoot = resolve(__dirname, '../../../../..') + const srcRoot = join(repoRoot, 'src') + + it('inserts guarded tables only from dispatch-row-writer.ts', () => { + const offenders: string[] = [] + for (const file of collectSourceFiles(srcRoot)) { + const rel = relative(repoRoot, file).split('\\').join('/') + if (rel === WRITER_MODULE || isTestFile(rel)) { + continue + } + if (EXEMPT_PATH_FRAGMENTS.some((fragment) => rel.includes(fragment))) { + continue + } + const contents = readFileSync(file, 'utf8') + for (const table of GUARDED_TABLES) { + if (insertPattern(table).test(contents)) { + offenders.push(`${rel} inserts ${table}`) + } + } + } + expect(offenders).toEqual([]) + }) + + it('the writer module actually owns an insert for every guarded table', () => { + const contents = readFileSync(join(repoRoot, WRITER_MODULE), 'utf8') + for (const table of GUARDED_TABLES) { + expect(insertPattern(table).test(contents)).toBe(true) + } + }) + + it('does not fire on schema DDL, migration DDL, or reset SQL', () => { + // Why explicit: a naive identifier scan flags all three, which is how the + // first two drafts of this ratchet failed against their own tree. + const exempt = [ + 'src/main/runtime/orchestration/db/schema/create-graph-tables-sql.ts', + 'src/main/runtime/orchestration/db/schema/migrate-v13-v30.ts', + 'src/main/runtime/orchestration/db/reset/orchestration-reset.ts' + ] + for (const rel of exempt) { + expect( + EXEMPT_PATH_FRAGMENTS.some((fragment) => rel.includes(fragment)), + `${rel} must be exempt` + ).toBe(true) + } + }) + + it('detects the insert forms it claims to detect', () => { + expect(insertPattern('dispatch_contexts').test('INSERT INTO dispatch_contexts (id)')).toBe(true) + expect( + insertPattern('dispatch_contexts').test('INSERT OR REPLACE INTO dispatch_contexts (id)') + ).toBe(true) + expect(insertPattern('dispatch_contexts').test('INSERT\n INTO dispatch_contexts')).toBe(true) + expect(insertPattern('dispatch_contexts').test('SELECT * FROM dispatch_contexts')).toBe(false) + expect(insertPattern('dispatch_contexts').test('DELETE FROM dispatch_contexts')).toBe(false) + // Guards against matching the longer sibling table name by prefix. + expect(insertPattern('dispatch_contexts').test('INSERT INTO dispatch_contexts_archive')).toBe( + false + ) + }) + + it('records the evasions this scanner cannot catch', () => { + // Why asserted rather than commented: these are the scanner's known blind + // spots. Centralization is the convention; this test only guards the common + // form. If any of these ever becomes reachable in production SQL, the + // boundary needs an AST-level check instead. + const dynamicTable = 'const t = "dispatch_contexts"; db.prepare(`INSERT INTO ${t} (id)`)' + const splitLiteral = 'db.prepare("INSERT INTO " + "dispatch_contexts (id)")' + const queryBuilder = 'db.insertInto("dispatch_contexts").values({ id })' + expect(insertPattern('dispatch_contexts').test(dynamicTable)).toBe(false) + expect(insertPattern('dispatch_contexts').test(splitLiteral)).toBe(false) + expect(insertPattern('dispatch_contexts').test(queryBuilder)).toBe(false) + }) +}) diff --git a/src/main/runtime/orchestration/db/dispatch-row-writer.ts b/src/main/runtime/orchestration/db/dispatch-row-writer.ts new file mode 100644 index 00000000000..606081e58a3 --- /dev/null +++ b/src/main/runtime/orchestration/db/dispatch-row-writer.ts @@ -0,0 +1,144 @@ +import type Database from '../../../sqlite/sync-database' +import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL } from './pane-key-match' + +/** + * The only place that inserts rows representing a live supervised worker. + * + * Why centralized: nesting depth must be stamped on every such row, and three + * separate modules used to own their own INSERT. A boundary test forbids these + * statements anywhere else, so a new spawn path cannot skip the stamp. + * + * Transaction-neutral on purpose — each caller keeps its own BEGIN IMMEDIATE or + * SAVEPOINT, mutation-receipt write, and companion inserts. + */ + +export const DISPATCH_CONTEXT_CLAIM_SQL = `INSERT INTO dispatch_contexts ( + id, run_id, task_id, contract_version, launch_token_hash, + assignee_handle, assignee_pane_key, process_incarnation, + status, failure_count, depth, dispatched_at +) +SELECT ?, run_id, id, ?, ?, ?, ?, ?, 'dispatched', ?, ?, datetime('now') +FROM tasks +WHERE id = ? AND status = 'ready' + AND NOT EXISTS ( + SELECT 1 FROM dispatch_contexts active + WHERE active.assignee_handle = ? + AND active.status IN ('pending', 'dispatched') + ) + AND ( + ? IS NULL OR NOT EXISTS ( + SELECT 1 FROM dispatch_contexts active + WHERE active.assignee_pane_key = ? + AND active.status IN ('pending', 'dispatched') + ) + ) + AND ( + ? IS NULL OR NOT EXISTS ( + SELECT 1 FROM dispatch_contexts active + WHERE active.assignee_pane_key IS NOT NULL + AND active.status IN ('pending', 'dispatched') + AND instr(active.assignee_pane_key, ':') > 1 + AND ${DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL} = ? + ) + )` + +const STARTING_DISPATCH_CONTEXT_SQL = `INSERT INTO dispatch_contexts ( + id, run_id, task_id, contract_version, launch_token_hash, depth, status, dispatched_at + ) VALUES (?, ?, ?, ?, ?, ?, 'pending', datetime('now'))` + +const REMOTE_DISPATCH_ATTACHMENT_SQL = `INSERT INTO remote_dispatch_attachments ( + dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch, depth + ) VALUES (?, ?, ?, ?, ?, ?)` + +/** Last line of defence: a row that reached here unstamped would read as a root. */ +function assertStampedDepth(depth: number): void { + if (!Number.isInteger(depth) || depth < 1) { + throw new Error( + `Refusing to write a live-worker row with depth ${depth}; expected an integer >= 1.` + ) + } +} + +/** Adopts an existing agent terminal, claiming a ready task atomically. */ +export function claimDispatchContextRow( + db: Database.Database, + params: { + id: string + contractVersion: number + launchTokenHash: string | null + assigneeHandle: string + assigneePaneKey: string | null + processIncarnation: string | null + priorFailures: number + depth: number + taskId: string + paneSuffix: string | null + } +): { changes: number | bigint } { + assertStampedDepth(params.depth) + return db + .prepare(DISPATCH_CONTEXT_CLAIM_SQL) + .run( + params.id, + params.contractVersion, + params.launchTokenHash, + params.assigneeHandle, + params.assigneePaneKey, + params.processIncarnation, + params.priorFailures, + params.depth, + params.taskId, + params.assigneeHandle, + params.assigneePaneKey, + params.assigneePaneKey, + params.paneSuffix, + params.paneSuffix + ) +} + +/** Supervised `worker-start`, including every retry and the federated home side. */ +export function insertStartingDispatchContextRow( + db: Database.Database, + params: { + id: string + runId: string + taskId: string + contractVersion: number + launchTokenHash: string | null + depth: number + } +): void { + assertStampedDepth(params.depth) + db.prepare(STARTING_DISPATCH_CONTEXT_SQL).run( + params.id, + params.runId, + params.taskId, + params.contractVersion, + params.launchTokenHash, + params.depth + ) +} + +/** The worker host's record of a live worker driven by a remote Run home. */ +export function insertRemoteDispatchAttachmentRow( + db: Database.Database, + params: { + dispatchId: string + taskId: string + homePeerFingerprint: string + protocolVersion: number + runtimeEpoch: string + /** Propagated from the Run home, not computed here; absent (old client) = 1. */ + depth: number + } +): void { + assertStampedDepth(params.depth) + db.prepare(REMOTE_DISPATCH_ATTACHMENT_SQL).run( + params.dispatchId, + params.taskId, + params.homePeerFingerprint, + params.protocolVersion, + params.runtimeEpoch, + params.depth + ) +} diff --git a/src/main/runtime/orchestration/db/federation/remote-dispatch-attachment-create.ts b/src/main/runtime/orchestration/db/federation/remote-dispatch-attachment-create.ts index bb724f74cb8..56d26ccfe3b 100644 --- a/src/main/runtime/orchestration/db/federation/remote-dispatch-attachment-create.ts +++ b/src/main/runtime/orchestration/db/federation/remote-dispatch-attachment-create.ts @@ -2,6 +2,7 @@ import type { WorkerDispatchState, RemoteDispatchAttachmentRow } from '../../typ import { OrchestrationError } from '../../orchestration-error' import { ensureMutationReceiptCapacity } from '../../mutation-receipt-capacity' import type { OrchestrationDb } from '../orchestration-db' +import { insertRemoteDispatchAttachmentRow } from '../dispatch-row-writer' export function createRemoteDispatchAttachment( this: OrchestrationDb, @@ -11,6 +12,8 @@ export function createRemoteDispatchAttachment( homePeerFingerprint: string protocolVersion: number runtimeEpoch: string + /** Child depth computed by the Run home; absent from an old client = 1 (fails closed). */ + depth?: number mutationReceipt: { callerFingerprint: string requestId: string @@ -54,19 +57,14 @@ export function createRemoteDispatchAttachment( params.mutationReceipt.payloadHash, JSON.stringify({ accepted: { dispatchId: params.dispatchId } }) ) - this.db - .prepare( - `INSERT INTO remote_dispatch_attachments ( - dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch - ) VALUES (?, ?, ?, ?, ?)` - ) - .run( - params.dispatchId, - params.taskId, - params.homePeerFingerprint, - params.protocolVersion, - params.runtimeEpoch - ) + insertRemoteDispatchAttachmentRow(this.db, { + dispatchId: params.dispatchId, + taskId: params.taskId, + homePeerFingerprint: params.homePeerFingerprint, + protocolVersion: params.protocolVersion, + runtimeEpoch: params.runtimeEpoch, + depth: params.depth ?? 1 + }) this.db.exec('COMMIT') return this.getRemoteDispatchAttachment(params.dispatchId) as RemoteDispatchAttachmentRow } catch (error) { diff --git a/src/main/runtime/orchestration/db/orchestration-db-methods.ts b/src/main/runtime/orchestration/db/orchestration-db-methods.ts index 8fa48a0edd3..7f25b209c54 100644 --- a/src/main/runtime/orchestration/db/orchestration-db-methods.ts +++ b/src/main/runtime/orchestration/db/orchestration-db-methods.ts @@ -4,6 +4,7 @@ import type { DispatchCapabilityMethods } from './dispatch-context/dispatch-capa import type { DispatchCompletionMethods } from './dispatch-context/dispatch-completion' import type { DispatchContextStoreMethods } from './dispatch-context/dispatch-context-store' import type { DispatchLookupMethods } from './dispatch-context/dispatch-lookup' +import type { DispatchDepthMethods } from './dispatch-depth' import type { WorkerReportSettlementMethods } from './dispatch-context/worker-report-settlement' import type { FederatedDispatchStoreMethods } from './federation/federated-dispatch-store' import type { FederationRelayAckMethods } from './federation/federation-relay-ack' @@ -114,6 +115,7 @@ export type OrchestrationDbMethods = CreateTablesMethods & DispatchContextStoreMethods & DispatchCapabilityMethods & DispatchLookupMethods & + DispatchDepthMethods & DispatchCompletionMethods & WorkerReportSettlementMethods & DecisionGateStoreMethods & diff --git a/src/main/runtime/orchestration/db/root-dispatch-test-fixture.ts b/src/main/runtime/orchestration/db/root-dispatch-test-fixture.ts new file mode 100644 index 00000000000..4b16c95e0a6 --- /dev/null +++ b/src/main/runtime/orchestration/db/root-dispatch-test-fixture.ts @@ -0,0 +1,28 @@ +import type { DispatchContextRow } from '../types' +import type { OrchestrationDb } from './orchestration-db' + +/** + * Dispatch as a root coordinator with no nesting cap. + * + * Tests that predate nesting depth care about dispatch behaviour, not the cap; + * this keeps them at their original call shape instead of repeating the same + * creator/maxDepth pair at every site. + */ +export function createRootDispatch( + db: OrchestrationDb, + taskId: string, + assigneeHandle: string, + assigneePaneKey?: string, + launchTokenHash?: string, + processIncarnation?: string +): DispatchContextRow { + return db.createDispatchContext({ + taskId, + assigneeHandle, + assigneePaneKey, + launchTokenHash, + processIncarnation, + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER + }) +} diff --git a/src/main/runtime/orchestration/db/schema/create-graph-tables-sql.ts b/src/main/runtime/orchestration/db/schema/create-graph-tables-sql.ts index 720b7153cea..07a47c3c80c 100644 --- a/src/main/runtime/orchestration/db/schema/create-graph-tables-sql.ts +++ b/src/main/runtime/orchestration/db/schema/create-graph-tables-sql.ts @@ -42,17 +42,24 @@ CREATE TABLE IF NOT EXISTS remote_dispatch_attachments ( effects TEXT NOT NULL DEFAULT '[]', residual_resources TEXT NOT NULL DEFAULT '[]', to_worker_imported_sequence INTEGER NOT NULL DEFAULT 0, + -- Nesting depth of the worker this attachment represents. Propagated from the + -- Run home; absent from an old client means 1, which fails closed. + depth INTEGER NOT NULL DEFAULT 1, last_error TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), updated_at TEXT NOT NULL DEFAULT (datetime('now')) ); +-- Why five states: 'start_unknown', 'stopping', and 'stop_unknown' do not +-- establish process exit, and a potentially-live worker must still count as a +-- nesting parent. See docs/reference/ssh-execution-boundary.md. CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane ON remote_dispatch_attachments(pane_key) - WHERE state IN ('starting', 'ready'); + WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown'); CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane_suffix ON remote_dispatch_attachments(${REMOTE_ATTACHMENT_PANE_KEY_MATCH_SUFFIX_SQL}) - WHERE state IN ('starting', 'ready') AND pane_key IS NOT NULL; + WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown') + AND pane_key IS NOT NULL; CREATE TABLE IF NOT EXISTS federation_relay_items ( dispatch_id TEXT NOT NULL, @@ -127,6 +134,9 @@ CREATE TABLE IF NOT EXISTS dispatch_contexts ( last_failure TEXT, -- Why the process is gone, when Orca could establish it. See TerminalExitCause. termination_reason TEXT, + -- Nesting depth: a root coordinator's worker is 1, its worker's worker is 2. + -- Defaults to 1 so an unstamped row fails closed rather than reading as a root. + depth INTEGER NOT NULL DEFAULT 1, dispatched_at TEXT, completed_at TEXT, created_at TEXT NOT NULL DEFAULT (datetime('now')), diff --git a/src/main/runtime/orchestration/db/schema/migrate-v13-v29.ts b/src/main/runtime/orchestration/db/schema/migrate-v13-v30.ts similarity index 84% rename from src/main/runtime/orchestration/db/schema/migrate-v13-v29.ts rename to src/main/runtime/orchestration/db/schema/migrate-v13-v30.ts index b3129dc7ed6..654395bcd2c 100644 --- a/src/main/runtime/orchestration/db/schema/migrate-v13-v29.ts +++ b/src/main/runtime/orchestration/db/schema/migrate-v13-v30.ts @@ -1,8 +1,11 @@ import { migrateMutationReceiptCapacity } from '../../mutation-receipt-capacity' -import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL } from '../pane-key-match' +import { + DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL, + REMOTE_ATTACHMENT_PANE_KEY_MATCH_SUFFIX_SQL +} from '../pane-key-match' import type { OrchestrationDb } from '../orchestration-db' -export function applySchemaMigrationsV13ToV29(this: OrchestrationDb, current: number): void { +export function applySchemaMigrationsV13ToV30(this: OrchestrationDb, current: number): void { if (current < 13 && !this.hasColumn('worker_dispatches', 'runtime_epoch')) { this.db.exec('ALTER TABLE worker_dispatches ADD COLUMN runtime_epoch TEXT') } @@ -157,6 +160,29 @@ export function applySchemaMigrationsV13ToV29(this: OrchestrationDb, current: nu if (current < 29 && !this.hasColumn('dispatch_contexts', 'termination_reason')) { this.db.exec('ALTER TABLE dispatch_contexts ADD COLUMN termination_reason TEXT') } + if (current < 30) { + if (!this.hasColumn('dispatch_contexts', 'depth')) { + this.db.exec('ALTER TABLE dispatch_contexts ADD COLUMN depth INTEGER NOT NULL DEFAULT 1') + } + if (!this.hasColumn('remote_dispatch_attachments', 'depth')) { + this.db.exec( + 'ALTER TABLE remote_dispatch_attachments ADD COLUMN depth INTEGER NOT NULL DEFAULT 1' + ) + } + // Why drop first: CREATE INDEX IF NOT EXISTS cannot widen an existing + // partial index predicate, and these two covered only starting/ready. + this.db.exec(` + DROP INDEX IF EXISTS idx_remote_dispatch_attachments_active_pane; + DROP INDEX IF EXISTS idx_remote_dispatch_attachments_active_pane_suffix; + CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane + ON remote_dispatch_attachments(pane_key) + WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown'); + CREATE INDEX IF NOT EXISTS idx_remote_dispatch_attachments_active_pane_suffix + ON remote_dispatch_attachments(${REMOTE_ATTACHMENT_PANE_KEY_MATCH_SUFFIX_SQL}) + WHERE state IN ('starting', 'ready', 'start_unknown', 'stopping', 'stop_unknown') + AND pane_key IS NOT NULL; + `) + } this.db.exec(` CREATE INDEX IF NOT EXISTS idx_dispatch_assignee_pane_leaf ON dispatch_contexts(${DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL}) diff --git a/src/main/runtime/orchestration/db/schema/migrate.ts b/src/main/runtime/orchestration/db/schema/migrate.ts index 3c33b5c8ce5..b29debf6aa1 100644 --- a/src/main/runtime/orchestration/db/schema/migrate.ts +++ b/src/main/runtime/orchestration/db/schema/migrate.ts @@ -1,7 +1,7 @@ import { resolveOrchestrationMigrationStartVersion } from '../../orchestration-schema-version-skew' import { SCHEMA_VERSION } from '../contract-constants' import type { OrchestrationDb } from '../orchestration-db' -import { applySchemaMigrationsV13ToV29 } from './migrate-v13-v29' +import { applySchemaMigrationsV13ToV30 } from './migrate-v13-v30' import { applySchemaMigrationsV2ToV12 } from './migrate-v2-v12' // Why: CREATE TABLE IF NOT EXISTS won't alter existing DBs; migrate in a txn that bumps user_version only on success (atomic all-or-nothing). @@ -15,7 +15,7 @@ export function migrate(this: OrchestrationDb): void { this.db.exec('BEGIN IMMEDIATE') try { applySchemaMigrationsV2ToV12.call(this, current) - applySchemaMigrationsV13ToV29.call(this, current) + applySchemaMigrationsV13ToV30.call(this, current) this.db.pragma(`user_version = ${SCHEMA_VERSION}`) this.db.exec('COMMIT') } catch (err) { diff --git a/src/main/runtime/orchestration/db/worker-dispatch/worker-dispatch-start.ts b/src/main/runtime/orchestration/db/worker-dispatch/worker-dispatch-start.ts index 91f590895dc..e26369e7fc1 100644 --- a/src/main/runtime/orchestration/db/worker-dispatch/worker-dispatch-start.ts +++ b/src/main/runtime/orchestration/db/worker-dispatch/worker-dispatch-start.ts @@ -4,6 +4,8 @@ import { ensureMutationReceiptCapacity } from '../../mutation-receipt-capacity' import { CURRENT_CONTRACT_VERSION } from '../contract-constants' import { generateId } from '../generated-id' import type { OrchestrationDb } from '../orchestration-db' +import { insertStartingDispatchContextRow } from '../dispatch-row-writer' +import type { DispatchCreator } from '../dispatch-depth' export function createStartingWorkerDispatch( this: OrchestrationDb, @@ -25,6 +27,9 @@ export function createStartingWorkerDispatch( method: string payloadHash: string } + /** Who is dispatching, for nesting depth. Required so a new caller must decide. */ + creator: DispatchCreator + maxDepth: number } ): { dispatch: DispatchContextRow; worker: WorkerDispatchRow } { this.db.exec('BEGIN IMMEDIATE') @@ -95,13 +100,14 @@ export function createStartingWorkerDispatch( params.mutationReceipt.requestId ) } - this.db - .prepare( - `INSERT INTO dispatch_contexts ( - id, run_id, task_id, contract_version, launch_token_hash, status, dispatched_at - ) VALUES (?, ?, ?, ?, ?, 'pending', datetime('now'))` - ) - .run(id, task.run_id, task.id, CURRENT_CONTRACT_VERSION, params.launchTokenHash ?? null) + insertStartingDispatchContextRow(this.db, { + id, + runId: task.run_id, + taskId: task.id, + contractVersion: CURRENT_CONTRACT_VERSION, + launchTokenHash: params.launchTokenHash ?? null, + depth: this.resolveChildDispatchDepth(params.creator, params.maxDepth) + }) this.db .prepare( `INSERT INTO worker_dispatches ( diff --git a/src/main/runtime/orchestration/dispatch-failure-idempotency.test.ts b/src/main/runtime/orchestration/dispatch-failure-idempotency.test.ts index 9b526c480cc..c6f75723cf3 100644 --- a/src/main/runtime/orchestration/dispatch-failure-idempotency.test.ts +++ b/src/main/runtime/orchestration/dispatch-failure-idempotency.test.ts @@ -1,12 +1,13 @@ import { describe, expect, it } from 'vitest' import type Database from '../../sqlite/sync-database' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('dispatch failure idempotency', () => { it('counts an active dispatch failure only once', () => { const db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') expect(db.failDispatch(dispatch.id, 'exit')?.failure_count).toBe(1) const duplicate = db.failDispatch(dispatch.id, 'duplicate escalation') @@ -19,7 +20,7 @@ describe('dispatch failure idempotency', () => { it('does not overwrite a completed dispatch', () => { const db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') db.completeDispatch(dispatch.id) const lateFailure = db.failDispatch(dispatch.id, 'late exit') @@ -33,7 +34,7 @@ describe('dispatch failure idempotency', () => { const db = new OrchestrationDb(':memory:') const sqlite = (db as unknown as { db: Database.Database }).db const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') sqlite.exec(` CREATE TRIGGER reject_task_failure_update BEFORE UPDATE ON tasks WHEN OLD.id = '${task.id}' diff --git a/src/main/runtime/orchestration/federation-sync.test.ts b/src/main/runtime/orchestration/federation-sync.test.ts index 55372924d57..a944b494660 100644 --- a/src/main/runtime/orchestration/federation-sync.test.ts +++ b/src/main/runtime/orchestration/federation-sync.test.ts @@ -128,6 +128,8 @@ describe('federation relay parsing', () => { }) const task = db.createTask({ spec: 'Remote work', runId: run.id }) const { dispatch } = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, federation: { @@ -196,6 +198,8 @@ describe('federation relay acknowledgments', () => { }) const task = db.createTask({ spec: 'Remote work', runId: run.id }) const { dispatch } = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, federation: { diff --git a/src/main/runtime/orchestration/lifecycle-reconciliation.test.ts b/src/main/runtime/orchestration/lifecycle-reconciliation.test.ts index 3a916751533..7011c191c00 100644 --- a/src/main/runtime/orchestration/lifecycle-reconciliation.test.ts +++ b/src/main/runtime/orchestration/lifecycle-reconciliation.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it } from 'vitest' import { OrchestrationDb } from './db' import { reconcileLifecycleMessage } from './lifecycle-reconciliation' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('lifecycle reconciliation', () => { let db: OrchestrationDb @@ -10,7 +11,7 @@ describe('lifecycle reconciliation', () => { it('rejects handle churn when neither side has stable pane identity', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_before_restart') + const dispatch = createRootDispatch(db, task.id, 'term_before_restart') const logs: string[] = [] const message = db.insertMessage({ from: 'term_after_restart', @@ -37,7 +38,7 @@ describe('lifecycle reconciliation', () => { it('completes worker_done from the dispatched pane after a handle remint', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_before_restart', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_before_restart', `tab_w:${LEAF_A}`) const message = db.insertMessage({ from: 'term_after_restart', to: 'term_coordinator', @@ -54,7 +55,7 @@ describe('lifecycle reconciliation', () => { it('fails both the dispatch and task from an authenticated failed worker report', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`) const message = db.insertMessage({ from: 'term_worker', to: 'term_coordinator', @@ -87,7 +88,7 @@ describe('lifecycle reconciliation', () => { it('replays an identical terminal outcome without mutating settled state', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const makeMessage = () => db.insertMessage({ from: 'term_worker', @@ -144,7 +145,7 @@ describe('lifecycle reconciliation', () => { const task = db.createTask({ spec: 'work' }) // Dispatch recorded the post-break-out pane key; the worker shell still // holds the spawn-time key with the old tab id. - const dispatch = db.createDispatchContext(task.id, 'term_before_restart', `tab_new:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_before_restart', `tab_new:${LEAF_A}`) const message = db.insertMessage({ from: 'term_after_restart', to: 'term_coordinator', @@ -161,7 +162,7 @@ describe('lifecycle reconciliation', () => { it('rejects mismatched opaque pane keys instead of treating them as legacy', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w:${LEAF_A}`) const message = db.insertMessage({ from: 'term_reminted', to: 'term_coordinator', @@ -178,7 +179,7 @@ describe('lifecycle reconciliation', () => { it('rejects worker_done from a foreign pane that claims the assignee handle', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`) const message = db.insertMessage({ from: 'term_owner', to: 'term_coordinator', @@ -221,7 +222,7 @@ describe('lifecycle reconciliation', () => { it('does not let a caller-supplied rejection marker turn completion into success', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`) const message = db.insertMessage({ from: 'term_worker', to: 'term_coordinator', @@ -250,7 +251,7 @@ describe('lifecycle reconciliation', () => { it('rejects a coordinator completion for a pane-bound dispatch', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`) const message = db.insertMessage({ from: 'term_coordinator', to: 'term_coordinator', @@ -269,7 +270,7 @@ describe('lifecycle reconciliation', () => { it('uses exact handle equality only for a legacy dispatch without a pane key', () => { db = new OrchestrationDb(':memory:') const acceptedTask = db.createTask({ spec: 'legacy work' }) - const acceptedDispatch = db.createDispatchContext(acceptedTask.id, 'term_legacy') + const acceptedDispatch = createRootDispatch(db, acceptedTask.id, 'term_legacy') const accepted = db.insertMessage({ from: 'term_legacy', to: 'term_coordinator', @@ -284,7 +285,7 @@ describe('lifecycle reconciliation', () => { expect(reconcileLifecycleMessage(db, accepted).action).toBe('completed') const rejectedTask = db.createTask({ spec: 'other legacy work' }) - const rejectedDispatch = db.createDispatchContext(rejectedTask.id, 'term_other_legacy') + const rejectedDispatch = createRootDispatch(db, rejectedTask.id, 'term_other_legacy') const rejected = db.insertMessage({ from: 'term_foreign', to: 'term_coordinator', @@ -307,7 +308,7 @@ describe('lifecycle reconciliation', () => { db = new OrchestrationDb(':memory:') const parent = db.createTask({ spec: 'parent' }) const child = db.createTask({ spec: 'child', deps: [parent.id] }) - const dispatch = db.createDispatchContext(parent.id, 'term_worker', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, parent.id, 'term_worker', `tab_w:${LEAF_A}`) const payload = JSON.stringify({ taskId: parent.id, dispatchId: dispatch.id, @@ -343,7 +344,7 @@ describe('lifecycle reconciliation', () => { it('does not let a foreign replay overwrite an authorized completion', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', `tab_w:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_worker', `tab_w:${LEAF_A}`) const payload = JSON.stringify({ taskId: task.id, dispatchId: dispatch.id, @@ -378,7 +379,7 @@ describe('lifecycle reconciliation', () => { it('surfaces worker_done sent from a different pane as rejected', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`) const logs: string[] = [] const message = db.insertMessage({ from: 'term_other_worker', @@ -401,7 +402,7 @@ describe('lifecycle reconciliation', () => { it('surfaces a heartbeat sent from a different pane without recording liveness', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`) const heartbeat = db.insertMessage({ from: 'term_other_worker', to: 'term_coordinator', @@ -435,7 +436,7 @@ describe('lifecycle reconciliation', () => { it('surfaces a foreign heartbeat that claims the assignee handle', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_w1:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_w1:${LEAF_A}`) const heartbeat = db.insertMessage({ from: 'term_owner', to: 'term_coordinator', @@ -455,7 +456,7 @@ describe('lifecycle reconciliation', () => { it('records a heartbeat whose pane key drifted only in the tab half', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_owner', `tab_new:${LEAF_A}`) + const dispatch = createRootDispatch(db, task.id, 'term_owner', `tab_new:${LEAF_A}`) const heartbeat = db.insertMessage({ from: 'term_owner', to: 'term_coordinator', @@ -475,9 +476,9 @@ describe('lifecycle reconciliation', () => { it('suppresses same-dispatch heartbeats once worker_done is reconciled', () => { db = new OrchestrationDb(':memory:') const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const otherTask = db.createTask({ spec: 'other work' }) - const otherDispatch = db.createDispatchContext(otherTask.id, 'term_other') + const otherDispatch = createRootDispatch(db, otherTask.id, 'term_other') const insertHeartbeat = (dispatchId: string, from: string) => db.insertMessage({ from, diff --git a/src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts b/src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts index 43c0cc7a367..f34b6352e06 100644 --- a/src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts +++ b/src/main/runtime/orchestration/lightweight-run-worker-exit-escalation.test.ts @@ -5,6 +5,7 @@ import { DISPATCH_CIRCUIT_BREAK_FAILURES } from './db/dispatch-context/dispatch- import { makePaneKey } from '../../../shared/stable-pane-id' import { getDefaultWorkspaceSession } from '../../../shared/constants' import type { WorkspaceSessionState } from '../../../shared/workspace-session-state-types' +import { createRootDispatch } from './db/root-dispatch-test-fixture' // STA-4604: failActiveDispatchOnExit fails the dispatch on worker PTY exit but used to // gate the "Agent exited unexpectedly" escalation on the legacy coordinator_runs table. @@ -130,7 +131,7 @@ async function gradeWorkerExit( db.createCoordinatorRun({ spec: 'legacy coordinator loop', coordinatorHandle }) } const task = db.createTask({ spec: 'do the work', runId }) - const dispatch = db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY) + const dispatch = createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY) runtime.setOrchestrationDb(db as never) runtime.onPtyExit(WORKER_PTY_ID, 137) @@ -239,7 +240,7 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => { coordinatorPaneKey: COORDINATOR_PANE_KEY }) const task = db.createTask({ spec: 'do the work', runId: run.id }) - db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY) + createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY) runtime.setOrchestrationDb(db as never) const waiting = runtime.waitForMessage(`run:${run.id}`, { @@ -281,7 +282,7 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => { coordinatorPaneKey: makePaneKey('tab-other', '33333333-3333-4333-8333-333333333333') }) const task = db.createTask({ spec: 'owned work', runId: ownRun.id }) - db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY) + createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY) runtime.setOrchestrationDb(db as never) runtime.onPtyExit(WORKER_PTY_ID, 137) @@ -306,7 +307,12 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => { coordinatorPaneKey: COORDINATOR_PANE_KEY }) const task = db.createTask({ spec: 'supervised work', runId: run.id }) - const started = db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) db.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: workerHandle, @@ -350,10 +356,10 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => { const task = db.createTask({ spec: 'repeatedly failing work', runId: run.id }) // Burn the breaker down to its last life so this exit is the one that trips it. for (let attempt = 1; attempt < DISPATCH_CIRCUIT_BREAK_FAILURES; attempt += 1) { - const previous = db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY) + const previous = createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY) db.failDispatch(previous.id, `attempt ${attempt}`, { workerProcessExited: true }) } - const dispatch = db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY) + const dispatch = createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY) runtime.setOrchestrationDb(db as never) runtime.onPtyExit(WORKER_PTY_ID, 137) @@ -414,7 +420,7 @@ describe('STA-4604 worker PTY exit escalation reaches the coordinator', () => { coordinatorPaneKey: COORDINATOR_PANE_KEY }) const task = db.createTask({ spec: 'work outliving its coordinator', runId: run.id }) - db.createDispatchContext(task.id, workerHandle, WORKER_PANE_KEY) + createRootDispatch(db, task.id, workerHandle, WORKER_PANE_KEY) // Rebinding the pane to a newer Run clears the old Run's coordinator_handle. db.createRun({ objective: 'newer run on the same coordinator pane', diff --git a/src/main/runtime/orchestration/nested-worker-depth-migration.test.ts b/src/main/runtime/orchestration/nested-worker-depth-migration.test.ts new file mode 100644 index 00000000000..fa955b7cf08 --- /dev/null +++ b/src/main/runtime/orchestration/nested-worker-depth-migration.test.ts @@ -0,0 +1,114 @@ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import Database from '../../sqlite/sync-database' +import { OrchestrationDb } from './db' +import { resolveOrchestrationMigrationStartVersion } from './orchestration-schema-version-skew' +import { SCHEMA_VERSION } from './db/contract-constants' + +/** + * Backfilling to 1 rather than 0 is the whole point: every pre-v30 row belongs to + * a worker that was already dispatched, so reading it as a root coordinator would + * hand every in-flight worker a free generation of sub-workers at upgrade. + */ +describe('nested worker depth migration (v30)', () => { + let db: OrchestrationDb | undefined + let tempDir: string | undefined + + afterEach(() => { + db?.close() + db = undefined + if (tempDir) { + rmSync(tempDir, { recursive: true, force: true }) + tempDir = undefined + } + }) + + function createV29Database(): string { + tempDir = mkdtempSync(join(tmpdir(), 'orca-nested-depth-migration-')) + const dbPath = join(tempDir, 'orchestration.db') + const fresh = new OrchestrationDb(dbPath) + fresh.close() + + const oldDb = new Database(dbPath) + oldDb.exec('ALTER TABLE dispatch_contexts DROP COLUMN depth') + oldDb.exec('ALTER TABLE remote_dispatch_attachments DROP COLUMN depth') + oldDb.pragma('user_version = 29') + oldDb + .prepare( + `INSERT INTO dispatch_contexts (id, run_id, task_id, contract_version, status) + VALUES ('ctx_inflight', 'run_legacy', 'task_legacy', 1, 'dispatched')` + ) + .run() + oldDb + .prepare( + `INSERT INTO remote_dispatch_attachments + (dispatch_id, task_id, home_peer_fingerprint, protocol_version, runtime_epoch, state) + VALUES ('ctx_remote', 'task_remote', 'peer', 1, 'epoch', 'ready')` + ) + .run() + oldDb.close() + return dbPath + } + + it('backfills in-flight rows to depth 1, not 0', () => { + const dbPath = createV29Database() + db = new OrchestrationDb(dbPath) + const sqlite = (db as unknown as { db: Database.Database }).db + + expect(sqlite.pragma('user_version', { simple: true })).toBe(SCHEMA_VERSION) + expect( + sqlite.prepare("SELECT depth FROM dispatch_contexts WHERE id = 'ctx_inflight'").get() + ).toEqual({ depth: 1 }) + expect( + sqlite + .prepare("SELECT depth FROM remote_dispatch_attachments WHERE dispatch_id = 'ctx_remote'") + .get() + ).toEqual({ depth: 1 }) + }) + + it('leaves an upgraded in-flight worker unable to spawn at the default cap', () => { + const dbPath = createV29Database() + db = new OrchestrationDb(dbPath) + const sqlite = (db as unknown as { db: Database.Database }).db + sqlite + .prepare( + "UPDATE dispatch_contexts SET assignee_handle = 'term_upgraded' WHERE id = 'ctx_inflight'" + ) + .run() + + const task = db.createTask({ spec: 'post-upgrade nesting attempt' }) + expect(() => + db!.createDispatchContext({ + taskId: task.id, + assigneeHandle: 'term_sub', + creator: { kind: 'terminal', handle: 'term_upgraded' }, + maxDepth: 1 + }) + ).toThrow(/depth 2 \(max 1\)/) + }) + + it('does not mistake a real v29 database for a broken v30 one', () => { + // An unconditional column check here would report the schema incomplete and + // replay migrations from v6 instead of starting at 29. + const dbPath = createV29Database() + const oldDb = new Database(dbPath) + expect(resolveOrchestrationMigrationStartVersion(oldDb, 29, SCHEMA_VERSION)).toBe(29) + oldDb.close() + }) + + it('widens the attachment pane indexes to the potentially-live states', () => { + const dbPath = createV29Database() + db = new OrchestrationDb(dbPath) + const sqlite = (db as unknown as { db: Database.Database }).db + const sql = sqlite + .prepare( + "SELECT sql FROM sqlite_master WHERE type = 'index' AND name = 'idx_remote_dispatch_attachments_active_pane'" + ) + .get() as { sql: string } + for (const state of ['start_unknown', 'stopping', 'stop_unknown']) { + expect(sql.sql).toContain(state) + } + }) +}) diff --git a/src/main/runtime/orchestration/orchestration-adopted-run-binding.test.ts b/src/main/runtime/orchestration/orchestration-adopted-run-binding.test.ts index db6a63ef790..800a7511451 100644 --- a/src/main/runtime/orchestration/orchestration-adopted-run-binding.test.ts +++ b/src/main/runtime/orchestration/orchestration-adopted-run-binding.test.ts @@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest' import type Database from '../../sqlite/sync-database' import SyncDatabase from '../../sqlite/sync-database' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' const LEGACY_COORDINATOR_HANDLE = 'term_legacy_coord' const LEGACY_COORDINATOR_PANE = 'tab_coord:44444444-4444-4444-8444-444444444444' @@ -49,7 +50,7 @@ function createAdoptedFixture(options: { settleWork: boolean }): AdoptedFixture spec: 'legacy assignment', createdByTerminalHandle: LEGACY_COORDINATOR_HANDLE }) - const dispatch = before.createDispatchContext(task.id, LEGACY_WORKER_HANDLE, LEGACY_WORKER_PANE) + const dispatch = createRootDispatch(before, task.id, LEGACY_WORKER_HANDLE, LEGACY_WORKER_PANE) const recovery = before.insertMessage({ from: LEGACY_WORKER_HANDLE, to: LEGACY_COORDINATOR_HANDLE, diff --git a/src/main/runtime/orchestration/orchestration-creator-authority-performance.test.ts b/src/main/runtime/orchestration/orchestration-creator-authority-performance.test.ts index af67cf49ba9..df7f3592355 100644 --- a/src/main/runtime/orchestration/orchestration-creator-authority-performance.test.ts +++ b/src/main/runtime/orchestration/orchestration-creator-authority-performance.test.ts @@ -1,6 +1,7 @@ import { afterEach, describe, expect, it } from 'vitest' import type Database from '../../sqlite/sync-database' import { DISPATCH_CONTEXT_CLAIM_SQL, OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' const CREATOR_PANE = 'tab-creator:11111111-1111-4111-8111-111111111111' const CREATOR_PROCESS = 'pty-creator:incarnation-a' @@ -84,13 +85,7 @@ describe('creator authority lookup performance', () => { coordinatorPaneKey: 'tab-coordinator:22222222-2222-4222-8222-222222222222' }) const creatorTask = db.createTask({ spec: 'creator', runId: run.id }) - db.createDispatchContext( - creatorTask.id, - 'term-creator', - CREATOR_PANE, - undefined, - CREATOR_PROCESS - ) + createRootDispatch(db, creatorTask.id, 'term-creator', CREATOR_PANE, undefined, CREATOR_PROCESS) const workerTask = db.createTask({ spec: 'worker', runId: run.id, @@ -145,7 +140,8 @@ describe('creator authority lookup performance', () => { ) .run(retainedDispatchCount, run.id) const creatorTask = db.createTask({ spec: 'creator', runId: run.id }) - const creatorDispatch = db.createDispatchContext( + const creatorDispatch = createRootDispatch( + db, creatorTask.id, 'term-creator', CREATOR_PANE, @@ -173,9 +169,14 @@ describe('creator authority lookup performance', () => { const elapsedMs = performance.now() - startedAt const competingTask = db.createTask({ spec: 'competing creator', runId: run.id }) - expect(() => db!.createDispatchContext(competingTask.id, 'term-creator')).toThrow( - `Terminal term-creator already has an active dispatch (${creatorDispatch.id}` - ) + expect(() => + db!.createDispatchContext({ + taskId: competingTask.id, + assigneeHandle: 'term-creator', + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER + }) + ).toThrow(`Terminal term-creator already has an active dispatch (${creatorDispatch.id}`) expect(elapsedMs).toBeLessThan(200) } ) diff --git a/src/main/runtime/orchestration/orchestration-db-retention-pagination.test.ts b/src/main/runtime/orchestration/orchestration-db-retention-pagination.test.ts index d95524d17fa..38eeca64337 100644 --- a/src/main/runtime/orchestration/orchestration-db-retention-pagination.test.ts +++ b/src/main/runtime/orchestration/orchestration-db-retention-pagination.test.ts @@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest' import Database from '../../sqlite/sync-database' import { OrchestrationDb } from './db' import { SCHEMA_VERSION } from './db/contract-constants' +import { createRootDispatch } from './db/root-dispatch-test-fixture' const MUTATION_RECEIPT_MAX_ROWS = 10_000 @@ -155,6 +156,8 @@ describe('OrchestrationDb bounded mutation receipts', () => { expect(() => db!.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, mutationReceipt: { @@ -224,7 +227,7 @@ describe('OrchestrationDb dispatch assignee index migration', () => { const dbPath = join(tempDir, 'orchestration.db') db = new OrchestrationDb(dbPath) const task = db.createTask({ spec: 'indexed lookup' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') db.close() db = undefined @@ -299,7 +302,7 @@ describe('OrchestrationDb dispatch assignee index migration', () => { createdByProcessIncarnation: 'pty_creator:incarnation-a', createdByRunGeneration: run.consumer_generation }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') db.close() db = undefined diff --git a/src/main/runtime/orchestration/orchestration-legacy-storage-db.test.ts b/src/main/runtime/orchestration/orchestration-legacy-storage-db.test.ts index 6efdf16f1ff..c23c966f849 100644 --- a/src/main/runtime/orchestration/orchestration-legacy-storage-db.test.ts +++ b/src/main/runtime/orchestration/orchestration-legacy-storage-db.test.ts @@ -157,7 +157,13 @@ describe('OrchestrationDb legacy contract storage', () => { createdByTerminalHandle: 'term_legacy_coord' }) - const dispatch = db!.createDispatchContext(task.id, 'term_legacy_coord', 'tab_mixed:leaf_mixed') + const dispatch = db!.createDispatchContext({ + taskId: task.id, + assigneeHandle: 'term_legacy_coord', + assigneePaneKey: 'tab_mixed:leaf_mixed', + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER + }) expect(dispatch.contract_version).toBe(CURRENT_CONTRACT_VERSION) expect(db!.getRunMailboxOwnerIdsForHandle('term_legacy_coord')).toEqual([]) @@ -749,7 +755,12 @@ describe('OrchestrationDb legacy contract storage', () => { ).toThrow(/different answer/) const currentTask = db!.createTask({ runId: state.adoptedRunId, spec: 'current retry' }) - const currentDispatch = db!.createDispatchContext(currentTask.id, 'term_current_retry') + const currentDispatch = db!.createDispatchContext({ + taskId: currentTask.id, + assigneeHandle: 'term_current_retry', + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER + }) const currentQuestion = db!.createQuestion({ runId: state.adoptedRunId, dispatchId: currentDispatch.id, diff --git a/src/main/runtime/orchestration/orchestration-legacy-storage-test-fixture.ts b/src/main/runtime/orchestration/orchestration-legacy-storage-test-fixture.ts index d28498e9c2f..9ae0df316b3 100644 --- a/src/main/runtime/orchestration/orchestration-legacy-storage-test-fixture.ts +++ b/src/main/runtime/orchestration/orchestration-legacy-storage-test-fixture.ts @@ -3,6 +3,7 @@ import { tmpdir } from 'node:os' import { join } from 'node:path' import Database from '../../sqlite/sync-database' import { LEGACY_RUN_ID, OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' export type LegacyStorageCutoverFixture = { dbPath: string @@ -38,7 +39,8 @@ export function createLegacyStorageCutoverFixture(): { coordinatorHandle: 'term_unrelated_coord', coordinatorPaneKey: 'tab_unrelated:55555555-5555-4555-8555-555555555555' }) - const currentDispatch = first.createDispatchContext( + const currentDispatch = createRootDispatch( + first, currentTask.id, 'term_current_worker', 'tab_current:22222222-2222-4222-9222-222222222222', @@ -55,7 +57,8 @@ export function createLegacyStorageCutoverFixture(): { spec: 'legacy', createdByTerminalHandle: 'term_legacy_coord' }) - first.createDispatchContext( + createRootDispatch( + first, legacyTask.id, 'term_legacy_worker', 'tab_legacy:33333333-3333-4333-8333-333333333333' @@ -65,7 +68,8 @@ export function createLegacyStorageCutoverFixture(): { question: 'Retained gate?' }) first.resolveGate(legacyGate.id, 'continue') - const retryDispatch = first.createDispatchContext( + const retryDispatch = createRootDispatch( + first, legacyTask.id, 'term_legacy_worker', 'tab_legacy:33333333-3333-4333-8333-333333333333' diff --git a/src/main/runtime/orchestration/orchestration-mutation-question-db.test.ts b/src/main/runtime/orchestration/orchestration-mutation-question-db.test.ts index a6039c381f9..5a26448bd98 100644 --- a/src/main/runtime/orchestration/orchestration-mutation-question-db.test.ts +++ b/src/main/runtime/orchestration/orchestration-mutation-question-db.test.ts @@ -1,5 +1,6 @@ import { afterEach, describe, expect, it } from 'vitest' import { OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('OrchestrationDb mutation and question state', () => { let db: OrchestrationDb | undefined @@ -93,7 +94,7 @@ describe('OrchestrationDb mutation and question state', () => { coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111' }) const task = d.createTask({ spec: 'ask', runId: run.id }) - const dispatch = d.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(d, task.id, 'term_worker') const created = d.createQuestion({ runId: run.id, dispatchId: dispatch.id, @@ -144,7 +145,7 @@ describe('OrchestrationDb mutation and question state', () => { coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111' }) const task = d.createTask({ spec: 'ask', runId: run.id }) - const dispatch = d.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(d, task.id, 'term_worker') const created = d.createQuestion({ runId: run.id, dispatchId: dispatch.id, diff --git a/src/main/runtime/orchestration/orchestration-reset-db.test.ts b/src/main/runtime/orchestration/orchestration-reset-db.test.ts index 0ec8d3bf193..4b3e033a019 100644 --- a/src/main/runtime/orchestration/orchestration-reset-db.test.ts +++ b/src/main/runtime/orchestration/orchestration-reset-db.test.ts @@ -15,6 +15,8 @@ describe('OrchestrationDb reset scopes', () => { }) const task = db.createTask({ spec: 'work', runId: run.id }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: { worktree: 'current' }, runtimeEpoch: 'runtime_1', diff --git a/src/main/runtime/orchestration/orchestration-run-delivery-db.test.ts b/src/main/runtime/orchestration/orchestration-run-delivery-db.test.ts index 1d7a3ce45d5..fdaa5e92a80 100644 --- a/src/main/runtime/orchestration/orchestration-run-delivery-db.test.ts +++ b/src/main/runtime/orchestration/orchestration-run-delivery-db.test.ts @@ -3,6 +3,7 @@ import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterEach, describe, expect, it } from 'vitest' import { LEGACY_RUN_ID, OrchestrationDb } from './db' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('OrchestrationDb Run state', () => { let db: OrchestrationDb | undefined @@ -172,7 +173,7 @@ describe('OrchestrationDb Run state', () => { coordinatorPaneKey: 'tab_other:22222222-2222-4222-9222-222222222222' }) const task = d.createTask({ spec: 'work', runId: runB.id }) - const dispatch = d.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(d, task.id, 'term_worker') const mismatched = d.insertMessage({ from: 'worker', to: `dispatch:${dispatch.id}`, @@ -285,7 +286,7 @@ describe('OrchestrationDb Run state', () => { coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111' }) const task = d.createTask({ spec: 'work', runId: run.id }) - const dispatch = d.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(d, task.id, 'term_worker') const message = d.insertMessage({ runId: run.id, from: 'term_worker', diff --git a/src/main/runtime/orchestration/orchestration-schema-version-skew.ts b/src/main/runtime/orchestration/orchestration-schema-version-skew.ts index 8fb3db36777..95ecbe6b034 100644 --- a/src/main/runtime/orchestration/orchestration-schema-version-skew.ts +++ b/src/main/runtime/orchestration/orchestration-schema-version-skew.ts @@ -26,7 +26,9 @@ const POST_V6_COLUMNS = [ ] as const const VERSIONED_POST_V6_COLUMNS = [ - { version: 27, table: 'federated_dispatches', column: 'to_home_acknowledged_sequence' } + { version: 27, table: 'federated_dispatches', column: 'to_home_acknowledged_sequence' }, + { version: 30, table: 'dispatch_contexts', column: 'depth' }, + { version: 30, table: 'remote_dispatch_attachments', column: 'depth' } ] as const const POST_V6_INDEXES = [ diff --git a/src/main/runtime/orchestration/orchestration-version-skew-migration.test.ts b/src/main/runtime/orchestration/orchestration-version-skew-migration.test.ts index 763301bde22..6cb58f00ba4 100644 --- a/src/main/runtime/orchestration/orchestration-version-skew-migration.test.ts +++ b/src/main/runtime/orchestration/orchestration-version-skew-migration.test.ts @@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest' import Database from '../../sqlite/sync-database' import { LEGACY_CONTRACT_VERSION, LEGACY_RUN_ID, OrchestrationDb } from './db' import { resolveOrchestrationMigrationStartVersion } from './orchestration-schema-version-skew' +import { createRootDispatch } from './db/root-dispatch-test-fixture' describe('OrchestrationDb version-skew migration', () => { let db: OrchestrationDb | undefined @@ -158,7 +159,7 @@ describe('OrchestrationDb version-skew migration', () => { coordinatorPaneKey: 'tab_v2:leaf_coord' }) const task = db.createTask({ spec: 'reply with ack', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_worker_v2', 'tab_v2:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker_v2', 'tab_v2:leaf_worker') const question = db.createQuestion({ runId: run.id, dispatchId: dispatch.id, diff --git a/src/main/runtime/orchestration/orchestration-worker-dispatch-db.test.ts b/src/main/runtime/orchestration/orchestration-worker-dispatch-db.test.ts index 5e147f51176..9e4a2768a1d 100644 --- a/src/main/runtime/orchestration/orchestration-worker-dispatch-db.test.ts +++ b/src/main/runtime/orchestration/orchestration-worker-dispatch-db.test.ts @@ -17,6 +17,8 @@ describe('OrchestrationDb worker Dispatch state', () => { const d = createDb() const task = d.createTask({ spec: 'worker' }) const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: { topology: 'current', agent: 'codex' } }) @@ -57,7 +59,12 @@ describe('OrchestrationDb worker Dispatch state', () => { it('retains an active supervised worker terminal', () => { const d = createDb() const task = d.createTask({ spec: 'retain active worker' }) - const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: 'term_worker', @@ -81,6 +88,8 @@ describe('OrchestrationDb worker Dispatch state', () => { const d = createDb() const task = d.createTask({ spec: 'recover missing worker' }) const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: { topology: 'current', agent: 'codex' } }) @@ -125,6 +134,8 @@ describe('OrchestrationDb worker Dispatch state', () => { } const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: { topology: 'current' }, mutationReceipt @@ -146,6 +157,8 @@ describe('OrchestrationDb worker Dispatch state', () => { expect(() => d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: 'task_missing', startOptions: {}, mutationReceipt: { @@ -162,7 +175,12 @@ describe('OrchestrationDb worker Dispatch state', () => { it('fails a composed start without losing residual resource receipts', () => { const d = createDb() const task = d.createTask({ spec: 'worker' }) - const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.recordWorkerStage({ dispatchId: started.dispatch.id, stage: 'terminal_created', @@ -182,9 +200,16 @@ describe('OrchestrationDb worker Dispatch state', () => { it('allows retry only from the Task current terminal Dispatch', () => { const d = createDb() const task = d.createTask({ spec: 'retry current' }) - const first = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const first = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.failWorkerStart(first.dispatch.id, 'agent_readiness', 'first failed') const second = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, retryOf: first.dispatch.id, startOptions: {} @@ -193,6 +218,8 @@ describe('OrchestrationDb worker Dispatch state', () => { expect(() => d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, retryOf: first.dispatch.id, startOptions: {} @@ -200,6 +227,8 @@ describe('OrchestrationDb worker Dispatch state', () => { ).toThrow('cannot retry') expect( d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, retryOf: second.dispatch.id, startOptions: {} @@ -210,9 +239,16 @@ describe('OrchestrationDb worker Dispatch state', () => { it('treats abandon of a superseded Dispatch as a no-op', () => { const d = createDb() const task = d.createTask({ spec: 'stale abandon' }) - const first = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const first = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.failWorkerStart(first.dispatch.id, 'agent_readiness', 'first failed') const second = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, retryOf: first.dispatch.id, startOptions: {} @@ -248,7 +284,12 @@ describe('OrchestrationDb worker Dispatch state', () => { it('lets the stop fence win before a late worker completion', () => { const d = createDb() const task = d.createTask({ spec: 'race' }) - const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: 'term_worker', @@ -276,7 +317,12 @@ describe('OrchestrationDb worker Dispatch state', () => { it('allows explicit stop recovery from uncertain local and remote starts', () => { const d = createDb() const task = d.createTask({ spec: 'uncertain local start' }) - const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.markWorkerStartUnknown(started.dispatch.id, 'agent_readiness', 'connection lost') expect(d.beginWorkerStop(started.dispatch.id, 'runtime_test')).toMatchObject({ @@ -359,7 +405,12 @@ describe('OrchestrationDb worker Dispatch state', () => { it('returns already-settled when completion wins before stop', () => { const d = createDb() const task = d.createTask({ spec: 'race' }) - const started = d.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = d.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) d.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: 'term_worker', diff --git a/src/main/runtime/orchestration/types.ts b/src/main/runtime/orchestration/types.ts index 7b73f227a0a..b34f69e3c22 100644 --- a/src/main/runtime/orchestration/types.ts +++ b/src/main/runtime/orchestration/types.ts @@ -205,6 +205,8 @@ export type RemoteDispatchAttachmentRow = { effects: string residual_resources: string to_worker_imported_sequence: number + /** Nesting depth propagated from the Run home; 1 when an old client omitted it. */ + depth: number last_error: string | null created_at: string updated_at: string @@ -278,6 +280,8 @@ export type DispatchContextRow = { /** Why the dispatch ended, when Orca could establish it — `operator_close`, * `signaled`, `exited`, `unknown`. Null on rows written before STA-4603. */ termination_reason: TerminalExitCause['kind'] | null + /** Nesting depth; a root coordinator's worker is 1. Never 0 on a persisted row. */ + depth: number dispatched_at: string | null completed_at: string | null created_at: string diff --git a/src/main/runtime/rpc/methods/orchestration-ask.test.ts b/src/main/runtime/rpc/methods/orchestration-ask.test.ts index deaaf12d438..18c1e9f415d 100644 --- a/src/main/runtime/rpc/methods/orchestration-ask.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-ask.test.ts @@ -4,6 +4,7 @@ import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness' import type { OrchestrationDb } from '../../orchestration/db' import type { OrcaRuntimeService } from '../../orca-runtime' import { ORCHESTRATION_ASK_MAX_TIMEOUT_MS } from '../../../../shared/orchestration-ask-timeout' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' describe('orchestration RPC methods', () => { const h = createOrchestrationRpcHarness() @@ -59,7 +60,7 @@ describe('orchestration RPC methods', () => { it('records one idempotent answer from the current Run consumer', async () => { setup() const task = db.createTask({ spec: 'question work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const created = db.createQuestion({ runId: activeRunId!, dispatchId: dispatch.id, @@ -103,7 +104,7 @@ describe('orchestration RPC methods', () => { describe('orchestration.ask', () => { function createAskingDispatch(handle = 'term_worker') { const task = db.createTask({ spec: 'question work' }) - const dispatch = db.createDispatchContext(task.id, handle) + const dispatch = createRootDispatch(db, task.id, handle) return { task, dispatch } } diff --git a/src/main/runtime/rpc/methods/orchestration-check.test.ts b/src/main/runtime/rpc/methods/orchestration-check.test.ts index b933bbb0831..331e7448734 100644 --- a/src/main/runtime/rpc/methods/orchestration-check.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-check.test.ts @@ -4,6 +4,7 @@ import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness' import type { OrchestrationDb } from '../../orchestration/db' import { reconcileLifecycleMessage } from '../../orchestration/lifecycle-reconciliation' import type { OrcaRuntimeService } from '../../orca-runtime' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' describe('orchestration RPC methods', () => { const h = createOrchestrationRpcHarness() @@ -28,7 +29,7 @@ describe('orchestration RPC methods', () => { describe('orchestration.check', () => { function createDispatchedTask(assigneeHandle = 'term_worker', assigneePaneKey?: string) { const task = db.createTask({ spec: 'manual check work' }) - const dispatch = db.createDispatchContext(task.id, assigneeHandle, assigneePaneKey) + const dispatch = createRootDispatch(db, task.id, assigneeHandle, assigneePaneKey) return { task, dispatch } } @@ -432,9 +433,9 @@ describe('orchestration RPC methods', () => { it('does not complete worker_done for a stale inactive dispatch', async () => { setup() const task = db.createTask({ spec: 'retry-sensitive work' }) - const staleDispatch = db.createDispatchContext(task.id, 'term_old') + const staleDispatch = createRootDispatch(db, task.id, 'term_old') db.failDispatch(staleDispatch.id, 'retry elsewhere') - const activeDispatch = db.createDispatchContext(task.id, 'term_current') + const activeDispatch = createRootDispatch(db, task.id, 'term_current') insertWorkerDone({ from: 'term_old', taskId: task.id, diff --git a/src/main/runtime/rpc/methods/orchestration-composed-workers.test.ts b/src/main/runtime/rpc/methods/orchestration-composed-workers.test.ts index 7080f927995..10dbfa97a53 100644 --- a/src/main/runtime/rpc/methods/orchestration-composed-workers.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-composed-workers.test.ts @@ -66,6 +66,65 @@ describe('orchestration RPC methods', () => { }) } + it('rejects a declared caller that disagrees with complete attested evidence', async () => { + setup() + mockCurrentWorkerStart() + vi.mocked(runtime.getTerminalPaneKey).mockImplementation((handle) => + handle === 'term_coord' || handle === 'term_other' + ? coordinatorPaneKey + : handle === 'term_worker' + ? 'tab_worker:leaf_worker' + : null + ) + const attestedEvidence = { + terminalHandle: 'term_attested', + paneKey: 'tab_attested:leaf_attested', + launchToken: 'attested-launch-token' + } as const + vi.spyOn(runtime, 'verifyOrchestrationCompatibilityCaller').mockReturnValue({ + terminalHandle: attestedEvidence.terminalHandle, + paneKey: attestedEvidence.paneKey, + processIncarnation: 'runtime_test:attested:1', + launchTokenHash: 'attested-launch-token-hash', + hostScope: { kind: 'local', hostId: 'local' } + }) + ctx = { ...ctx, orchestrationCompatibilityEvidence: attestedEvidence } + const task = db.createTask({ spec: 'mismatched caller' }) + + await expect( + call('orchestration.workerStart', { + task: task.id, + from: 'term_other', + agent: 'codex' + }) + ).rejects.toMatchObject({ code: 'consumer_fenced' }) + expect(db.getDispatchContext(task.id)).toBeUndefined() + }) + + it('deliberately permits present but unverifiable restored-terminal evidence', async () => { + setup() + mockCurrentWorkerStart() + // Restored/adopted terminals have no launch token, so verification returns null; this + // fail-open is deliberate compatibility behavior, not an oversight. + const task = db.createTask({ spec: 'restored caller limitation' }) + ctx = { + ...ctx, + orchestrationCompatibilityEvidence: { + terminalHandle: 'term_worker', + paneKey: 'tab_worker:leaf_worker' + } + } + + const result = (await call('orchestration.workerStart', { + task: task.id, + from: 'term_coord', + agent: 'codex' + })) as { state: string } + + expect(result.state).toBe('ready') + expect(db.getDispatchContext(task.id)).toBeDefined() + }) + it('starts a fresh agent in the coordinator current worktree', async () => { setup() mockCurrentWorkerStart() diff --git a/src/main/runtime/rpc/methods/orchestration-dispatch-creator.ts b/src/main/runtime/rpc/methods/orchestration-dispatch-creator.ts new file mode 100644 index 00000000000..4da46b6eeb0 --- /dev/null +++ b/src/main/runtime/rpc/methods/orchestration-dispatch-creator.ts @@ -0,0 +1,27 @@ +import type { DispatchCreator } from '../../orchestration/db/dispatch-depth' +import type { OrcaRuntimeService } from '../../orca-runtime' + +/** + * Identify a CLI caller for nesting-depth purposes. + * + * Pane key and process incarnation come from the runtime's dispatch authority + * rather than the caller's params: remote attachment matching needs the exact + * incarnation, and a caller cannot be trusted to report its own. + */ +export function resolveDispatchCreator( + runtime: OrcaRuntimeService, + callerHandle: string | undefined +): DispatchCreator { + if (!callerHandle) { + // No declared caller means no resolvable parent. Depth 0 is the same answer + // the pre-existing Run-binding check already gives this case. + return { kind: 'system' } + } + const authority = runtime.getOrchestrationDispatchAuthority?.(callerHandle) + return { + kind: 'terminal', + handle: callerHandle, + paneKey: authority?.paneKey ?? runtime.getTerminalPaneKey(callerHandle) ?? undefined, + processIncarnation: authority?.processIncarnation ?? undefined + } +} diff --git a/src/main/runtime/rpc/methods/orchestration-federated-worker-start.ts b/src/main/runtime/rpc/methods/orchestration-federated-worker-start.ts index 561c35603f2..28c6530d2a0 100644 --- a/src/main/runtime/rpc/methods/orchestration-federated-worker-start.ts +++ b/src/main/runtime/rpc/methods/orchestration-federated-worker-start.ts @@ -21,6 +21,7 @@ import { type OrchestrationWorkerLaunchReceipt } from './orchestration-worker-launch-preferences' import { validateFederatedWorkerStartPlacement } from './orchestration-worker-start-validation' +import { resolveDispatchCreator } from './orchestration-dispatch-creator' export async function startFederatedWorker(args: { params: WorkerStartInput @@ -97,6 +98,8 @@ export async function startFederatedWorker(args: { const setupDecision = createsWorktree ? (params.setup ?? 'run') : 'not_applicable' const started = db.createStartingWorkerDispatch({ + creator: resolveDispatchCreator(runtime, params.from), + maxDepth: runtime.getNestedWorkerMaxDepth(), taskId: task.id, retryOf: params.retryOf, startOptions: { @@ -135,6 +138,9 @@ export async function startFederatedWorker(args: { dispatchId: started.dispatch.id, taskId: task.id, taskSpec: task.spec, + // Carry the home dispatch depth across the federation boundary so a + // remote worker cannot be mistaken for a root when it dispatches again. + depth: started.dispatch.depth, protocolVersion: federationProtocolVersion, worktree, name: params.name, diff --git a/src/main/runtime/rpc/methods/orchestration-federation-agent-launch.test.ts b/src/main/runtime/rpc/methods/orchestration-federation-agent-launch.test.ts index 5009f6ba027..4948e196c66 100644 --- a/src/main/runtime/rpc/methods/orchestration-federation-agent-launch.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-federation-agent-launch.test.ts @@ -59,6 +59,7 @@ describe('federated worker agent launch', () => { dispatchId: 'ctx_remote', taskId: 'task_remote', taskSpec: 'remote cursor worker', + depth: 2, protocolVersion: 3, worktree: 'folder:remote-workspace', agent: 'cursor', @@ -90,6 +91,7 @@ describe('federated worker agent launch', () => { effective: { agent: 'cursor', model: 'gpt-5.3-codex', effort: 'high' } } }) + expect(db.getRemoteDispatchAttachment('ctx_remote')?.depth).toBe(2) expect(createTerminal).toHaveBeenCalledWith( 'id:folder:remote-workspace', expect.objectContaining({ diff --git a/src/main/runtime/rpc/methods/orchestration-federation-control-mail.test.ts b/src/main/runtime/rpc/methods/orchestration-federation-control-mail.test.ts index 43bc0bc96ff..401318d82c0 100644 --- a/src/main/runtime/rpc/methods/orchestration-federation-control-mail.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-federation-control-mail.test.ts @@ -91,6 +91,8 @@ describe('orchestration federation control mail', () => { runId = run.id const task = homeDb.createTask({ spec: 'Wait for coordinator guidance', runId }) const started = homeDb.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, federation: { diff --git a/src/main/runtime/rpc/methods/orchestration-federation-setup.test.ts b/src/main/runtime/rpc/methods/orchestration-federation-setup.test.ts index 6eec52cd93f..c1d91c18479 100644 --- a/src/main/runtime/rpc/methods/orchestration-federation-setup.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-federation-setup.test.ts @@ -118,6 +118,8 @@ describe('orchestration federated setup evidence', () => { }) const task = db.createTask({ spec: 'remote setup', runId: run.id }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, runtimeEpoch: runtime.getRuntimeId(), diff --git a/src/main/runtime/rpc/methods/orchestration-federation-start-schema.ts b/src/main/runtime/rpc/methods/orchestration-federation-start-schema.ts index 9c03345ef88..61bf385b782 100644 --- a/src/main/runtime/rpc/methods/orchestration-federation-start-schema.ts +++ b/src/main/runtime/rpc/methods/orchestration-federation-start-schema.ts @@ -6,6 +6,8 @@ export const FederationAttachStartParams = z.object({ dispatchId: requiredString('Missing Dispatch ID'), taskId: requiredString('Missing Task ID'), taskSpec: requiredString('Missing Task spec'), + /** Depth stamped by the Run home; omitted by older clients and defaults to 1. */ + depth: z.number().int().min(1).optional(), protocolVersion: z.union([z.literal(1), z.literal(2), z.literal(3)]), worktree: requiredString('Missing remote worktree selector'), name: OptionalString, diff --git a/src/main/runtime/rpc/methods/orchestration-federation.ts b/src/main/runtime/rpc/methods/orchestration-federation.ts index 83cbbfe385f..da4e29d4fc0 100644 --- a/src/main/runtime/rpc/methods/orchestration-federation.ts +++ b/src/main/runtime/rpc/methods/orchestration-federation.ts @@ -57,6 +57,7 @@ export const ORCHESTRATION_FEDERATION_ATTACH_METHODS: RpcMethod[] = [ homePeerFingerprint: orchestrationMutation.callerFingerprint, protocolVersion: params.protocolVersion, runtimeEpoch: runtime.getRuntimeId(), + depth: params.depth, mutationReceipt: orchestrationMutation }) const effects: FederationEffect[] = [] diff --git a/src/main/runtime/rpc/methods/orchestration-manual-dispatch-observation.test.ts b/src/main/runtime/rpc/methods/orchestration-manual-dispatch-observation.test.ts index 47fbc8067cc..c99fcb1c328 100644 --- a/src/main/runtime/rpc/methods/orchestration-manual-dispatch-observation.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-manual-dispatch-observation.test.ts @@ -2,6 +2,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { OrcaRuntimeService } from '../../orca-runtime' import { OrchestrationDb } from '../../orchestration/db' import { ORCHESTRATION_METHODS } from './orchestration' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' describe('manual Dispatch observation', () => { let db: OrchestrationDb | undefined @@ -111,7 +112,8 @@ describe('manual Dispatch observation', () => { coordinatorPaneKey: 'tab_coord:leaf_coord' }) const task = db.createTask({ spec: 'injected lane', runId: run.id }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, 'term_worker', 'tab_worker:leaf_worker', @@ -207,6 +209,39 @@ describe('manual Dispatch observation', () => { }) }) + it('lists an unsupervised context-only dispatch even when process identity is absent', async () => { + db = new OrchestrationDb(':memory:') + const runtime = new OrcaRuntimeService() + runtime.setOrchestrationDb(db) + const run = db.createRun({ + objective: 'context-only listing', + coordinatorHandle: 'term_coord', + coordinatorPaneKey: 'tab_coord:leaf_coord' + }) + const task = db.createTask({ spec: 'operator lane', runId: run.id }) + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') + + const workerListMethod = ORCHESTRATION_METHODS.find( + (candidate) => candidate.name === 'orchestration.workerList' + ) + if (!workerListMethod) { + throw new Error('Missing method orchestration.workerList') + } + const result = (await workerListMethod.handler( + workerListMethod.params?.parse({ run: run.id }), + { runtime } + )) as { workers: { dispatchId: string; workerState: string; terminalState: string | null }[] } + + expect(result.workers).toEqual([ + expect.objectContaining({ + dispatchId: dispatch.id, + workerState: 'unsupervised', + terminalState: 'retained' + }) + ]) + expect(db.getDispatchContextById(dispatch.id)?.process_incarnation).toBeNull() + }) + it.each([ ['orchestration.workerStop', 'stopped'], ['orchestration.workerAbandon', 'abandoned'] @@ -216,7 +251,8 @@ describe('manual Dispatch observation', () => { runtime.setOrchestrationDb(db) const closeTerminal = vi.spyOn(runtime, 'closeTerminal') const task = db.createTask({ spec: 'operator-owned lane' }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, 'term_worker', 'tab_worker:leaf_worker', diff --git a/src/main/runtime/rpc/methods/orchestration-manual-dispatch-release.test.ts b/src/main/runtime/rpc/methods/orchestration-manual-dispatch-release.test.ts index 6f98b3b8cf2..d684adbfacc 100644 --- a/src/main/runtime/rpc/methods/orchestration-manual-dispatch-release.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-manual-dispatch-release.test.ts @@ -158,6 +158,8 @@ describe('manual Dispatch release', () => { function createSupervisedWorker(): string { const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: createTask('supervised'), startOptions: {} }) diff --git a/src/main/runtime/rpc/methods/orchestration-migration-behavior.test.ts b/src/main/runtime/rpc/methods/orchestration-migration-behavior.test.ts index a8528803bf1..1d562e1ee5a 100644 --- a/src/main/runtime/rpc/methods/orchestration-migration-behavior.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-migration-behavior.test.ts @@ -8,6 +8,7 @@ import { OrchestrationDb } from '../../orchestration/db' import { RpcDispatcher } from '../dispatcher' import { ORCHESTRATION_METHODS } from './orchestration' import { startFederatedWorker } from './orchestration-federated-worker-start' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' describe('orchestration migration behavior', () => { const databases: OrchestrationDb[] = [] @@ -124,7 +125,7 @@ describe('orchestration migration behavior', () => { coordinatorPaneKey: 'tab_coord:leaf_coord' }) const task = db.createTask({ spec: 'legacy worker', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') const dispatcher = new RpcDispatcher({ runtime, methods: ORCHESTRATION_METHODS }) const response = await dispatcher.dispatch({ diff --git a/src/main/runtime/rpc/methods/orchestration-recipient-routing.test.ts b/src/main/runtime/rpc/methods/orchestration-recipient-routing.test.ts index 99a4c238807..41d10266dee 100644 --- a/src/main/runtime/rpc/methods/orchestration-recipient-routing.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-recipient-routing.test.ts @@ -7,6 +7,7 @@ import type { RpcContext, RpcRequest } from '../core' import { RpcDispatcher } from '../dispatcher' import { ORCHESTRATION_METHODS } from './orchestration' import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' type SendWarning = { code: string; recipient: string; message: string } type SendResult = { @@ -170,7 +171,7 @@ describe('orchestration recipient routing oracle', () => { it('normalizes an active Dispatch owner even when no pane is live', async () => { setup() const task = db.createTask({ spec: 'detached worker' }) - const dispatch = db.createDispatchContext(task.id, 'term_detached', 'tab_gone:leaf_gone') + const dispatch = createRootDispatch(db, task.id, 'term_detached', 'tab_gone:leaf_gone') const result = (await call({ from: 'term_coord', @@ -194,7 +195,7 @@ describe('orchestration recipient routing oracle', () => { coordinatorPaneKey: 'tab_foreign:leaf_coord' }) const task = db.createTask({ spec: 'detached foreign worker', runId: foreignRun.id }) - db.createDispatchContext(task.id, 'term_detached_foreign', 'tab_gone:leaf_gone') + createRootDispatch(db, task.id, 'term_detached_foreign', 'tab_gone:leaf_gone') await expect( call({ @@ -211,7 +212,7 @@ describe('orchestration recipient routing oracle', () => { setup() const overlapPane = 'tab_overlap:leaf_overlap' const task = db.createTask({ spec: 'overlapped worker' }) - db.createDispatchContext(task.id, 'term_overlap', overlapPane) + createRootDispatch(db, task.id, 'term_overlap', overlapPane) const recipientRun = db.createRun({ objective: 'Overlapping coordinator', coordinatorHandle: 'term_overlap', diff --git a/src/main/runtime/rpc/methods/orchestration-run-scope.ts b/src/main/runtime/rpc/methods/orchestration-run-scope.ts index 113c135c153..cdf6968bcbc 100644 --- a/src/main/runtime/rpc/methods/orchestration-run-scope.ts +++ b/src/main/runtime/rpc/methods/orchestration-run-scope.ts @@ -2,7 +2,10 @@ import type { OrchestrationCompatibilityEvidence } from '../../../../shared/orch import { orchestrationSkillRecoveryData } from '../../../../shared/orchestration-rpc-contract' import { OrchestrationError } from '../../orchestration/orchestration-error' import type { RunRow } from '../../orchestration/types' -import type { OrcaRuntimeService } from '../../orca-runtime' +import type { + OrcaRuntimeService, + OrchestrationCompatibilityCallerAuthority +} from '../../orca-runtime' export type RunScopeParams = { runId?: string @@ -33,6 +36,49 @@ export function assertCallerHandleMatchesEvidence( } } +export type OrchestrationCallerParams = { + callerTerminalHandle: string + callerEvidence?: OrchestrationCompatibilityEvidence + callerAuthority?: OrchestrationCompatibilityCallerAuthority + /** Preserve legacy callers that treated a missing pane as an ordinary fence. */ + requireStablePane?: boolean + /** + * Skip attestation here because the caller performs it itself — run-use must run + * its legacy-takeover check between pane resolution and attestation. Setting this + * without asserting elsewhere reopens the hole this helper exists to close. + */ + evidenceAssertedByCaller?: boolean +} + +/** Resolve the caller's runtime pane and, by default, attest its declared handle. */ +export function resolveOrchestrationCaller( + runtime: OrcaRuntimeService, + params: OrchestrationCallerParams & { requireStablePane: true } +): string +export function resolveOrchestrationCaller( + runtime: OrcaRuntimeService, + params: OrchestrationCallerParams +): string | null +export function resolveOrchestrationCaller( + runtime: OrcaRuntimeService, + params: OrchestrationCallerParams +): string | null { + if (!params.evidenceAssertedByCaller) { + assertCallerHandleMatchesEvidence(runtime, params.callerTerminalHandle, params.callerEvidence) + } + const paneKey = + params.callerAuthority?.terminalHandle === params.callerTerminalHandle + ? params.callerAuthority.paneKey + : runtime.getTerminalPaneKey(params.callerTerminalHandle) + if (!paneKey && params.requireStablePane) { + throw new OrchestrationError( + 'stable_pane_required', + 'The coordinator terminal has no stable pane identity. Run this command inside a live Orca terminal.' + ) + } + return paneKey ?? null +} + // Why: task and gate mutations must share one Run-binding rule. export function resolveRunScope(runtime: OrcaRuntimeService, params: RunScopeParams): RunRow { const db = runtime.getOrchestrationDb() diff --git a/src/main/runtime/rpc/methods/orchestration-runs.ts b/src/main/runtime/rpc/methods/orchestration-runs.ts index aa902171303..7938bc240f1 100644 --- a/src/main/runtime/rpc/methods/orchestration-runs.ts +++ b/src/main/runtime/rpc/methods/orchestration-runs.ts @@ -2,12 +2,11 @@ import { z } from 'zod' import { defineMethod, type RpcMethod } from '../core' import { OptionalBoolean, OptionalString, requiredString } from '../schemas' import { ORCHESTRATION_RUN_PAGE_LIMIT } from '../../../../shared/orchestration-run-pagination' -import type { - OrcaRuntimeService, - OrchestrationCompatibilityCallerAuthority -} from '../../orca-runtime' import { OrchestrationError } from '../../orchestration/orchestration-error' -import { assertCallerHandleMatchesEvidence } from './orchestration-run-scope' +import { + assertCallerHandleMatchesEvidence, + resolveOrchestrationCaller +} from './orchestration-run-scope' const RunCreateParams = z.object({ objective: requiredString('Missing --objective'), @@ -27,31 +26,16 @@ const RunListParams = z.object({ }) const RunShowParams = z.object({ id: requiredString('Missing --id'), from: OptionalString }) -function requireCallerPane( - runtime: OrcaRuntimeService, - handle: string, - callerAuthority?: OrchestrationCompatibilityCallerAuthority -): string { - const paneKey = - callerAuthority?.terminalHandle === handle - ? callerAuthority.paneKey - : runtime.getTerminalPaneKey(handle) - if (!paneKey) { - throw new OrchestrationError( - 'stable_pane_required', - 'The coordinator terminal has no stable pane identity. Run this command inside a live Orca terminal.' - ) - } - return paneKey -} - export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [ defineMethod({ name: 'orchestration.runCreate', params: RunCreateParams, handler: (params, { orchestrationCompatibilityEvidence, runtime }) => { - assertCallerHandleMatchesEvidence(runtime, params.from, orchestrationCompatibilityEvidence) - const paneKey = requireCallerPane(runtime, params.from) + const paneKey = resolveOrchestrationCaller(runtime, { + callerTerminalHandle: params.from, + callerEvidence: orchestrationCompatibilityEvidence, + requireStablePane: true + }) const db = runtime.getOrchestrationDb() const priorRun = db.getCurrentRunForPane(paneKey) const run = db.createRun({ @@ -78,7 +62,13 @@ export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [ orchestrationCompatibilityCallerAuthority: callerAuthority } ) => { - const paneKey = requireCallerPane(runtime, params.from, callerAuthority) + const paneKey = resolveOrchestrationCaller(runtime, { + callerTerminalHandle: params.from, + callerEvidence: orchestrationCompatibilityEvidence, + callerAuthority, + requireStablePane: true, + evidenceAssertedByCaller: true + }) if ( params.takeoverLegacy && (callerAuthority?.terminalHandle !== params.from || callerAuthority.paneKey !== paneKey) @@ -117,8 +107,11 @@ export const ORCHESTRATION_RUN_METHODS: RpcMethod[] = [ name: 'orchestration.runCurrent', params: RunCurrentParams, handler: (params, { orchestrationCompatibilityEvidence, runtime }) => { - assertCallerHandleMatchesEvidence(runtime, params.from, orchestrationCompatibilityEvidence) - const paneKey = requireCallerPane(runtime, params.from) + const paneKey = resolveOrchestrationCaller(runtime, { + callerTerminalHandle: params.from, + callerEvidence: orchestrationCompatibilityEvidence, + requireStablePane: true + }) return { run: runtime.getOrchestrationDb().getCurrentRunForPane(paneKey) ?? null } } }), diff --git a/src/main/runtime/rpc/methods/orchestration-send-dispatch-authority.test.ts b/src/main/runtime/rpc/methods/orchestration-send-dispatch-authority.test.ts index cbba56c1c2a..7eaa19d9d8f 100644 --- a/src/main/runtime/rpc/methods/orchestration-send-dispatch-authority.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-send-dispatch-authority.test.ts @@ -5,6 +5,7 @@ import type { OrcaRuntimeService } from '../../orca-runtime' import { openDecisionGateFromMessage } from '../../orchestration/coordinator-decision-gates' import { applyEscalationToDispatch } from '../../orchestration/coordinator-escalation-triage' import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' describe('orchestration.send Dispatch authority', () => { const harness = createOrchestrationRpcHarness() @@ -29,7 +30,8 @@ describe('orchestration.send Dispatch authority', () => { async (legacyAuthority) => { setup() const attackerTask = db.createTask({ spec: 'attacker assignment' }) - const attacker = db.createDispatchContext( + const attacker = createRootDispatch( + db, attackerTask.id, 'term_attacker', 'tab_attacker:leaf_attacker', @@ -37,7 +39,7 @@ describe('orchestration.send Dispatch authority', () => { legacyAuthority ? undefined : 'runtime_test:term_attacker:1' ) const victimTask = db.createTask({ spec: 'victim assignment' }) - const victim = db.createDispatchContext(victimTask.id, 'term_victim') + const victim = createRootDispatch(db, victimTask.id, 'term_victim') vi.mocked(runtime.getTerminalPaneKey).mockImplementation((handle) => handle === 'term_attacker' ? 'tab_attacker:leaf_attacker' : harness.coordinatorPaneKey ) @@ -74,7 +76,7 @@ describe('orchestration.send Dispatch authority', () => { it('rejects a caller-spoofed canonical Dispatch sender', async () => { setup() const task = db.createTask({ spec: 'legacy victim assignment' }) - const dispatch = db.createDispatchContext(task.id, 'term_victim') + const dispatch = createRootDispatch(db, task.id, 'term_victim') const result = (await send({ from: `dispatch:${dispatch.id}`, @@ -97,7 +99,7 @@ describe('orchestration.send Dispatch authority', () => { async (type) => { setup() const task = db.createTask({ spec: 'legacy owned assignment' }) - db.createDispatchContext(task.id, 'term_legacy') + createRootDispatch(db, task.id, 'term_legacy') vi.mocked(runtime.getTerminalPaneKey).mockImplementation((handle) => handle === 'term_legacy' ? 'tab_legacy:leaf_legacy' : harness.coordinatorPaneKey ) @@ -123,7 +125,7 @@ describe('orchestration.send Dispatch authority', () => { async (type) => { setup() const task = db.createTask({ spec: 'legacy re-dispatch target' }) - const first = db.createDispatchContext(task.id, 'term_legacy') + const first = createRootDispatch(db, task.id, 'term_legacy') const sent = (await send({ from: 'term_legacy', @@ -137,7 +139,7 @@ describe('orchestration.send Dispatch authority', () => { expect(JSON.parse(sent.message.payload)).toMatchObject({ dispatchId: first.id }) db.failDispatch(first.id, 'worker stopped before coordinator read its mail') - const second = db.createDispatchContext(task.id, 'term_legacy') + const second = createRootDispatch(db, task.id, 'term_legacy') if (type === 'escalation') { applyEscalationToDispatch(db, db.getMessageById(sent.message.id)!, () => {}) diff --git a/src/main/runtime/rpc/methods/orchestration-send.test.ts b/src/main/runtime/rpc/methods/orchestration-send.test.ts index dafcbcc4327..36eaaa8b0a9 100644 --- a/src/main/runtime/rpc/methods/orchestration-send.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-send.test.ts @@ -7,6 +7,7 @@ import type { OrchestrationDb } from '../../orchestration/db' import type { OrcaRuntimeService } from '../../orca-runtime' import type { RuntimeTerminalSummary } from '../../../../shared/runtime-types' import { ORCHESTRATION_CONTRACT_VERSION } from '../../../../shared/protocol-version' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' function lifecycleGroupRecipientError( type: 'worker_done' | 'heartbeat' | 'escalation' | 'decision_gate' @@ -92,7 +93,7 @@ describe('orchestration RPC methods', () => { it('routes exact Dispatch mail independently of terminal handles', async () => { setup() const task = db.createTask({ spec: 'controlled worker' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const result = (await call('orchestration.send', { from: 'term_coord', @@ -117,7 +118,8 @@ describe('orchestration RPC methods', () => { it('routes Dispatch mail by stable pane identity after worker handle remint', async () => { setup() const task = db.createTask({ spec: 'controlled worker after restart' }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, 'term_worker_before', 'tab_worker:leaf_worker' @@ -185,7 +187,7 @@ describe('orchestration RPC methods', () => { it('completes an identity-less injected send through its explicit worker handle', async () => { setup() const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') vi.spyOn(runtime, 'getTerminalPaneKey').mockImplementation((handle) => handle === 'term_worker' ? 'tab_worker:leaf_worker' : null ) @@ -211,7 +213,8 @@ describe('orchestration RPC methods', () => { it('fences a replacement process for a capability-less manual Dispatch', async () => { setup() const task = db.createTask({ spec: 'process-bound manual work' }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, 'term_worker', 'tab_worker:leaf_worker', @@ -257,7 +260,8 @@ describe('orchestration RPC methods', () => { async (type) => { setup() const task = db.createTask({ spec: `process-bound ${type}` }) - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, 'term_worker', 'tab_worker:leaf_worker', @@ -313,7 +317,7 @@ describe('orchestration RPC methods', () => { setup() const task = db.createTask({ spec: 'work' }) const dependent = db.createTask({ spec: 'dependent', deps: [task.id] }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') vi.spyOn(runtime, 'getTerminalPaneKey').mockImplementation((handle) => handle === 'term_coord' ? 'tab_coord:leaf_coord' : null ) @@ -338,7 +342,7 @@ describe('orchestration RPC methods', () => { it('ignores caller-supplied pane claims and uses the runtime-observed pane', async () => { setup() const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue('tab_worker:leaf_worker') vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {}) vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {}) @@ -374,7 +378,7 @@ describe('orchestration RPC methods', () => { it('requires the minted capability, exact pane, and process incarnation', async () => { setup() const task = db.createTask({ spec: 'capability work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') const capability = db.mintDispatchCapability({ dispatchId: dispatch.id, paneKey: 'tab_worker:leaf_worker', @@ -456,7 +460,7 @@ describe('orchestration RPC methods', () => { it('does not wake waiters for a heartbeat suppressed at send time', async () => { setup() const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') db.updateTaskStatus(task.id, 'completed') vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {}) const notify = vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {}) @@ -476,7 +480,7 @@ describe('orchestration RPC methods', () => { it('still wakes waiters for a heartbeat on an active dispatch', async () => { setup() const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {}) const notify = vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {}) @@ -685,7 +689,7 @@ describe('orchestration RPC methods', () => { it('continues to send worker_done to a concrete terminal handle', async () => { setup() const task = db.createTask({ spec: 'work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') const result = (await call('orchestration.send', { from: 'term_worker', @@ -832,7 +836,7 @@ describe('orchestration RPC methods', () => { it('releases dispatch lock before waking recipients when worker_done is sent via send', async () => { setup() const task = db.createTask({ spec: 'lock-release work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') // Why: waiter notification must observe the settled Dispatch, not stale lifecycle state. vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => { @@ -857,13 +861,13 @@ describe('orchestration RPC methods', () => { expect(db.getActiveDispatchForTerminal('term_worker')).toBeUndefined() // Lock released — a new dispatch to the same terminal must succeed. const t2 = db.createTask({ spec: 'follow-up work' }) - expect(() => db.createDispatchContext(t2.id, 'term_worker')).not.toThrow() + expect(() => createRootDispatch(db, t2.id, 'term_worker')).not.toThrow() }) it('records heartbeat when heartbeat is sent via send', async () => { setup() const task = db.createTask({ spec: 'heartbeat work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {}) await call('orchestration.send', { @@ -883,7 +887,7 @@ describe('orchestration RPC methods', () => { it('does not release dispatch lock for non-lifecycle sends', async () => { setup() const task = db.createTask({ spec: 'in-flight work' }) - const dispatch = db.createDispatchContext(task.id, 'term_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker') vi.spyOn(runtime, 'deliverPendingMessagesForHandle').mockImplementation(() => {}) await call('orchestration.send', { diff --git a/src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts b/src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts index 8c6c3156c52..e77f1fd78fe 100644 --- a/src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-tasks-dispatch.test.ts @@ -4,6 +4,7 @@ import { createOrchestrationRpcHarness } from './orchestration-rpc-test-harness' import type { OrchestrationDb } from '../../orchestration/db' import type { OrcaRuntimeService } from '../../orca-runtime' import { buildInjectRejectionMessage } from './orchestration-inject-rejection-message' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' describe('orchestration RPC methods', () => { const h = createOrchestrationRpcHarness() @@ -116,7 +117,7 @@ describe('orchestration RPC methods', () => { setup() const t1 = db.createTask({ spec: 'ready work' }) const t2 = db.createTask({ spec: 'active work' }) - const ctx = db.createDispatchContext(t2.id, 'term_worker') + const ctx = createRootDispatch(db, t2.id, 'term_worker') const result = (await call('orchestration.taskList', {})) as { tasks: { @@ -176,7 +177,7 @@ describe('orchestration RPC methods', () => { it('completion frees the active dispatch context', async () => { setup() const task = db.createTask({ spec: 'work' }) - db.createDispatchContext(task.id, 'term_a') + createRootDispatch(db, task.id, 'term_a') await call('orchestration.taskUpdate', { id: task.id, @@ -397,7 +398,7 @@ describe('orchestration RPC methods', () => { setup() const t1 = db.createTask({ spec: 'first' }) const t2 = db.createTask({ spec: 'second' }) - db.createDispatchContext(t1.id, 'term_a') + createRootDispatch(db, t1.id, 'term_a') await expect(call('orchestration.dispatch', { task: t2.id, to: 'term_a' })).rejects.toThrow( /already has an active dispatch/ @@ -453,7 +454,7 @@ describe('orchestration RPC methods', () => { it('shows dispatch context for a task', async () => { setup() const task = db.createTask({ spec: 'work' }) - db.createDispatchContext(task.id, 'term_a') + createRootDispatch(db, task.id, 'term_a') const result = (await call('orchestration.dispatchShow', { task: task.id @@ -474,7 +475,7 @@ describe('orchestration RPC methods', () => { it('--preamble returns the preamble text', async () => { setup() const task = db.createTask({ spec: 'refactor auth' }) - db.createDispatchContext(task.id, 'term_a') + createRootDispatch(db, task.id, 'term_a') const result = (await call('orchestration.dispatchShow', { task: task.id, diff --git a/src/main/runtime/rpc/methods/orchestration-worker-interactive-wait.test.ts b/src/main/runtime/rpc/methods/orchestration-worker-interactive-wait.test.ts index 2392e3e4490..dac76afea71 100644 --- a/src/main/runtime/rpc/methods/orchestration-worker-interactive-wait.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-worker-interactive-wait.test.ts @@ -6,6 +6,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { OrcaRuntimeService } from '../../orca-runtime' import { OrchestrationDb } from '../../orchestration/db' import { ORCHESTRATION_METHODS } from './orchestration' +import { createRootDispatch } from '../../orchestration/db/root-dispatch-test-fixture' vi.mock('electron', () => ({ BrowserWindow: { fromId: vi.fn(() => null) }, @@ -100,12 +101,12 @@ describe('worker-show interactive wait (STA-3714, STA-4513)', () => { if (!paneKey || !incarnation) { throw new Error('Runtime did not expose the worker pane identity.') } - const dispatch = db.createDispatchContext( + const dispatch = createRootDispatch( + db, task.id, terminal.handle, paneKey, - 'launch-hash', - // A dispatch recorded against a process that has since been replaced. + 'launch-hash', // A dispatch recorded against a process that has since been replaced. opts?.breakIdentity === true ? `${incarnation}:replaced` : incarnation ) db.mintDispatchCapability({ diff --git a/src/main/runtime/rpc/methods/orchestration-worker-stop-liveness-verdict.test.ts b/src/main/runtime/rpc/methods/orchestration-worker-stop-liveness-verdict.test.ts index c08cc90554d..d381fa965ad 100644 --- a/src/main/runtime/rpc/methods/orchestration-worker-stop-liveness-verdict.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-worker-stop-liveness-verdict.test.ts @@ -44,6 +44,8 @@ describe('worker-stop against a terminal we lost contact with', () => { }) const task = db.createTask({ spec: 'stop worker', runId: run.id }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, runtimeEpoch: runtime.getRuntimeId() diff --git a/src/main/runtime/rpc/methods/orchestration-workers-recovery.test.ts b/src/main/runtime/rpc/methods/orchestration-workers-recovery.test.ts index 35798feed76..3e8f9ec27be 100644 --- a/src/main/runtime/rpc/methods/orchestration-workers-recovery.test.ts +++ b/src/main/runtime/rpc/methods/orchestration-workers-recovery.test.ts @@ -61,6 +61,8 @@ describe('orchestration worker recovery', () => { }) const task = db.createTask({ spec: 'recover worker', runId: run.id }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, runtimeEpoch @@ -206,6 +208,8 @@ describe('orchestration worker recovery', () => { }) const task = db.createTask({ spec: 'interrupted', runId: run.id }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, runtimeEpoch: 'previous_runtime' @@ -244,6 +248,8 @@ describe('orchestration worker recovery', () => { }) const task = db.createTask({ spec: 'stop remote worker', runId: run.id }) const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: task.id, startOptions: {}, runtimeEpoch: runtime.getRuntimeId(), diff --git a/src/main/runtime/rpc/methods/orchestration-workers.ts b/src/main/runtime/rpc/methods/orchestration-workers.ts index d1d94d897a5..6bc37f0c0b3 100644 --- a/src/main/runtime/rpc/methods/orchestration-workers.ts +++ b/src/main/runtime/rpc/methods/orchestration-workers.ts @@ -20,14 +20,24 @@ import { } from './orchestration-worker-setup-gate' import { failWorkerStartWithReceipt } from './orchestration-worker-start-receipt' import { prepareLocalWorkerStart } from './orchestration-worker-start-validation' +import { resolveDispatchCreator } from './orchestration-dispatch-creator' +import { resolveOrchestrationCaller } from './orchestration-run-scope' export const ORCHESTRATION_WORKER_START_METHODS: RpcMethod[] = [ defineMethod({ name: 'orchestration.workerStart', params: WorkerStartParams, - handler: async (params, { runtime, orchestrationMutation }) => { + handler: async ( + params, + { runtime, orchestrationMutation, orchestrationCompatibilityEvidence } + ) => { const db = runtime.getOrchestrationDb() - const coordinatorPane = runtime.getTerminalPaneKey(params.from) + // Why: worker-start was the only Run-scoped verb that skipped this, so a + // declared --from could name someone else's pane and inherit their depth. + const coordinatorPane = resolveOrchestrationCaller(runtime, { + callerTerminalHandle: params.from, + callerEvidence: orchestrationCompatibilityEvidence + }) const run = coordinatorPane ? db.getCurrentRunForPane(coordinatorPane) : undefined if (!run || (params.run && params.run !== run.id)) { throw new OrchestrationError( @@ -110,6 +120,8 @@ export const ORCHESTRATION_WORKER_START_METHODS: RpcMethod[] = [ : 'existing_worktree' } const started = db.createStartingWorkerDispatch({ + creator: resolveDispatchCreator(runtime, params.from), + maxDepth: runtime.getNestedWorkerMaxDepth(), taskId: task.id, retryOf: params.retryOf, startOptions, diff --git a/src/main/runtime/rpc/methods/orchestration.ts b/src/main/runtime/rpc/methods/orchestration.ts index b986fa25148..2cf7c8b1a10 100644 --- a/src/main/runtime/rpc/methods/orchestration.ts +++ b/src/main/runtime/rpc/methods/orchestration.ts @@ -2,6 +2,7 @@ import { z } from 'zod' import { setImmediate as yieldToEventLoop } from 'node:timers/promises' import { defineMethod, type RpcMethod } from '../core' +import { resolveDispatchCreator } from './orchestration-dispatch-creator' import { OptionalFiniteNumber, OptionalString, OptionalBoolean, requiredString } from '../schemas' import { LEGACY_CONTRACT_VERSION, @@ -1663,13 +1664,15 @@ export const ORCHESTRATION_METHODS: RpcMethod[] = [ } revalidateLegacyCoordinator?.() - const ctx = db.createDispatchContext( - params.task, - to, + const ctx = db.createDispatchContext({ + taskId: params.task, + assigneeHandle: to, assigneePaneKey, - dispatchAuthority?.launchTokenHash ?? undefined, - processIncarnation - ) + launchTokenHash: dispatchAuthority?.launchTokenHash ?? undefined, + processIncarnation, + creator: resolveDispatchCreator(runtime, params.from), + maxDepth: runtime.getNestedWorkerMaxDepth() + }) const dispatchCapability = params.inject ? db.mintDispatchCapability({ dispatchId: ctx.id, diff --git a/src/main/runtime/rpc/orchestration-11745-regression-verification.test.ts b/src/main/runtime/rpc/orchestration-11745-regression-verification.test.ts index 73f201a91a2..47d585ca244 100644 --- a/src/main/runtime/rpc/orchestration-11745-regression-verification.test.ts +++ b/src/main/runtime/rpc/orchestration-11745-regression-verification.test.ts @@ -23,6 +23,7 @@ import { request, type LegacyCompatibilityDispatcherHarness } from './orchestration-legacy-compatibility-dispatcher-test-fixture' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' // Why: an unrelated caller the runtime CAN resolve to a pane — otherwise the refusal would be // stable_pane_required and would prove nothing about Run authorization. @@ -642,7 +643,8 @@ function createAdoptedDb(options: { settleWork: boolean }): { const before = new OrchestrationDb(dbPath) const task = before.createTask({ spec: 'legacy assignment', createdByTerminalHandle: 'term_old' }) - before.createDispatchContext( + createRootDispatch( + before, task.id, 'term_old_worker', 'tab_old:33333333-3333-4333-8333-333333333333' diff --git a/src/main/runtime/rpc/orchestration-current-authority-precedence.test.ts b/src/main/runtime/rpc/orchestration-current-authority-precedence.test.ts index 27e5b0e1bea..fc6cd30f82e 100644 --- a/src/main/runtime/rpc/orchestration-current-authority-precedence.test.ts +++ b/src/main/runtime/rpc/orchestration-current-authority-precedence.test.ts @@ -15,6 +15,7 @@ import { WORKER_HANDLE, WORKER_PANE } from './orchestration-legacy-compatibility-dispatcher-test-fixture' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' afterEach(() => { cleanupLegacyCompatibilityDispatcherHarnesses() @@ -287,7 +288,8 @@ function createCurrentDispatch(harness: ReturnType): { coordinatorPaneKey: CURRENT_COORDINATOR_PANE }) const task = harness.db.createTask({ spec: 'current assignment', runId: run.id }) - const dispatch = harness.db.createDispatchContext( + const dispatch = createRootDispatch( + harness.db, task.id, CURRENT_WORKER_HANDLE, CURRENT_WORKER_PANE @@ -326,7 +328,7 @@ async function createReusedCurrentDispatch( coordinatorPaneKey: CURRENT_COORDINATOR_PANE }) const task = harness.db.createTask({ spec: 'reused terminal assignment', runId: run.id }) - const dispatch = harness.db.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(harness.db, task.id, WORKER_HANDLE, WORKER_PANE) const capability = harness.db.mintDispatchCapability({ dispatchId: dispatch.id, paneKey: WORKER_PANE, diff --git a/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher-test-fixture.ts b/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher-test-fixture.ts index b2c31d741c0..cca68da8f63 100644 --- a/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher-test-fixture.ts +++ b/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher-test-fixture.ts @@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db' import type { RpcRequest, RpcResponse } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' export const WORKER_HANDLE = 'term_legacy_worker' export const WORKER_PANE = 'tab_worker:33333333-3333-4333-8333-333333333333' @@ -55,7 +56,7 @@ export function createHarness(): LegacyCompatibilityDispatcherHarness { spec: 'legacy assignment', createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE) before.close() const raw = new Database(dbPath) diff --git a/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher.test.ts b/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher.test.ts index 0379646cb52..ca19570cbf5 100644 --- a/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher.test.ts +++ b/src/main/runtime/rpc/orchestration-legacy-compatibility-dispatcher.test.ts @@ -15,6 +15,7 @@ import { WORKER_HANDLE, WORKER_PANE } from './orchestration-legacy-compatibility-dispatcher-test-fixture' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' afterEach(() => { cleanupLegacyCompatibilityDispatcherHarnesses() @@ -191,7 +192,8 @@ describe('legacy compatibility through RpcDispatcher', () => { processIncarnation: 'process-1' }) harness.db.updateTaskStatus(harness.taskId, 'ready') - const currentDispatch = harness.db.createDispatchContext( + const currentDispatch = createRootDispatch( + harness.db, harness.taskId, 'term_current_worker', 'tab_current_worker:77777777-7777-4777-8777-777777777777', @@ -395,7 +397,8 @@ describe('legacy compatibility through RpcDispatcher', () => { coordinatorPaneKey: 'tab_current_coord:55555555-5555-4555-8555-555555555555' }) const task = harness.db.createTask({ spec: 'current assignment', runId: run.id }) - const dispatch = harness.db.createDispatchContext( + const dispatch = createRootDispatch( + harness.db, task.id, 'term_current_worker', 'tab_current_worker:66666666-6666-4666-8666-666666666666', diff --git a/src/main/runtime/rpc/orchestration-legacy-coordinator-race.test.ts b/src/main/runtime/rpc/orchestration-legacy-coordinator-race.test.ts index 4b8bb388226..9236d643e40 100644 --- a/src/main/runtime/rpc/orchestration-legacy-coordinator-race.test.ts +++ b/src/main/runtime/rpc/orchestration-legacy-coordinator-race.test.ts @@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db' import type { RpcRequest, RpcResponse } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' const COORDINATOR_HANDLE = 'term_legacy_coord' const COORDINATOR_PANE = 'tab_coord:44444444-4444-4444-8444-444444444444' @@ -45,7 +46,7 @@ function createHarness(): Harness { spec: 'legacy assignment', createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE) before.close() const raw = new Database(dbPath) diff --git a/src/main/runtime/rpc/orchestration-legacy-question-takeover.test.ts b/src/main/runtime/rpc/orchestration-legacy-question-takeover.test.ts index 37fb70d116f..77d7e2d5a02 100644 --- a/src/main/runtime/rpc/orchestration-legacy-question-takeover.test.ts +++ b/src/main/runtime/rpc/orchestration-legacy-question-takeover.test.ts @@ -11,6 +11,7 @@ import { OrcaRuntimeService } from '../orca-runtime' import type { RpcRequest } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' const COORDINATOR_HANDLE = 'term_legacy_coord' const CURRENT_COORDINATOR_HANDLE = 'term_current_coord' @@ -42,7 +43,7 @@ function createHarness(options?: { seedCutoverQuestion?: boolean; seedCutoverAns spec: 'legacy assignment', createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE) const cutoverQuestion = options?.seedCutoverQuestion ? before.insertMessage({ from: WORKER_HANDLE, diff --git a/src/main/runtime/rpc/orchestration-legacy-takeover-delivery.test.ts b/src/main/runtime/rpc/orchestration-legacy-takeover-delivery.test.ts index 0451eed101a..bcaf5fca11f 100644 --- a/src/main/runtime/rpc/orchestration-legacy-takeover-delivery.test.ts +++ b/src/main/runtime/rpc/orchestration-legacy-takeover-delivery.test.ts @@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db' import type { RpcRequest } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' const WORKER_HANDLE = 'term_legacy_worker' const WORKER_PANE = 'tab_worker:33333333-3333-4333-8333-333333333333' @@ -53,7 +54,7 @@ function createHarness(): Harness { spec: 'legacy assignment', createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE) before.close() const raw = new Database(dbPath) diff --git a/src/main/runtime/rpc/orchestration-legacy-takeover-dispatcher.test.ts b/src/main/runtime/rpc/orchestration-legacy-takeover-dispatcher.test.ts index aa4c7daef01..b3774ce735f 100644 --- a/src/main/runtime/rpc/orchestration-legacy-takeover-dispatcher.test.ts +++ b/src/main/runtime/rpc/orchestration-legacy-takeover-dispatcher.test.ts @@ -11,6 +11,7 @@ import { OrchestrationDb } from '../orchestration/db' import type { RpcRequest } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' const WORKER_HANDLE = 'term_legacy_worker' const WORKER_PANE = 'tab_worker:33333333-3333-4333-8333-333333333333' @@ -48,7 +49,7 @@ function createHarness(): Harness { spec: 'legacy assignment', createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = before.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(before, task.id, WORKER_HANDLE, WORKER_PANE) before.close() const raw = new Database(dbPath) diff --git a/src/main/runtime/rpc/orchestration-mutation-ledger.test.ts b/src/main/runtime/rpc/orchestration-mutation-ledger.test.ts index bc54f1643ed..d44d3f153d7 100644 --- a/src/main/runtime/rpc/orchestration-mutation-ledger.test.ts +++ b/src/main/runtime/rpc/orchestration-mutation-ledger.test.ts @@ -10,6 +10,7 @@ import { OrchestrationDb } from '../orchestration/db' import { defineMethod, type RpcRequest } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' const Params = z.object({ subject: z.string() }) @@ -304,6 +305,8 @@ describe('durable orchestration mutation ledger', () => { .update(JSON.stringify({ method: 'orchestration.workerStart', params })) .digest('hex') const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, taskId: params.task, startOptions: {}, mutationReceipt: { @@ -371,7 +374,7 @@ describe('durable orchestration mutation ledger', () => { coordinatorPaneKey: 'tab_coord:leaf_coord' }) const task = db.createTask({ spec: 'ask', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_worker', 'tab_worker:leaf_worker') + const dispatch = createRootDispatch(db, task.id, 'term_worker', 'tab_worker:leaf_worker') const capability = db.mintDispatchCapability({ dispatchId: dispatch.id, paneKey: 'tab_worker:leaf_worker', diff --git a/src/main/runtime/rpc/orchestration-runtime-update-settlement.test.ts b/src/main/runtime/rpc/orchestration-runtime-update-settlement.test.ts index 4dc8cd892b1..52205d00791 100644 --- a/src/main/runtime/rpc/orchestration-runtime-update-settlement.test.ts +++ b/src/main/runtime/rpc/orchestration-runtime-update-settlement.test.ts @@ -10,6 +10,7 @@ import { OrchestrationDb } from '../orchestration/db' import type { RpcRequest, RpcResponse } from './core' import { RpcDispatcher } from './dispatcher' import { ORCHESTRATION_METHODS } from './methods/orchestration' +import { createRootDispatch } from '../orchestration/db/root-dispatch-test-fixture' const WORKER_HANDLE = 'term_pre_update_worker' const WORKER_PANE = 'tab_pre_update:33333333-3333-4333-8333-333333333333' @@ -64,7 +65,7 @@ function createUpdateHarness(): Harness { spec: 'finish work across an app update', createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = oldRuntimeDb.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(oldRuntimeDb, task.id, WORKER_HANDLE, WORKER_PANE) const capability = oldRuntimeDb.mintDispatchCapability({ dispatchId: dispatch.id, paneKey: WORKER_PANE, diff --git a/src/main/runtime/rpc/orchestration-task-dispatch-invariant.test.ts b/src/main/runtime/rpc/orchestration-task-dispatch-invariant.test.ts index 539ddd1429a..0d194a372ba 100644 --- a/src/main/runtime/rpc/orchestration-task-dispatch-invariant.test.ts +++ b/src/main/runtime/rpc/orchestration-task-dispatch-invariant.test.ts @@ -44,7 +44,12 @@ describe('Task/Dispatch state invariant', () => { const task = harness.db.createTask({ spec: 'retain assignment', runId: harness.runId }) const dispatch = dispatchStatus === 'pending' - ? harness.db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }).dispatch + ? harness.db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }).dispatch : await dispatchTask(harness, task.id, WORKER_HANDLE) const response = await updateTask(harness, task.id, 'ready', 'must not persist') @@ -285,7 +290,12 @@ async function createCapableDispatch( status: 'pending' | 'dispatched' ): Promise<{ dispatch: { id: string }; capability: string }> { if (status === 'pending') { - const dispatch = harness.db.createStartingWorkerDispatch({ taskId, startOptions: {} }).dispatch + const dispatch = harness.db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId, + startOptions: {} + }).dispatch const capability = harness.db.prepareStartingWorkerAuthority({ dispatchId: dispatch.id, handle: WORKER_HANDLE, @@ -312,7 +322,12 @@ function createSupervisedDispatch( taskId: string, status: 'pending' | 'dispatched' ): { dispatch: { id: string }; capability: string } { - const dispatch = harness.db.createStartingWorkerDispatch({ taskId, startOptions: {} }).dispatch + const dispatch = harness.db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId, + startOptions: {} + }).dispatch const capability = harness.db.prepareStartingWorkerAuthority({ dispatchId: dispatch.id, handle: WORKER_HANDLE, diff --git a/src/main/runtime/runtime-rpc-long-poll-transport.test.ts b/src/main/runtime/runtime-rpc-long-poll-transport.test.ts index e14b4561dcc..43279ea7931 100644 --- a/src/main/runtime/runtime-rpc-long-poll-transport.test.ts +++ b/src/main/runtime/runtime-rpc-long-poll-transport.test.ts @@ -14,6 +14,7 @@ import { waitFor, seedSupervisedAskWorkers } from './runtime-rpc-test-harness' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' vi.mock('../git/worktree', () => { const worktrees = [ @@ -129,7 +130,7 @@ describe('OrcaRuntimeRpcServer', () => { coordinatorPaneKey: 'tab_coord:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb' }) const task = db.createTask({ spec: 'Wait for an answer', runId: run.id }) - db.createDispatchContext(task.id, 'term_asker', askerPaneKey) + createRootDispatch(db, task.id, 'term_asker', askerPaneKey) const server = new OrcaRuntimeRpcServer({ runtime, userDataPath, diff --git a/src/main/runtime/runtime-rpc-test-harness.ts b/src/main/runtime/runtime-rpc-test-harness.ts index 709585cb987..917ea73bd73 100644 --- a/src/main/runtime/runtime-rpc-test-harness.ts +++ b/src/main/runtime/runtime-rpc-test-harness.ts @@ -1,6 +1,7 @@ import { createConnection, type Socket } from 'node:net' import type { OrchestrationDb } from './orchestration/db' import { ORCHESTRATION_CONTRACT_VERSION } from '../../shared/protocol-version' +import { createRootDispatch } from './orchestration/db/root-dispatch-test-fixture' export async function sendRequest( endpoint: string, @@ -112,6 +113,6 @@ export function seedSupervisedAskWorkers(db: OrchestrationDb, workerHandles: str }) for (const workerHandle of workerHandles) { const task = db.createTask({ spec: 'Wait for coordinator input', runId: run.id }) - db.createDispatchContext(task.id, workerHandle) + createRootDispatch(db, task.id, workerHandle) } } diff --git a/src/main/ssh/ssh-remote-orca-cli.test.ts b/src/main/ssh/ssh-remote-orca-cli.test.ts index e9cb35e352b..0b3695aec5c 100644 --- a/src/main/ssh/ssh-remote-orca-cli.test.ts +++ b/src/main/ssh/ssh-remote-orca-cli.test.ts @@ -15,6 +15,7 @@ import { OrchestrationDb } from '../runtime/orchestration/db' import { OrcaRuntimeService } from '../runtime/orca-runtime' import type { HostCliPassthroughOptions } from './ssh-remote-cli-host-passthrough' import { runRemoteOrcaCli } from './ssh-remote-orca-cli' +import { createRootDispatch } from '../runtime/orchestration/db/root-dispatch-test-fixture' // Why: pointing the passthrough at a missing CLI entry forces the legacy // in-process fallback, which is what these dispatch tests exercise. @@ -241,7 +242,7 @@ describe('runRemoteOrcaCli', () => { coordinatorPaneKey: 'tab_coord:leaf_coord' }) const task = db.createTask({ spec: 'remote work', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_ssh', 'tab_owner:leaf_owner') + const dispatch = createRootDispatch(db, task.id, 'term_ssh', 'tab_owner:leaf_owner') vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue('tab_foreign:leaf_foreign') try { @@ -299,7 +300,7 @@ describe('runRemoteOrcaCli', () => { coordinatorPaneKey: 'tab_coord:leaf_coord' }) const task = db.createTask({ spec: 'remote work', runId: run.id }) - const dispatch = db.createDispatchContext(task.id, 'term_ssh', 'tab_owner:leaf_owner') + const dispatch = createRootDispatch(db, task.id, 'term_ssh', 'tab_owner:leaf_owner') vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue('tab_owner:leaf_owner') try { @@ -358,7 +359,12 @@ describe('runRemoteOrcaCli', () => { coordinatorPaneKey: 'tab_coord:leaf_coord' }) const task = db.createTask({ spec: 'remote work', runId: run.id }) - const started = db.createStartingWorkerDispatch({ taskId: task.id, startOptions: {} }) + const started = db.createStartingWorkerDispatch({ + creator: { kind: 'system' }, + maxDepth: Number.MAX_SAFE_INTEGER, + taskId: task.id, + startOptions: {} + }) const capability = db.prepareStartingWorkerAuthority({ dispatchId: started.dispatch.id, handle: 'term_ssh', diff --git a/src/main/ssh/ssh-remote-orchestration-compatibility.test.ts b/src/main/ssh/ssh-remote-orchestration-compatibility.test.ts index 45e6eb57f41..17923f5911c 100644 --- a/src/main/ssh/ssh-remote-orchestration-compatibility.test.ts +++ b/src/main/ssh/ssh-remote-orchestration-compatibility.test.ts @@ -17,6 +17,7 @@ import type Database from '../sqlite/sync-database' import type { HostCliPassthroughOptions } from './ssh-remote-cli-host-passthrough' import { runRemoteOrcaCli } from './ssh-remote-orca-cli' import { acknowledgeRemoteOrcaCliPostOutput } from './ssh-remote-orchestration-post-output' +import { createRootDispatch } from '../runtime/orchestration/db/root-dispatch-test-fixture' const LEGACY_FALLBACK_OPTIONS: HostCliPassthroughOptions = { execPath: '/host/electron', @@ -57,7 +58,7 @@ function createLegacyRuntime() { runId: run.id, createdByTerminalHandle: COORDINATOR_HANDLE }) - const dispatch = db.createDispatchContext(task.id, WORKER_HANDLE, WORKER_PANE) + const dispatch = createRootDispatch(db, task.id, WORKER_HANDLE, WORKER_PANE) const sqlite = (db as unknown as { db: Database.Database }).db sqlite .prepare( diff --git a/src/shared/constants.ts b/src/shared/constants.ts index a6698d0108d..928e774eaa3 100644 --- a/src/shared/constants.ts +++ b/src/shared/constants.ts @@ -281,6 +281,7 @@ export function getDefaultSettings(homedir: string): GlobalSettings { artifactsEnabled: true, artifactSharingEnabled: false, agentSkillSharingEnabled: false, + nestedWorkerMaxDepth: 1, showArtifactsButton: false, showSkillsButton: false, showMobileButton: true, diff --git a/src/shared/global-settings-types.ts b/src/shared/global-settings-types.ts index 9899a3ffde3..0ed2c72154c 100644 --- a/src/shared/global-settings-types.ts +++ b/src/shared/global-settings-types.ts @@ -230,6 +230,10 @@ export type GlobalSettings = { artifactSharingEnabled?: boolean /** Capability gate for agent/CLI skill publishing; manual reviewed publishing remains available. */ agentSkillSharingEnabled?: boolean + /** How deep dispatched workers may nest. 1 = workers cannot dispatch sub-workers. + * Renderer-writable only: omitted from the SettingsUpdate RPC schema so a worker + * cannot raise its own cap via `orca settings update`. */ + nestedWorkerMaxDepth?: number /** Only toggles the sidebar shortcut; Artifacts stay reachable from Settings. */ showArtifactsButton?: boolean /** Only toggles the sidebar shortcut; Skills stay reachable from Settings. */ diff --git a/src/shared/nested-worker-depth.test.ts b/src/shared/nested-worker-depth.test.ts new file mode 100644 index 00000000000..a91dd87ad00 --- /dev/null +++ b/src/shared/nested-worker-depth.test.ts @@ -0,0 +1,44 @@ +import { describe, expect, it } from 'vitest' +import { + NESTED_WORKER_MAX_DEPTH_DEFAULT, + nestedWorkerDepthExceededMessage, + resolveNestedWorkerMaxDepth +} from './nested-worker-depth' + +describe('resolveNestedWorkerMaxDepth', () => { + it('defaults to 1 when unset', () => { + expect(resolveNestedWorkerMaxDepth(undefined)).toBe(1) + expect(resolveNestedWorkerMaxDepth(null)).toBe(1) + expect(resolveNestedWorkerMaxDepth({})).toBe(1) + }) + + it('accepts whole numbers at or above 1', () => { + expect(resolveNestedWorkerMaxDepth({ nestedWorkerMaxDepth: 1 })).toBe(1) + expect(resolveNestedWorkerMaxDepth({ nestedWorkerMaxDepth: 3 })).toBe(3) + }) + + // A malformed setting must not become a way to get unlimited nesting, so every + // rejected shape falls back to the default rather than disabling the cap. + it.each([ + ['a numeric string', '2'], + ['a boolean', true], + ['zero', 0], + ['negative', -1], + ['fractional', 1.5], + ['NaN', Number.NaN], + ['Infinity', Number.POSITIVE_INFINITY], + ['null', null] + ])('falls back to the default for %s', (_label, value) => { + expect(resolveNestedWorkerMaxDepth({ nestedWorkerMaxDepth: value as unknown as number })).toBe( + NESTED_WORKER_MAX_DEPTH_DEFAULT + ) + }) +}) + +describe('depth-exceeded message', () => { + it('names both depths and tells the worker to finish the task itself', () => { + const message = nestedWorkerDepthExceededMessage(2, 1) + expect(message).toContain('depth 2 (max 1)') + expect(message).toContain('Complete this task yourself') + }) +}) diff --git a/src/shared/nested-worker-depth.ts b/src/shared/nested-worker-depth.ts new file mode 100644 index 00000000000..927f9801c12 --- /dev/null +++ b/src/shared/nested-worker-depth.ts @@ -0,0 +1,42 @@ +import type { GlobalSettings } from './global-settings-types' + +/** + * How deep dispatched workers may nest. 1 means a coordinator dispatches + * workers and those workers may not dispatch further — the behaviour Orca + * documented but never actually enforced. + */ +export const NESTED_WORKER_MAX_DEPTH_DEFAULT = 1 + +/** Root coordinators are depth 0; the first generation of workers is depth 1. */ +export const ROOT_DISPATCH_DEPTH = 0 + +export const NESTED_WORKER_DEPTH_EXCEEDED_CODE = 'nested_worker_depth_exceeded' + +export function nestedWorkerDepthExceededMessage(childDepth: number, maxDepth: number): string { + // Why "complete this task yourself": a refusal alone leaves the worker looping + // on a capability it will never get. + return ( + `Sub-worker dispatch is not permitted at depth ${childDepth} (max ${maxDepth}). ` + + 'Complete this task yourself.' + ) +} + +export const NESTED_WORKER_DEPTH_EXCEEDED_NEXT_STEPS: readonly string[] = [ + 'Do the work in this terminal instead of dispatching a sub-worker.', + 'To allow deeper nesting, open Settings → Agents in the Orca desktop app and raise "Nested worker depth".' +] + +/** + * Clamp to a usable integer. Anything that is not a whole number >= 1 falls back + * to the default rather than disabling the fence: a malformed setting must not + * be a way to get unlimited nesting. + */ +export function resolveNestedWorkerMaxDepth( + settings: Pick | null | undefined +): number { + const raw = settings?.nestedWorkerMaxDepth + if (typeof raw !== 'number' || !Number.isInteger(raw) || raw < 1) { + return NESTED_WORKER_MAX_DEPTH_DEFAULT + } + return raw +}