Commit Graph
7 Commits
Author SHA1 Message Date
Jinwoo Hong 06a607a1d7 feat(orchestration): make multi-agent workflows durable (#16904)
<!-- orca-pr-loc -->
<!-- Programmatic LoC summary. Do not edit by hand; rewritten on every commit. -->

| | Files | Added | Deleted | Net |
| :--- | ---: | ---: | ---: | ---: |
| Test | 225 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​21666 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​2820 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18846 |
| Prod | 348 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​17107 | $\color{#cf222e}{\Huge{\mathbf{−}}}$​4706 | $\color{#1a7f37}{\Huge{\mathbf{+}}}$​12401 |

<!-- /orca-pr-loc -->

## ELI5

Orca now treats orchestration like a durable control plane instead of inferring success from terminal keystrokes. Agents can tell whether a prompt was accepted or a turn started, replay an ambiguous request without sending twice, and recover coordinator mail after a crash. Completed workers can be inspected, released, or retained, and their panes no longer auto-resume as if the work were still running.

## What changed

- **Run receipts** from `run-create/use/current/show/list` are the row without routing plumbing (`home_database`, `coordinator_pane_key`) and without the duplicate `binding` object.
- **`terminal send` receipts are honest and idempotent.** `input_accepted` and `turn_started` are the only stages; `--wait-submit` observes without resending; `--retry-request <uuid>` replays the exact request against the same process incarnation. A transport timeout keeps the retry ID; only a different runtime answering strips it. Value-less or non-UUID `--retry-request` is rejected on the CLI and the SSH shim.
- **Mailbox delivery is committed before wakeup.** Pointer writes are staged in the DB before any PTY byte, replayed once after restart, and never emit a naked Enter. The watermark that parks concurrent deliveries is released with the DB reservation. Restart rescans pointer-pending and `dispatch:` mailboxes.
- **Lifecycle is a guarded transition graph** (`lifecycle-transition.ts`) with a table-driven test over every caller edge. Task reopen/overturn stays in the public contract. A PTY exit during `worker-stop` is the stop succeeding, not a failure.
- **Worker lifecycle CLI:** `worker-start` (`--spec` creates Task + attempt in one call), `worker-show`, `worker-read` (provider transcript first, bounded terminal fallback with a typed reason, local/WSL/SSH), `worker-stop`, `worker-abandon`, `worker-release`, `worker-retain`, `worker-list` (rowid-fenced pagination, fleet liveness, `attention`, literal `nextAction`).
- **Release is an explicit ownership table** (`decideWorkerTerminalRelease`): only an `owned` resource can be settled, the archive is mandatory where reachable, and an owner whose process is proven exited can always get out of `retained` via `archive_status: unavailable`. User-taken-over, external, and transferred panes stay retained.
- **Settled-worker resume fence** (folds in #17651): a settled dispatch whose pane is still open is fenced at settlement, on stop/abandon/exit, and at startup; lifted on release, retain, takeover, and pane reuse.
- **Liveness is `live` / `unverifiable` / `exited` only**, from execution-host evidence. Fleet projection reads the evidence clock, not the relay delivery clock. A host-certified exit outranks the worker's settled state. `unverifiable` never authorizes stop, abandon, retry, or release, in code or in the guide.
- **Federation:** structured reads negotiate by `method_not_found` so every shipped host keeps transcript-first output; exited remote workers are closed before being reported closed; epoch fencing holds across peer restart, downgrade, and pairing rotation; no per-second forced capability probe.
- **Schema v35:** repairs databases stamped v34 by the pre-fix branch (mailbox_handle default, index predicates), drops the write-only `lifecycle_transition_receipts` ledger and five never-read v31 identity columns.
- **Schema v36:** `dispatch:<id>` mailboxes get a real consumer generation on `dispatch_contexts` and `remote_dispatch_attachments`, bumped and fenced in the same transaction on every re-attach (manual inject, worker-start, federated attach). A stale worker whose Dispatch moved to another process now gets `consumer_fenced` instead of silently acking the new worker's Delivery. Run mailboxes already worked this way.
- **Schema v37:** `dispatch_contexts` records its creator (`creator_handle`, `creator_pane_key`), so a coordinator's context-only self-dispatch is bookkeeping rather than a nesting parent; before this, one self-dispatch made every later `worker-start` from that coordinator fail the depth cap. Pre-v37 rows keep counting (fails closed).
- **Dispatch-mailbox ownership is checked, not inferred.** A `check` from a process whose pane no longer holds the Dispatch, or whose last Attempt was abandoned/failed and moved to another terminal, gets `consumer_fenced` instead of an empty inbox that reads as "no mail yet". `--peek`/`--all` stay readable. A paneless caller still gets `stable_pane_required` with the rebind recovery.
- **Liveness certification is stricter:** a `process_exited` stage whose termination reason is `unknown` (a stop that was issued but never observed) projects `unverifiable`, not `exited`. Federated `worker-show` carries the execution host's verdict and host kind instead of a local guess. A live, ready worker with nothing pending has `nextAction: none` rather than pointing at the `worker-show` that produced it.
- **Wire:** `workerShow` keeps `dispatch.task_id` next to `taskId` for shipped CLIs. `ask --json` uses the standard `{ok, result}` envelope like every sibling verb.
- **Migration start-version detection** treats the two v32 recovery columns as versioned. Before this, every shipped database stamped below 32 resolved to the v6 floor and replayed the whole chain (the v23 backfill synthesized 68 phantom retained workers on a real v30 profile). Verified on a copy of a real 62 MB v30 profile: starts at 30, no row delta, integrity ok, 11 ms.
- **Skill guide** rewritten as a ≤200-line kernel plus seven references, to the outcome-first standard (Result / Done / Safe failure first, conditions not case lists, one done bar, references loaded at the point of use). The canonical loop uses `worker-start --spec`, names `worker-list` for completion accounting, documents `--retry-request` / `request-show` / `--wait-submit`, and requires positive evidence before any stall action. The other seven guides get the same treatment in #18724, split out so this PR stays orchestration-only.
- **`rpc/methods/orchestration-*`** (126 flat files) regrouped into `orchestration/{worker,federation,messaging,runs,gates}/`.

## Why

User reports showed the same boundary failures: false `agent_prompt_stalled` causing duplicate sends (#15180), coordinators unable to trust screen scrapes, cold-parked terminals receiving a pointer without the submit, settled workers accumulating as live tabs and auto-resuming after restart, and no way to tell a stalled worker from a working one.

## Linked issues

Fixes #15180. Fixes #17935 (orchestration skill description is 866 characters; a guard now caps every bundled skill at 1,024). Supersedes #17651 (fence folded in). Advances #16660, #16522, #14907, #13047.

## Review record

This PR was reviewed adversarially after revival: eight independent lenses (lifecycle, mailbox, send, worker, federation, transcript, complexity, live ergonomics), each required to prove findings with a failing test. That produced 16 proven blockers, all fixed with red-then-green regression tests, followed by two re-review rounds and a third fix wave that caught 3 regressions introduced by the fixes and 7 fixes that missed their target; all closed. A final pass (five lenses incl. a live built-runtime smoke, then a re-review of the fix wave) found and fixed seven more, chiefly the stale-worker mailbox steal, the self-dispatch depth wedge, and the unproven-exit certification. Three independent Codex (gpt-6-astra) passes followed: the first found nothing new, the second found and fixed 3 defects (task-status reachability, WSL-local host classification, peer-capability epoch), the third found and fixed 6 (production PTY controller never installed settled writes, ambiguous in-flight pointer failures allowed duplicate replay, SSH/relay deadlines cut off a valid `--wait-submit`, stop-vs-exit race during inspection, and two release-recovery paths for vanished or exited terminals). The full record (findings, proof tests, triage, declines with reasons) is archived outside the repo.

**Rework after the live smoke.** A first live cross-host run on the shipped adhoc build (this Mac, a paired Windows host on the same build, a paired Mac on 1.4.195, and an SSH host) found a P1: a running local worker read `unverifiable`/`missing_status` because the fleet snapshot rows lacked the terminal handle the matcher keyed on. A 59-row failure table over every bug fixed during review showed the same two classes recurring: a fact dropped in transit through optional fields, and two authorities for one fact. Two blind designs (Opus, Codex) converged on the same mechanisms, and the scoped tranches landed here with red-then-green seam tests from the real producer to the real consumer, faults injected only at the transport or hook-ingest boundary:

- **Settlement (data-loss class):** one three-valued `WriteSettlement` (`accepted | refused{reason} | unverifiable{reason, bytesHandedToTransport}`) from the SSH multiplexer through daemon client, providers, controller, to pointer staging. No boolean, no rejection-as-third-state. The two silent degrades that fabricated a handoff are deleted; a provider that cannot settle refuses before any effect. Pointer text and Enter share the contract; a partial flush is `unverifiable`, never `refused`.
- **Evidence identity (false-liveness class):** fleet agent-status evidence is a tagged union (`binding: worker | pane | unresolved{reason}`, `clock: observed | delivery`) minted once at ingest, so a hook row captured on one process incarnation can never bind to a later dispatch on the same pane. The matcher's `!worker.paneKey ||` defaults are gone. One host-scope parser replaces two.
- **Small pre-merge items:** `capability_unsupported` from an old peer is no longer relabelled `host_unavailable`; a producer census test asserts every agent-status consumer path projects a pane-only hook row as `live`.

Two ergonomics defects the second live run surfaced on a real database are fixed here too: a pre-v3 dispatch already marked `completed` projected as `outcome_unknown` / `requiresAction: true` forever (three copies of the outcome ladder disagreed on legacy rows; now one resolver, legacy `completed` reads `succeeded` with nothing to act on, legacy `failed` stays actionable on the failure), and an unscoped `worker-list` enumerated the entire database (now defaults to the Run bound to the calling terminal, `--run` overrides, and the receipt's additive `scope` field says which).

A third live round on the shipped adhoc build of `b082443e1f` (same four hosts) plus an unscripted run in the user's own prompt style (a plain Claude Code shell, `/orchestration`, three workers, zero errors, bound-Run default confirmed) found two more branch defects, fixed with red-then-green tests: a worker freshly started on a paired server projected `unverifiable`/`host_indeterminate` with `requiresAction` for ~3 minutes, including after its own `worker_done`, because the host's federation observation returned `missing_liveness_verdict` for any PTY the liveness register had not yet swept (the host now reads a connected pane it owns locally as `live`; disconnected or SSH-scoped panes stay `unverifiable`); and six pre-v3 completed rows still carried an `input` category because settling through the task-status path or `failDispatch` never closed the Dispatch's pending question threads (both paths close them now, and schema v38 closes threads already pending on settled rows). The guide's `worker-start` examples now show `--model sonnet`, since an omitted model inherits the launcher's default.

A Codex adversarial pass on the tranche diff found one real design hole (identity minted at read time instead of ingest, now closed) and two daemon settlement paths that threw instead of settling (fixed). Two `@ts-nocheck` runtime mixins on these paths were extracted into checked modules; the repo-wide `@ts-nocheck` count is unchanged at 171.

Deletions during review: ~1,900 lines (write-only ledger, unread columns, dead v1 archive path, test harnesses shipped in prod, duplicated liveness and state-machine copies, self-capability checks that were compile-time true).

## Testing

- `pnpm typecheck:tsc:node|cli|web` clean
- `pnpm run check:code-quality:changed` 0 findings; `check:react-doctor:changed` 0
- `pnpm verify:bundled-skill-guides`, `verify:skill-bundle-manifest`
- full `pnpm test` on the integrated head: 72,332 pass / 292 skipped; the only failures were three non-PR files (two zsh live-shell suites hit a node-pty spawn-helper ENOENT while a concurrent native rebuild ran, 44/44 in isolation; `release-checkout.unit.test.ts` is a known 30 s load timeout that passes in isolation on `origin/main` too).
- CI on 70b4811267 (rerun, pre-Codex): the only reds are five SSH e2e specs plus `terminal-send-agent-prompt-submit:198`, each shown failing identically on main (main's E2E workflow is red on its last 40 runs). The terminal-send spec is root-caused and fixed separately in #18707. The Windows hook-service flake (#17721) and the federation load flake did not recur.
- Skills: `pnpm exec vitest run` over the skill gate files plus `src/cli`, `config/scripts`, `src/main/skills` pass; live smoke on the built CLI of `skills get orchestration` and `--full` (7 references).
- live headless runtime (`orca-dev serve`, isolated profile): canonical loop, stop, release, archive read, retry rejection, stale-handle check, SIGKILL-and-replay all verified with receipts
- Live cross-host smoke on the shipped adhoc build of `0d465e7931` (this Mac and a paired Windows host on the build, a paired Mac left on 1.4.195, an SSH host): local, paired-new, paired-old and SSH loops all settle; running workers read `live` on every host and `exited` after release; the old peer reads `capability_unsupported` and refuses release honestly. Injected 10 s relay stall with a send in flight: delivered exactly once after recovery, zero duplicates. Every liveness field across 104 receipts is only `live` / `unverifiable` / `exited`.
- Final live cross-host smoke on the shipped adhoc build of `b082443e1f` (same hosts): every loop settles; 942 of 948 legacy completed rows read settled with `requiresAction: false` before the question-thread fix and all of them after; `worker-list` scope reads `bound` / `flag` / `all` correctly; 122 JSON receipts carry only `live` / `unverifiable` / `exited`. Unscripted prompt-style run: clean.
- Confirmation smoke on the shipped adhoc build of `2da076d4e9` (this Mac and the paired Windows host, both updated): a freshly started Windows worker reads `live` on the first fleet poll and on all 20 that follow, with no `host_indeterminate` at any point, and `exited` after release; all 948 legacy completed rows read `requiresAction: false` with `nextAction: none` after schema v38; every verdict across 60 receipts is `live` / `unverifiable` / `exited`.
- Not physically exercised: WSL hosts, the renderer notification bell (headless has no renderer), same-session fence via a real pane close (renderer-only state), restart mid-delivery on a real app (covered by e2e only).

## Notes

- Remote-wire additions are optional fields or `method_not_found`-negotiated methods; one new Electron-only IPC channel (`agentStatus:legacyWorkerTerminalResumeFence`) never crosses the wire.
- SSH contact loss remains `unverifiable`; the execution host stays authoritative.
- Intentional wire projection change: an SSH host scope with an empty `targetId` now projects host id `ssh` instead of an empty string (remote-wire-compatibility rule 3, old clients decode the same field). A fleet pane key without a terminal handle is now `unidentifiable` rather than matched by pane key alone.
- Found live but pre-existing on main, filed separately: a relay daemon-start collision during transport loss rewrites the endpoint credential and wedges the surviving relay (host needs a manual kill); `terminal create` on a reconnecting SSH host reports an opaque `No PTY provider for connection`; `terminal list` reports `orphaned:false` and `terminal close` reports `ptyKilled:true` for a pane whose relay is gone (orchestration's own projection reads `unverifiable` correctly at the same moment).
- Downgrade after this PR is not a supported path: main opens a v37 database and early-returns (its inserts still work against the v36/v37 defaulted columns), but its one-outstanding-Delivery-per-Run index is a no-op against the branch's mailbox-scoped index of the same name.
- Known follow-ups (not blockers): `worker-list` materializes every dispatch row per call; a positive "agent absent" signal distinct from PTY liveness is a product decision left open (a headless fake agent never reaches `live`, so its `nextAction` stays `inspect`); a context-only self-dispatch still lists as `role: worker` in `worker-list`; `dispatch` task-not-found / task-not-ready / inject-rejected still surface as `runtime_error`; task and inbox receipts still expose raw row columns. Deferred skill product decisions live on #18724.
2026-09-06 14:34:03 -04:00
Neil 1924c8f5b1 feat(perf): lint repeated sort setup and schedule regression contracts (#18822)
* feat(perf): audit comparator setup and schedule performance contracts

* test(sqlite): close readers after expected busy failures

* ci(perf): trigger contract workflow on the contract files themselves

Without these paths a contract rename lands green on PR CI and only
breaks the next nightly, where nobody owns the failure. Also run the
OS-independent source audit once instead of on all three runners.
2026-09-05 13:56:06 -07:00
Brennan BensonandMerge Sim f4c2821167 refactor(agent-session-journal): move the session journal onto SQLite (#18652)
* refactor(agent-session-journal): move the session journal onto SQLite

The agent-session journal kept its state in three hand-rolled file formats: an
append-only `log.jsonl` with torn-tail repair, a `snapshot.json` holding folded
state plus a retained tail, and byte-quarantine files for anything unreadable.
This replaces all of it with one SQLite database per session — `journal.db`
beside the existing `blobs/` store — using the in-house adapter and the
open/pragma/migrate/harden pattern the orchestration database already follows.

Two tables: `journal_rows` (the append-only log, keyed by
`(session_id, epoch, seq)`) and `journal_sessions` (the derived projection,
upserted in the SAME transaction as every insert). Rows stay JSON in one
column, so the row schema, the version upcast chain, and the reducer survive
byte for byte — `journal-reducer.test.ts` and four other suites pass unchanged
and are the regression proof.

Deleted: `journal-log-file.ts`, `journal-compaction.ts`,
`journal-corruption-quarantine.ts`, and the public `compact()` /
`compactionBoundary` / `autoCompact` members, none of which had a non-test
caller.

Existing `log.jsonl` / `snapshot.json` journals are deliberately abandoned. No
importer: a session created on the old path stops working, which is acceptable
because the feature is off by default.

## The physical quota is repriced, because SQLite does not charge like a file

The 256 MiB per-session bound is unchanged, but the arithmetic under it could
not survive: SQLite grows the database in pages and the WAL in frames, and the
checkpoint that copies the WAL forward holds the same pages in both files at
once, so a transaction's peak is about twice its content. Admission now charges
the candidate transaction's own measured page cost, validated against a sweep
that runs as a regression test (`journal-database-space.test.ts`) rather than
derived from reasoning about the allocator.

Four things are load-bearing rather than tuning, each measured:

- `auto_vacuum = INCREMENTAL` must be set BEFORE `journal_mode = WAL`. Set it
  after and it is ignored with no error, reclamation silently becomes a no-op,
  and the file never shrinks again. Both halves are asserted.
- `wal_autocheckpoint = 0` plus an explicit `wal_checkpoint(TRUNCATE)` at the
  end of every write path, so the one moment the same pages live in two files is
  a moment the charge accounts for.
- Reclamation runs in bounded chunks. A single unbounded `incremental_vacuum`
  took a 252 MB directory to 504 MB — the reclamation added to defend the bound
  would have breached it. `PRAGMA incremental_vacuum(N)` also frees exactly one
  page unless it is stepped to completion, which no size assertion catches, so
  the freed page count is asserted directly.
- A blocked checkpoint leaves the WAL on disk together with the database growth
  it already copied, so admission charges that deferred copy explicitly. The
  term is zero whenever the last checkpoint succeeded, so the uncontended path
  admits and refuses an identical set.

The epoch discard is `DELETE FROM journal_rows` with no WHERE clause, which
takes SQLite's truncate optimization: measured at ~0.26% of the database in WAL
bytes where the `WHERE session_id = ?` form rewrote every emptied leaf at up to
99%. One database per session is what makes the unqualified form correct.

An open, empty journal costs 57,344 bytes before a single row exists, so a
configured quota below `JOURNAL_MIN_SESSION_BYTES` now fails loudly at open with
the existing `journal_bound_exceeded` instead of as a run of identical append
failures. No production caller configures one; the affected surface is test
fixtures, rescaled to the smallest value that restores what each case proves.

## One deliberate behaviour change

Compaction was the only mechanism that shed bytes inside an epoch, and the write
path called it precisely so an append at the bound was not refused. The
SQLite-shaped replacement — a bounded prefix delete — cannot be used: with the
snapshot gone the surviving rows ARE the state, so dropping the oldest of them
loses the oldest transcript silently at the next reopen. So no row is ever shed
inside an epoch, and a session whose row bytes alone reach the bound now refuses
every append where it previously compacted and continued. A loud typed refusal
beats silent data loss.

What still sheds is unreferenced BLOB bytes — the dominant and unbounded byte
source — on the same write-path hook. The escape from the hard stop is the fold
that already exists, `replaceEpochItems`, which now actually returns bytes to
the filesystem instead of leaving them on the freelist.

The prune's protected set is a union of live reducer digests AND the candidate
row's own digests, including those cited only by a nested lifecycle-batch
mutation. Content addressing never rewrites a digest already on disk, so
protecting live state alone deletes the blob the append is about to cite — a
dangling reference that surfaces one reopen later as an empty expansion on an
item the user can see. `journal-store-blob-budget.test.ts` pins it, and it goes
red when the set is narrowed back.

## Handle ownership

A file handle used to be opened and closed per append; a SQLite handle is held
for the session's lifetime. Every path that can open a connection now has one
owner: the open function owns its raw connection until it returns, the store
owns its retained one and releases it in a new `close()`, and every other
connection is closed by the call that opened it. The attach, recovery,
eviction, map-overwrite and host-teardown paths close what they drop, and host
teardown is failure-complete — the sink-barrier flush throws by design, so a
trailing close statement would be skipped on exactly the path that leaks.

`close()` has a stated contract: admission at enqueue and permanent, the close
step on the same queue past that gate, one shared in-flight attempt, fulfilment
terminal, and the release last and deliberately unguarded so a retry re-enters
it. Guarding the release would skip it on retry, guaranteeing a permanent leak
in exactly the case where it did not release.

`journal_closed` joins the error union for a write after `close()`; no file
outside the directory references any of these codes.

* fix(agent-session-journal): make a COMMIT final, stop repairs deleting valid rows, and keep rejected closes retryable

Six review findings on the SQLite journal migration.

1. A successful COMMIT is now the point of no return. The ordinary append,
   the epoch roll and the epoch replacement each adopt the committed row or
   epoch BEFORE any post-commit filesystem work; checkpoint, reclaim, blob
   prune and directory measurement run through `runJournalPostCommit`, which
   is best-effort by design and falls back to the transaction's own charge as
   a conservative footprint. Previously a post-COMMIT scan failure rejected a
   durable append and the next one reused its sequence, and a failed epoch
   housekeeping step left the store writing into a prefix already deleted.

2. Corruption repair preserves instead of destroying. A rejected suffix is
   copied into a new `journal_quarantine` table and removed from the live
   epoch in ONE transaction per chunk, charged against the session bound
   before a byte is written; a journal that cannot afford the copy refuses to
   open rather than falling back to deletion. The repair state is exposed as
   `journal.repair` and the rows are readable through
   `recoverQuarantinedRows()`, so Orca-owned submission, receipt and
   lifecycle identity survives a gap or a malformed row.

3. The physical charge covers the B-tree key payload. `session_id` and
   `epoch` are stored in both tables and both primary-key indexes and appear
   nowhere in `row_json`, so the journal boundary now bounds them and
   `journalTxnPhysicalCost` charges those bounds plus the projection upsert.
   The charge sweep runs the exact production transaction at maximum admitted
   key sizes.

4. A rejected `close()` no longer orphans its handle. Callers hand the
   journal to `agentSessionJournalCloseRetries` instead of swallowing the
   rejection, the attach map replacement is ABORTED when the previous
   journal will not close, host teardown retries what the registry holds, and
   a failed runtime teardown is retained so the next stop is a real retry.

5. `journalWalBytes()` returns zero only for ENOENT and propagates every
   other stat error, so admission and reclamation fail closed.

6. The WAL contention test closes the writer before removing its temp root
   and asserts the directory is removable once handles close.

Regression coverage: post-commit divergence (4), corruption repair (5),
key bounds (5), WAL stat (8), close retry (5), plus a runtime stop-retry
case. Each fix was ablated on this head and the matching tests go red.

* fix(agent-session-journal): anchor replay at sequence 1, make quarantine append-only, and charge it in bytes

Three ways the corruption quarantine still lost rows it was written to keep.

Replay validated contiguity from the first row that HAPPENED to remain, so an
epoch missing only its sequence-1 row declared the leftovers contiguous and set
no `truncateFrom`. The load was still corrupt, so recovery imported provider
history and `replaceEpochItems` deleted every live row — including Orca-minted
submission, receipt and lifecycle identity that no transcript can reconstruct,
and that nothing had quarantined. Replay now anchors at sequence 1, so a missing
epoch row rejects the whole surviving range before any replacement runs.

`journal_quarantine` was keyed on `(session_id, epoch, seq)` and copied with
`INSERT OR REPLACE`. A repair frees the sequences it removed and the live epoch
reuses them, so a second repair in the same epoch silently deleted what the
first preserved. The table is now keyed on a surrogate `quarantine_id`, the copy
is a plain append, and `(epoch, seq)` is metadata; existing v1 databases are
rekeyed in the migration that already bumps `user_version`.

The admission charge read `length(row_json)`, which counts CHARACTERS for a TEXT
value where `journalTxnPhysicalCost` expects physical UTF-8 bytes. A multibyte
suffix was charged at up to a third of what it writes, which defeats the
pre-write physical bound — over a megabyte on a maximum-size lifecycle batch.

* fix(agent-session-journal): keep a repaired epoch anchored and stop the v1 quarantine migration doubling the file

Replay validated numeric contiguity from sequence 1 but never that sequence 1
IS the epoch row. When the anchor was missing the repair set aside every
surviving row, and if provider-history import then failed — a transcript that
is temporarily gone is enough — the journal reopened as a clean, row-less
epoch: an ordinary append took sequence 1, replay accepted it, read-restore
published it as history, and automatic recovery never ran again while the
user's real messages sat in quarantine.

Replay now rejects an unanchored prefix, the open publishes an
`unreconcilable_prefix` anchor for an epoch its repair emptied, and that anchor
keeps reporting corrupt — so provider history is retried on every attach —
until the timeline is rebuilt or the session writes content of its own. A
repair also discloses rows it set aside when no line was unreadable at all,
which is the case that removes the most.

The v1 quarantine rekey copied every legacy row into the new table inside one
transaction and dropped the old one. A quarantine holds whole rejected rows: a
single 8 MiB row nearly doubled the database past the physical bound the open
had already checked, the dropped pages only reached the freelist, and the next
open refused the session it had just migrated. The v1 table is renamed and
frozen instead, and reads take both generations. Table creation also moves
inside the migration transaction, so a crash can no longer leave a v2-shaped
database still reporting version 0 for an older build to write into.

* fix(agent-session-journal): stop an empty provider transcript retiring the repair marker

A transcript that exists but decodes to zero messages was imported as a
success: the import published an empty `legacy_import` replacement that
deleted the `unreconcilable_prefix` anchor and its disclosure, so the next
probe read the session as clean and every later attach skipped provider
recovery while the user's rows sat in quarantine for good.

The import now leaves the epoch untouched when nothing decodes, reporting
`replaced: false`, and recovery treats that like a transcript it could not
read — the marker stands and a later attach with real history rebuilds the
timeline.

* style(agent-session-journal): merge the duplicate journal-database-space import

* refactor(agent-session-journal): drop quarantine, byte bound, blob spill and rate limit

Match what comparable implementations do: the journal is an unbounded
append-only SQLite log with no side tables and no admission control.

Corruption: the rejected suffix is DELETED rather than copied into a
quarantine table. The load still reports `corrupt` and recovery still
rebuilds the epoch from provider history, so the observable outcome is
unchanged — only the preservation half is gone. The schema is back to one
version with two tables; no v1 database exists outside unmerged commits of
this branch, so the rekey migration and the two-generation read path go with
it. Sequence-1 epoch anchoring and the empty-provider-transcript retry are
kept: both are about the corrupt signal being correct.

Size: no `maxSessionBytes`, so no page-cost arithmetic, reclaim band,
incremental vacuum, lifecycle byte reservations or `journal_bound_exceeded`.
`auto_vacuum` and `wal_autocheckpoint = 0` existed only to make a
transaction's physical cost predictable for that charge; with the charge gone
SQLite's default checkpointing is what the journal wants, and the explicit
pre-close checkpoint is redundant with the one `db.close()` performs. WAL,
`synchronous = FULL` and `busy_timeout` stay.

Payloads: an oversized body is truncated at the existing inline cap with the
existing marker and the remainder is discarded, bounded at the translation
layer that already calls these helpers. The truncation point and message do
not change; the content-addressed blob directory and all digest tracking do.

Rate: no `maxAppendsPerWindow` and no `journal_rate_exceeded`.

`JournalPayloadLimits` is now just the inline cap.

* fix(agent-session-journal): mark a partial repair pending and bound multi-block tool input

A repair that keeps its prefix had nothing durable to show for the suffix it
deleted: a sequence gap costs no malformed row, so no disclosure is appended,
and the surviving rows keep their epoch anchor. The next probe read a
contiguous anchored prefix, called it clean, and the deleted stretch of
timeline was never asked for again — silent loss, with the deletion already
committed. The deletion now writes a `journal_repairs` marker in the SAME
transaction, and replay keeps reporting corrupt while it stands. It retires
under exactly the rule the emptied-epoch anchor takes: a fresh epoch carries
the rebuild, or the session writes content of its own past the sequence the
repair left free. The repair's own disclosure is not that content.

Legacy import bounded a tool call's input only when it was the message's sole
block; the multi-block path returned `tool-call` unchanged, so a mixed message
from Claude, Grok or an omp execution cell persisted the whole input despite
`inlineHeadBytes`. `boundBlock` now routes it through `boundToolInput`.

Also drops canonical comments describing quarantine, snapshot files, blob
storage and blob compaction — none of which exist any more.

* fix(agent-session-wire): stop awaiting the synchronous journal probe

loadJournal runs on a sync-database connection and returns JournalLoad | null, so both wire call sites were awaiting a non-Promise. The type-aware code-quality gate flags it; the native gate does not.

---------

Co-authored-by: Merge Sim <sim@local>
2026-09-04 15:09:23 -07:00
Neil 9062494f9b fix(ai-vault): stop a whole opencode.db failure reading as one skipped transcript (#16587)
* fix(ai-vault): stop a whole opencode.db failure reading as one skipped transcript

#15036 reported "1 transcript skipped / database is locked" with both Agent
Session History scopes empty. Two separate defects.

The panel counts every unkinded scan issue as a skipped transcript, so a
failure that lost an entire *source* was reported as one lost *file*. The
whole-database failure is now kinded `scope`, and an unknown `kind` from a
newer host degrades to `scope` instead of failing validation and coming back
unkinded — a mixed-version remote host previously turned a source-level
failure into a phantom skipped transcript.

The read also inherited sqlite3's 0 ms busy timeout, so a genuinely contended
open failed in ~1 ms. It now opens once with a bounded timeout. No retry loop:
sqlite's own busy handler already blocks and retries internally for the whole
timeout, and WAL readers do not block on a writer at all (measured: 547/547
cross-process reads at timeout=0 while a writer held open transactions).

Measured against a real Ubuntu-24.04 distro, Windows cannot take SQLite's file
locks over \\wsl.localhost at all: an idle, never-WAL, nothing-attached
database still answers SQLITE_BUSY, a 5 s busy timeout does not change it, and
the identical bytes open fine once copied to local disk. So a lock-family error
on that share never means "a writer holds it" and no timeout can help. The copy
says so rather than sending the user after a write-ahead log that is not the
problem. Restoring those sessions needs an in-distro read; that is a follow-up,
and this PR no longer pretends a timeout will do it.

immutable=1 is deliberately not used as a workaround: over the same share it
opens and returns 100 of 150 rows, silently dropping everything still in the
uncheckpointed -wal — in a history panel, exactly the newest sessions.

* skip the provably futile busy wait on \\wsl.localhost paths
2026-08-26 23:37:16 -07:00
NeilandOrca 0fe04e2c91 perf(sqlite): cache prepared statements in SyncDatabase (#13769)
prepare() recompiled every statement, so orchestration reads re-parsed the same SQL
on the main thread. Adds a bounded LRU keyed by SQL, cleared on close() and before
schema-changing exec().

Wildcard selects are excluded: node:sqlite builds the first post-schema-change row
from stale column names, so a reused SELECT * can silently drop a freshly added
column. PRAGMAs stay uncached.

Co-authored-by: Orca <help@stably.ai>
2026-08-11 01:48:14 -07:00
Jinwoo HongandJinwoo-H cf16eac7f6 fix(agent-hooks): keep Node 18 relay companion loadable (#13135)
Co-authored-by: Jinwoo-H <Jinwoo-H@users.noreply.github.com>
2026-08-07 22:54:42 -07:00
Neil 1009ac9083 chore: update Electron to 42 (#3919) 2026-05-30 13:26:48 -07:00