Commit Graph
9 Commits
Author SHA1 Message Date
Brennan Benson 88a266e003 Make SQLite the source of truth for native chats (#39256)
* Retire projection adapter tests and use real SQL journal fixtures

* Persist bounded history detail and conversation outline projections

* Select rewind cuts by provider indexes and delete uniform rows as a set

* Cache bounded conversation outlines after reader projection

* Remove retired journal fold and message adapters

* Add the 4a main-vs-candidate native chat comparison harness

One entry (native-chat-4a-comparison.ts) seeds the 55k-history/310-chat
profile once through the seed arm, clones it per arm at a fixed path,
launches each built checkout in the background with an isolated home,
and drives the same stub provider events through the host RPC surface
both builds serve. Envelopes come from each arm's own source bundle.
Rows cover send/queue, 1/8/16 MiB streams with a competing send,
maintenance windows, open/reopen, history/detail, outline/fork,
rewind, record operations, R3 retained-history admission and, reported
separately, first-launch conversion (optional 20 GiB source). Output is
one JSON and one markdown table keyed by both head SHAs.

* Fail the 4a harness launch fast on a non-e2e build and stop the app it launched

* Shim tsx's __name in 4a harness pages and install the probe inside the guarded launch

* Read the native chat working flag from the 4a harness's visible root itself

* Retry transient main-process evaluates in the 4a harness host-work probe

* Use the history page's itemId in the 4a harness and name what a visible-state wait timed out on

* Activate 4a harness chat tabs through their tab group and seed enough smoke filler chats for the unopened-chat row

* WIP: migrate native chat tests to SQL journal readers

* Treat refusals carried inside a successful 4a harness RPC result as refusals

* Give the 4a stub's user echo the uuid its transcript row records

* Remove retired journal writer test machinery

* Chain the 4a stub's resumed transcript to the last message, past a fork copy's title row

* Fork the 4a harness's large chat through its latest response, not its latest prompt

* Port persisted chat tests to SQL fixtures (red WIP)

* Add the 4a main-vs-candidate native chat comparison harness

One entry (native-chat-4a-comparison.ts) seeds the 55k-history/310-chat
profile once through the seed arm, clones it per arm at a fixed path,
launches each built checkout in the background with an isolated home,
and drives the same stub provider events through the host RPC surface
both builds serve. Envelopes come from each arm's own source bundle.
Rows cover send/queue, 1/8/16 MiB streams with a competing send,
maintenance windows, open/reopen, history/detail, outline/fork,
rewind, record operations, R3 retained-history admission and, reported
separately, first-launch conversion (optional 20 GiB source). Output is
one JSON and one markdown table keyed by both head SHAs.

* Fail the 4a harness launch fast on a non-e2e build and stop the app it launched

* Shim tsx's __name in 4a harness pages and install the probe inside the guarded launch

* Read the native chat working flag from the 4a harness's visible root itself

* Retry transient main-process evaluates in the 4a harness host-work probe

* Use the history page's itemId in the 4a harness and name what a visible-state wait timed out on

* Activate 4a harness chat tabs through their tab group and seed enough smoke filler chats for the unopened-chat row

* Treat refusals carried inside a successful 4a harness RPC result as refusals

* Give the 4a stub's user echo the uuid its transcript row records

* Chain the 4a stub's resumed transcript to the last message, past a fork copy's title row

* Fork the 4a harness's large chat through its latest response, not its latest prompt

* Port ACP tests to SQL journal named reads

* Port Claude tests to SQL journal named reads

* Port Codex tests to SQL journals and drop the legacy row rollback test

* Port failure and persisted-read fixtures to the active chat database

* Remove obsolete internal-work counting from chat inventory tests

* test(native-chat): repair SQL epoch fixtures and sender closure case

* fix(native-chat): own async option preparation and port SQL runtime fixtures

* test(native-chat): encode the full founding identity

* fix(native-chat): recover saved live work on SQL open

* fix(native-chat): select retained image sources before history paging

* fix(native-chat): preserve catch-up authority and scoped journal teardown

* Cover the 4a conversion, history-shape and maintenance-debt cases through outside observation

Conversion: seed a medium chat (310 chats on the full profile) and refuse a
cached seed from another recipe. Observe the candidate's existing
epoch-installed ready boundary with a read-only poll of journal_chat_state.
An idle seed copy yields launch to every chat ready and the per-size
ready-mark intervals; the families copy verifies an unconverted first open and
waits for every chat ready before any steady-state family.

History/detail: the stub emits ordinary Claude subagent frames (interleaved
and sparse-root corpora) and a shapes turn (huge completed tool originals, one
above the detail reply cap, an oversized ordinary message). Each named shape is
verified through the real history/detail route before its samples count; diff
items stay explicit missing evidence because the Claude adapter writes none.

Maintenance: write and rewind large completed rows on a disposable seeded chat
under a held outside reader, then measure other-chat sends during the hold,
after release and while freed pages are reclaimed, from read-only page/WAL
state. Retired-epoch pruning is skipped unless such rows are observed.

* fix(native-chat): preserve stopped result disclosure and async fixture ownership

* Resolve lost-session warning identity at the journal writer

* Cover the 4a conversion, history-shape and maintenance-debt cases through outside observation

Conversion: seed a medium chat (310 chats on the full profile) and refuse a
cached seed from another recipe. Observe the candidate's existing
epoch-installed ready boundary with a read-only poll of journal_chat_state.
An idle seed copy yields launch to every chat ready and the per-size
ready-mark intervals; the families copy verifies an unconverted first open and
waits for every chat ready before any steady-state family.

History/detail: the stub emits ordinary Claude subagent frames (interleaved
and sparse-root corpora) and a shapes turn (huge completed tool originals, one
above the detail reply cap, an oversized ordinary message). Each named shape is
verified through the real history/detail route before its samples count; diff
items stay explicit missing evidence because the Claude adapter writes none.

Maintenance: write and rewind large completed rows on a disposable seeded chat
under a held outside reader, then measure other-chat sends during the hold,
after release and while freed pages are reclaimed, from read-only page/WAL
state. Retired-epoch pruning is skipped unless such rows are observed.

* Remove the retired ACP warning-reader fixture field

* Validate warning writer authority before duplicate selection

* Wake queued delivery when pending requests change

* Validate authority before all resolved item selection

* Use current chat test fixtures and name the handover journal owner

* Keep warning authority validation on the semantic selection branch

* test: lose unsaved ACP creations before a successful resume

* refactor: tighten SQL journal types and publication reads

* test: use SQL chat writers and async journal observations

* Replace casts and empty type imports in the 4a harness scripts with annotations and guards

Renderer times and tab ids from waitForFunction handles go through typeof
guards, the visibility probe is a declared Window global, the runtime call
keeps its own typed response, and cached manifests, Electron paths, profiles
and families are narrowed by guards. The empty `import type {}` blocks name
AppStore and use it. Three casts remain, each with a line-specific SAFETY:
the generic RPC result and Node's performance inside main-process callbacks.

* refactor: type handover owners and finite recovery snapshots

* Name the 4a arm's bundled envelope builders by their tracked source files

The protocol bundle imported from './src/shared/agent-session/...', a folder
the path-literal ratchet reads as an unresolved import. Each module is now a
tracked-file constant joined onto the arm's own checkout, the same form
runtime-file-watcher-resource-probe.mjs uses; the bundle and its exports are
unchanged.

* Replace casts and empty type imports in the 4a harness scripts with annotations and guards

Renderer times and tab ids from waitForFunction handles go through typeof
guards, the visibility probe is a declared Window global, the runtime call
keeps its own typed response, and cached manifests, Electron paths, profiles
and families are narrowed by guards. The empty `import type {}` blocks name
AppStore and use it. Three casts remain, each with a line-specific SAFETY:
the generic RPC result and Node's performance inside main-process callbacks.

* Name the 4a arm's bundled envelope builders by their tracked source files

The protocol bundle imported from './src/shared/agent-session/...', a folder
the path-literal ratchet reads as an unresolved import. Each module is now a
tracked-file constant joined onto the arm's own checkout, the same form
runtime-file-watcher-resource-probe.mjs uses; the bundle and its exports are
unchanged.

* Port cross-version wire tests to SQL journal reads

Hosts take the shared domain database, reads go through named async
journal reads and catch-up history pages, and released-build fixtures
type the release's own synchronous journal API.

* Align cross-version downgrade tests with one-way conversion

A downgraded build sees the chat and records as converted and its
later writes are not carried forward. Drop tests whose subject was
this build's legacy row and outcome writes; seed newer legacy rows
before the first conversion.

* Port cross-version wire tests to SQL journal reads

Hosts take the shared domain database, reads go through named async
journal reads and catch-up history pages, and released-build fixtures
type the release's own synchronous journal API.

* Align cross-version downgrade tests with one-way conversion

A downgraded build sees the chat and records as converted and its
later writes are not carried forward. Drop tests whose subject was
this build's legacy row and outcome writes; seed newer legacy rows
before the first conversion.

* test: open runtime fixtures with SQL journal workers

* test: combine runtime fixture imports

* test: prune resolved compatibility timeout finding

* test: use bundled SQL workers for runtime admission fixtures

* test: await SQL reader delivery and complete real rewind cuts

* test: include the medium chat in comparison profile metadata

* test: cover snapshot holds after replacement rollback and read failure

* test: keep clear history assertions on saved and rendered results

* Inject the deletion scenarios' record removal at the SQL record store's reads

The SQL record store has no transact or record-deletion command, so the
authoritative-deletion cases make the store's public record reads answer as
for a missing record and restore them before teardown. Assertions unchanged.

* Align cross-version fixtures with async SQL publication

Wait for frames that follow committed reads, await the async
completion feed, route the outline and item calls to their current
host methods, and expect the restart unreconciled latch on loaded
records.

* test: delete automation records with their command receipts

* Seed completed diff originals through the baseline writer and measure them

The 4a stub speaks Claude stream-json, whose adapter writes no diff items, so
completed diff originals were a skipped row. The seed now appends known
diffs once into its last filler chat through the seed arm's own original
journal writer, after that arm stops and before the seed is renamed or
copied, so every arm starts from the same rows and item IDs. Both arms verify
kind, original length, a short page head and the exact patch from detail
before timing the tail page and the detail read, and the conversion sweep
reports the diff chat apart from the small chats.

* Pin each 4a arm to a full SHA and prove its build came from it

Each --arm now names name=<checkout>@<full sha>. The harness refuses an arm whose HEAD
differs, whose product sources are dirty, whose renderer build-commit stamp is not that
HEAD, or whose sources changed after the build, and records that provenance in every
result. A cached seed from another seed-arm commit is refused, and report-only refuses to
mix results from different pins, seeds or recipes. Tables name full SHAs and disclose that
writer BEGIN-to-COMMIT spans are not observed.

* Establish restart recovery debt in the record writer

* Add active-chat resident memory comparison to 4a harness

* Commit complete provider dispatches and preserve Stop sources

* Coalesce pending publications across provider dispatches

* Capture CI watch actions at SQL writer publication

* Restore Claude task linkage in SQL acquisition facts

* Lift restarted queue holds through SQL admission

* Forward brief clear through Claude close

* Isolate SQL test worker lifetimes and prove fork durability

* Preserve saved queue order and recover restart custody on SQL open

* Keep saved queue contracts explicit and type fixtures accurately

* Run startup chat workspace conversion through its SQL owner

* Make journal command classifications exhaustive and remove unchecked casts

* Temporarily reverse quality cleanup to isolate translation regression

* Restore verified behavior-neutral journal quality cleanup

* Initialize real saved snapshot routes in Claude translation tests

* Repair native chat campaign readiness and contention measurements

* Measure fork contention under isolated retired history pressure

* Verify contention still targets a live stream at dispatch

* Run chat maintenance within the record-store lifetime

* Bound chat activity reads to work transitions

* test: adopt saved snapshots in direct Claude sinks

* Cover maintenance activity during registered starts

* test: initialize saved facts for Claude permission and resume fixtures

* test: await durable Claude resume and chain publications

* Complete provider dispatch identities in maintenance fixtures

* Preserve journal body admission and text close finality

* Scope optional permission backfill failure to its record change

* test: seed SQL reader fixtures through their owning contract

* test: assert saved Codex outcomes through SQL readers

* test: seed replay-budget withdrawals through command turns

* fix: refuse chat access on newer journal schemas

* test: migrate Claude echo wait; fails on known product finding

The retained Working assertion still fails after awaiting initial publication and accepted echo. The product finding is handed to the author.

* test: await SQL transcript and command publications

* test: migrate public rewind cursors and teardown warning scopes

* test: await child-work and status re-entry publications

* test: bind operation fixtures to SQL current-work reads

* test: establish recovery fixtures through owner restart reconciliation

* Keep child work owned across null status publications

* test: await subscriber and late-proof SQL observations

* test: resolve the profile transfer fixture from the repository root

* test: assert restart recovery capsules instead of internal call counts

* test: migrate Claude echo wait; fails on known product finding

The retained Working assertion still fails after awaiting initial publication and accepted echo. The product finding is handed to the author.

* test: await runtime session publications and final writer acknowledgement

* test: migrate coordinator mail SQL APIs; known product failures remain

* test: persist fork sources and use SQL RPC fixture contracts

* test: observe conversation publications in worker abandonment

* test: admit downgrade fixture owners through the SQL store

* test: await ordered attention prompt relay observations

* Keep status observer failures outside provider ingress

* Order provider facts before late delivery receipts

* test: submit replay fixture turns through the general writer command

* test: await durable worker delivery and lifecycle outcomes

* test: follow current provider ownership in coordinator mail

* fix(chat): settle mail custody from writer outcomes

* fix(perf): format retired fixture row count explicitly

* fix(chat): admit automatic worker close from current work

* Require live tool contention witness at send dispatch

* Await attention fixture updates and prune the fixed echo wait baseline

* Preserve the Pi startup exit verdict after recording process identity

* Check restored image deletion through the transcript view

* Adapt history pressure fixture and probes to current item storage

* Repair CI import paths and asynchronous lint findings

* Keep journal storage and projections acyclic

* Preserve proven startup facts across acquisition settlement

* Wait for the owning chat boundary in recovery and delivery tests

* Observe admitted delivery work in native chat fixtures

* Make native chat comparison measurements fail closed

* Isolate SQL worker fixtures across test files

* fix(sqlite): avoid retaining large bindings in cached statements

* perf(native-chat): reuse the captured queue publication

* test(orchestration): await refused mail settlement before recovery

* fix(orchestration): reconcile owned mail before consuming checks

* fix(orchestration): preserve each mail check response

* test(orchestration): diagnose surviving timers through public SQL reads

* perf(native-chat): place item metadata before large forms

* test(native-chat): require the octet-length column flag

* fix(native-chat): version persisted item presentations

* fix(native-chat): bound queued-image history admission

* test(orchestration): update shared mail source support

* fix(native-chat): preserve Stop transitions and admitted forms

* perf(native-chat): select the existing latest-turn index

* perf(native-chat): reuse history admission facts

* perf(native-chat): reuse shortened history wire forms

* perf(native-chat): batch bounded fork metadata reads

* fix(native-chat): synchronize CI fixtures and type mailbox receipts

* perf(native-chat): bound fork scratch presentation work

* test(native-chat): await subscriber catch-up completion

* fix(native-chat): retire readers after fork setup failure

* perf(native-chat): batch bounded fork metadata and output reads

* Reuse validated journal publication inputs on demand

* fix(chat): cancel reader requests on requested shutdown

* perf(native-chat): reuse narrow SQLite work reads

Read canonical current work and queue eligibility in one snapshot, and reuse validated work through the existing journal observation entry. Preserve full status publication, live authority validation, optional queue failures and final writer claims.

Frozen700 matches the reviewed 27-file source manifest and measured tree 6423b1cfbb. Cold and changed-head reads meet the one-point budget; unchanged publication frames improve.

Disclose report698 strict-rule misses accepted by the coordinator as noise: the approximately 4us warm in-memory hit has paired p50 ratio 1.047; one writer-ack cell (empty / 2 subscribers / changed) is +5.5% at p50, 0.259 to 0.274 ms, while completion is approximately twice as fast.

* test(native-chat): synchronize Stop with interrupt entry

The request note is written before the separate dispatch read and native cancel call, so observing the note cannot prove cancelTurn has started. Await the mock interrupt entry, then inspect the request note while its response stays unanswered.

Remove eventual polling and verify the single status note before and after the response without asserting its wording. No production ordering change, sleep, retry or timeout increase.

* fix(native-chat): refresh invalid full work projections

Use the existing SQL authority and context validation before preserving a
full projection, so refreshed narrow work can be reused after row-free
invalidation. Preserve a valid full projection arriving during a narrow
read and extend the existing owner scenario for both transitions.

Correct the description in b98a2544bf: the report698 strict misses are
small measured overhead, not proven noise. Empty warm currentWork p50 is
3.542 to 3.709 us (+0.167 us, 60/80 pairs slower, ratio 1.04715, 95% CI
1.02291-1.07259). Empty/2-subscriber/changed writer ack p50 rises 5.5%,
0.259459 to 0.273833 ms, with its CI excluding parity, on an unchanged
writer path; completion falls 1.546834 to 0.657125 ms. The coordinator
accepts these marginal costs; the strict FAILs remain disclosed.

Frozen714 patch 6a4ded1ee72e8c0134a9a2121fb39bde618213e3f740b4888fd13c8900274620.
Owning 53 tests pass on both configured entries; static gates pass. Private
point attribution verifies repeated narrow reuse and all 92 serving/frame
outputs match. Focused714 against pre-P7 base still measures a small warm
currentWork median overhead; report716 preserves raw samples and CIs.

* fix(native-chat): retain full work projections with saved Stop

Reuse the existing work-answer validity lookup when a narrow read returns.
A full status projection includes the saved Stop, while narrow work leaves
that overlay unrequested. Comparing only the authority work actually read
preserves a valid arriving full projection and retires stale authority or
context without a second cache or comparator.

Extend the held-read owner scenario with a real saved Stop. Private point
attribution shows unchanged full demand going from one reader point to
zero, with full and work outputs equal; invalidation reuse remains one
read followed by three cache hits. Owning 53 tests pass on both entries and
static gates pass. Frozen721 is parent plus two W3 files only; paused SQL
diagnostics work is excluded. Historical P7 benchmark costs remain small
measured overhead as corrected in 12d479e640 and report716.

* refactor(native-chat): remove obsolete work-state read plumbing

Remove the test-only combined work/queue wrapper and its queue callback after the work-state RPC removal. Delete unused projection and queue-list wrappers, keep internal functions private, and point the hydration overlap assertion at surviving current-work and queue owners.

Retain the narrow SQL work path, work-only projection entry and invalidation/Stop reuse guards: subscriber frames, history/current requests and background-task publication still demand them. Combined work and queue eligibility plus resolved-work reuse remain in production.

Validation: 7 owning files, 26 cases on direct Node and configured Bun entry; full configured Node typecheck and static gates pass. Focused parent-versus-change micro preserves exact output equality and point shape across 36 live queue/frame cells and 80 pairs; no paired interval excludes parity. Removed RPC work-state measurements are historical only.

Prior P7 strict misses remain disclosed small measured overhead: warm current-work median 3.542 to 3.709 us (+0.167 us, ratio 1.047; 60/80 pairs slower), and empty / 2 subscribers / changed writer acknowledgment 0.259459 to 0.273833 ms (+5.5%, CI excludes 1) on an unchanged writer path; completion 1.546834 to 0.657125 ms. Neither cost is established as noise.

* test(native-chat): preserve released chat queue publication coverage

Use the surviving queue publication owner after rewind and reopen,
preserving visible card persistence checks across both providers and all
three release cohorts without a production wrapper used only by tests.

Validation: all 49 cross-version and owning files pass on Node and the
configured Bun entry (148 passed, 1 skipped each); the e2e typecheck
ratchet has zero test errors and the owning static gates pass.

* Register native chat SQL workers with the main-blocking ratchet

The journal SQL writer and reader entries run on worker threads. Their
dependencies are grandfathered in the SQLite baseline as worker-only, the
same way existing worker dependencies are; none run SQL on Electron main.
Prune baseline rows the branch no longer needs.

* test(orchestration): hold native admission in mail reconciliation busy chats

The file-local busyChat answered turn/start, so main's automatic mail
steering (#41476) handed each queued mail card to the accepted turn about
80 ms after send. Cases that wait for a visible card then timed out (or
found it gone), and teardown logged the closed SQL reader. Reuse the
custody fixture's busyChat and settleTurn, which keep admission
unanswered until the scenario settles the turn.

Bun: 4/7 failed before; 35/35 passed across 5 runs after. Node: passes
with the custody file (18/18).

* test(agent-session): wait for the at-rest opening snapshot on the open stream

Subscribers queue the opening snapshot behind an asynchronous SQL work
read, so agentSession.subscribe can reply before the snapshot frame; the
queued-image wire contract depends on that ordering. The at-rest case
read frames only up to the reply and lost the race under CI load.

With a 50 ms opening delay injected locally: old assertion 3/3 failed
(expected undefined), new assertion 3/3 passed.

* Correct subscribe comment: the opening snapshot follows the reply

* test(agent-session): assert no agent start after the at-rest opening snapshot

* test(agent-session): wait for the status summary's own publication in the child-records switch

The session-list summary publishes after an async journal projection read that
neither flushStreamedEvents nor the strip's batch covers, so asserting it right
after either raced the SQL worker.

* fix(explorer): resync a recovered remote watch only after the host watcher is installed

The contact-regained resync fired when the files.watch subscription was accepted, before the
restarted host reported its watcher ready, so a file created in that gap was neither in the
re-read listing nor reported by the watcher. The managed-orcad explorer recovery spec now waits
for that resync before writing its probe file.

* fix(explorer): re-read a recovered remote explorer once when its host watcher fails to install

* test(explorer): the recovered watch resyncs after its ready signal

* test(agent-session): await the queued-operation results instead of polling a settled flag

The operation-order cases polled a settled flag and the compact spy with
vi.waitFor's 1 s default. Under SQL worker contention the goal or rewind
answers after more than a second, so the poll gave up while the operation
was still on its way (CI: "expected false to be true", then "closed by
quit" from the unhandled result). Await the result itself and a promise the
compact stub resolves.

With a 30 ms SQL reply delay injected locally plus CPU load, 3 repeats: the
old file failed "releases a startup goal" (expected false to be true) and
"answers a goal" (compact called 0 times); the awaited version passed all.

* fix(agent-session): reserve queued sends before a waiting operation's own place

DeliveryOrder.reconcile became async when the first-queued read moved to the
SQL readers, but the ready-operation path still called it without awaiting
and reserved the operation's place right after. The operation then took its
place before the queued send it was meant to follow, so a goal, rewind or
other operation requested while the agent starts could run ahead of a
message sent earlier; a failed read also became an unhandled rejection.
Await it, restoring main's order (reconcile, then reserve).

Found by typescript/no-floating-promises over src/main/native-chat.

* test(agent-session): let delivery park before the attention case's idle sweep

The single idle-sweep tick keeps a chat whose delivery loop is still
running, and the loop parks only after SQL reads that follow the second
send's completion event. When the tick ran first it touched the session
instead of retiring it, and the 1 s hasSession poll failed. Wait for the
loop to park (wake resolves when it does), as goal-steering does.

With an 8 ms SQL reply delay injected locally plus CPU load: old 2/5
failed (expected true to be false), new 5/5 passed.

* test(agent-session): await the fake Codex child's requests in compact-exit

Restarting after a mid-command exit runs the exit settlement, a fresh
acquire and the queued send's handover through the SQL workers before
turn/start, which outran vi.waitFor's 1 s poll under load. The shared fake
Codex now resolves nextRequest(method) when a child receives that request;
compact-exit awaits it instead of polling the calls list.

Seen once in a contended local sweep (8 ms SQL reply delay plus CPU load):
"expected [] to have a length of 1" at the turn/start poll. The new
version passed 5/5 under the same delay.

* fix(agent-session): keep a settled send's frame over an older work restatement

Subscriber delivery reads current work before the history page, then lets
that work restate unanswered sends over the page's rows. When a send's
settlement committed between the two reads, the page carried it accepted
but the earlier work restated it pending, so the frame at the new cursor
published the stale state. Later caught-up frames restate only still
unanswered sends, so the client never learned of the acceptance: the
Claude echo case kept a "pending" bubble although the journal held it
accepted. A restatement only describes sends its read saw unanswered, so
an accepted or rejected page row is newer and now wins.

structured-agent-session-claude-echo-working (failed once in CI, never
reproduced) with an 8 ms SQL reply delay injected locally plus CPU load:
before, most repeats stayed pending for 15 s while the stored row was
accepted at the subscriber's own cursor; after, 7/7 passed in ~100 ms.

* fix(native-chat): record the original journal copy identity

* chore(native-chat): grandfather the writer worker's one-shot source receipt

The receipt is captured once inside domain initialization, which runs only in
the SQL writer worker, so its synchronous stat and SQLite write never run on
Electron main or the orcad loop.

* test(native-chat): await current-request reads instead of polling for one second

The child-request case waited 1 s for the SQL reader's current-request
answer and failed on a loaded CI shard. Wake on each real state change
from the subscription and the request reader instead.

* fix(native-chat): re-answer current work when a catch-up page passes its cursor

A subscriber catch-up answered current work at one journal cursor, then read
the history page in a later reader transaction. The restated copy of a send
still pending at the work cursor replaced the page's accepted row, so the
renderer kept showing the send as sending.

Re-answer work when the page cursor has passed the work answer, refresh the
fan-out's shared work answer once the journal adopts a newer commit, and keep
the page's own rows when the page holds a commit not yet adopted.

* fix(native-chat): read subscriber page rows and current work in one reader snapshot

The catch-up overlaid a work answer from one journal cursor on a history page
read at another. Re-answering work afterwards still left a page that ran ahead
of the adopted cursor without its work, so a send shown as unknown could
revert to pending under an unchanged work fingerprint.

The history read now answers current work in its own transaction. The main
thread validates that answer against live owners and handover with the
existing currentness check, and reads the frame again until it has adopted
the frame's cursor. A cached answer is reused only when the page head is that
same cursor. The fan-out's queue now follows the work answer it was read
with.

* fix(native-chat): capture each subscriber frame at one held command-lane position

A subscriber frame paired rows, current work and the queue from different
positions, and waiting for the main thread to adopt a newer head re-read the
page until it did.

Each frame now captures its page and work (one reader transaction) and its
queue inside a single readInOrder on the chat's command lane, then releases
the lane before emitting. The queue reaches the frame through the explicit
queuePublication option, so emission never reads a newer one. A capture
whose live authority changed mid-read is discarded without advancing the
subscriber; the change's own publication delivers its replacement. Inactive
subscribers, closed journals and replaced session handles abandon the frame.
The resume path no longer decorates stop notes with work read before catch-up;
they restate the work of the frame the subscriber caught up to.

* fix(native-chat): validate whole captured frames and capture every work overlay

The queue gate inside a frame capture re-read work until it matched live
authority, holding the command lane, and the capture stopped validating
before its queue read. Stop-note pages, history replies, attach hydration
and queued-image decoration still paired rows and work from different
positions.

The queue gate now reads once at the frame's own authority, and outside a
capture once at the live one; a moved authority abandons the frame for the
change's own redelivery. The capture re-validates authority, handle and
cursor after its queue read. Stop-note pages are read on the lane only
while the journal is still at the delivered frame, otherwise the resume
catches up first. History replies and attach hydration use the same held
capture, and image decoration that finds the journal past its frame is
abandoned in favour of the newer commit's delivery. With every overlay
coherent, a settled page row can no longer meet an older restatement, so
the settled-send guard is removed.

* fix(native-chat): owe restatements after an emitted replacement and refuse exhausted captures as unavailable history

* test(native-chat): follow captured history frames in cross-version and queued-image wire tests

* Remove the converted legacy chat journal after its grace period (#42024)

* fix(native-chat): retire certified legacy journals after the grace period

* perf(native-chat): pace source proof through existing maintenance

* test(native-chat): exercise retirement with the native source lock

* fix(native-chat): account for queued card epochs during source proof

* fix(native-chat): settle retained journals and cancel stalled source leases

* fix(native-chat): keep the prepared retirement state narrowed

* fix(native-chat): apply retirement backoff before writer admission

* fix(native-chat): retry restored legacy retirement eligibility

* fix(native-chat): retain pending legacy eligibility helpers

* chore(native-chat): grandfather the writer worker's one-shot source receipt

The receipt is captured once inside domain initialization, which runs only in
the SQL writer worker, so its synchronous stat and SQLite write never run on
Electron main or the orcad loop.

* fix(native-chat): make retirement filesystem checks asynchronous

The four maintenance SQLite uses run only in the SQL writer worker and the 28 proof-page SQLite uses run only in the SQL reader worker; main-side proof coordination dispatches worker requests.

* fix(native-chat): build lock helper at the server glibc floor

Build Linux server helpers with the existing pinned glibc 2.28 slot images when the host is newer. Keep native floor and musl builds, and select the separate desktop toolchain explicitly. This prevents fast Ubuntu prebuilds from linking GLIBC_2.33/2.34 into a 2.28 payload.

Bound the helper PR workflow to 20 minutes and cover compiler selection without compiling native code locally.

* fix(native-chat): honor read-only retirement ownership

Do not acquire a writable retirement lifecycle for a newer-schema database, so shutdown never sends its read-only writer a retirement-credit command. Keep the existing preservation and empty-diagnostics regression unchanged.

Select the native exclusion test with the dedicated workflow flag instead of generic CI, and merge duplicate database imports in the retirement test.

* fix(native-chat): remove source proof table assertion

Annotate the content table inventory as readonly strings so membership checks accept schema table names without a type assertion.

* fix(native-chat): bound legacy retirement proof work

Persist source-intrinsic refusals for the exact copy, source and destination descriptor. Resume completed proof work after transient failures, and let the existing conversion owner finish pending imports.

Use covering-index key pages for history membership and exit terminal retirement states before writer admission. Preserve originals and prepared unlink manifests.

* fix(native-chat): recover legacy proof without repeating source work

Use the fixed profile-copy columns for source certification and keep damaged destination inventory retryable. Advance the shared refusal policy so earlier false inventory results are re-proved.

Complete verified empty heads after transient conversion failure, and carry proven original-source diagnoses through SQL errors without changing their ordinary identity. Persist only source-intrinsic refusals.

Validate in-place schema recovery, bounded empty completion, and malformed/newer histories at 128/1280 rows with zero retirement proof, conversion, restore or helper work across subsequent owners. Focused tests, tc:node, scoped checks and all 27 ratchets pass.

* test(native-chat): trace cursor resume frames on failure

Capture synthetic subscriber frame payloads and emission stacks when the existing restart assertion fails in CI. Keep the two-frame expectation and production behavior unchanged.

* test(native-chat): allow held-cursor roster frames on resume

A restarted host can publish an empty task and command roster before catch-up when its initial coherent capture is superseded. Validate that optional metadata frame, preserve the two history/work frames and no-replay checks, and await the work restatement instead of polling packet counts.

Remove temporary per-frame stack tracing now that CI identified the publisher. Production wire behavior is unchanged.

* test(native-chat): assert empty resume batches independently

Use literal empty arrays in the resume contract so a defect in the host's batch constructor cannot also change the expected value. Keep the optional held-cursor roster prefix and the two required catch-up/work frames.

* test(native-chat): expect the current conversation fence on resume

The roster publisher reads the conversation fence after restart and send, which can advance beyond the earlier runtime lease fence. Derive the expected fence from that current record and preserve every other resume assertion.

* test(native-chat): hold the domain database lock while a revoked history step is dispatched

* test(native-chat): accept held-cursor state frames before a resumed catch-up and fold them in a released client

* Start the chat journal writer earlier and defer maintenance until serving (#42045)

* Boot the chat journal writer during startup waits

Start the existing record-store slot after login env and before the daemon
wait in orcad, and right after runtime construction on Electron, through one
shared location builder that probes for saved chats once. Startup restoration
consumes that same attempt, so a refused journal is not reopened before RPC.
orcad's push-service start moves to its own module to keep the entry small.

* Start journal maintenance after startup restoration

The CPU profile showed the background sweep's reader spawn and writer
work competing with pre-RPC startup for the disk and CPU. A startup
open now holds maintenance until its restoration settles; other openers
start it on open as before. Reads still convert saved chats on demand.

* Move the headless-serve browser check to its own module

Behavior-neutral; keeps main-process-runtime-launch.ts within the line limit.

* Collect saved chat ids once and start journal maintenance after serving

orcad passes the ids it collected for the early open into runtime
construction instead of collecting them again. Background journal
maintenance now waits for the host's serving milestone (orcad readiness
published, Electron serve ready, desktop RPC start) rather than
restoration settling, so a stalled renderer cannot hold it. The desktop
early open starts after the first-window services fence. The refusal
test now observes that the journal is opened once.

* Keep the deferred desktop chat open inside shutdown, and start CI-line watches after serving

A first-window fence that settles after quit or a failed startup has
committed no longer opens the chat journal past its teardown; the
Electron startup-failure path now commits host shutdown as orcad's does.
Host installation re-arms CI-line watches only after a teardown stopped
them, so the first run starts at each host's serving milestone (orcad
after readiness, Electron after startup settles) instead of during
restoration.

* Admit CI-line watches at each host's serving milestone and let installs retry them

The first watch start no longer depends on renderer load: orcad admits it
after readiness, Electron serve after the serve-ready line, desktop once
the window and RPC start settle. Before admission, chat installation
leaves watches to that milestone; after it, installation's idempotent
start covers a failed first start as well as a teardown. Starts are
skipped once host shutdown has committed.

* Fence desktop watch admission, make warmup retry-only, and release failed watch starts

Desktop admits CI-line watches after the bounded first-window fence, so
automatic journal work never precedes it. The startup warmup timer only
retries a failed start; admission stays with each host's serving
milestone. A watch service whose start rejects after registering its
outcome observer and job hooks is stopped before a retry replaces it.

* Retry a failed CI-line watch start on the journal's maintenance pass

After admission, a saved-chat host whose first watch start failed had no
automatic retry. The journal's existing periodic maintenance pass now
calls the idempotent start when watches are admitted but neither running
nor starting; it does nothing after committed shutdown or while a service
runs. No new timer or loop, and nothing moves before readiness.

* Wire PR-watch recovery into every record-store opener

The chat host's install could open the once-slot first, without the
maintenance-tick hook, leaving a failed first watch start with no automatic
retry. The hook is now a required location field, set through one shared
builder by every production opener.

* Keep runtime identity state within the file-length limit

The saved-chat startup open's import now uses one namespace line, so the
file stays at the 300-line cap after the early-open field and method.

* Load the real chat host once before the PR-watch tests

The host-install-first regression loaded the real chat host module inside
the test, which CI's in-test vi.importActual gate refuses. It now loads in
beforeAll; the test body and its assertions are unchanged.

* Mark the desktop serving test's runtime stand-in as a reasoned cast

CI's changed-code gate requires the lint suppression form for a test cast;
the stand-in and its SAFETY reason are unchanged.

* Share journal database open options so the file stays within its line limit

* test(chat): restore released-client resume projection coverage

* test(chat): type the released-client fold accumulator

* test: fail the first opening subscription data frame

* test: update benchmark module paths after main sync

* test: resolve benchmark modules within each checkout

* test(native-chat): inject late receipts without a history read
2026-11-14 05:51:16 -08:00
Neil d3b6609ac9 refactor(rpc): group root modules by defining owner (#41475)
* refactor(rpc): group root modules by defining owner

* test(orcad): include JSON CLI failures in launcher diagnostics
2026-11-12 02:18:19 -08:00
Neil c10881fa8c refactor(shared): organize execution and automation defining owners (#39776)
* refactor: import journal queue state from defining owners

* refactor(shared): organize execution and automation owners

* fix: retain type imports used by surviving test cases

* refactor: finish work item owners and refresh validation metadata

* test: infer reconnect status fixture type

* test(claude): resolve source files beside same-named directories

* test(runtime): group repository creation mock registrations
2026-11-08 06:46:09 -08:00
OrcaWinandOrcaWin 8d2b5a2c07 test(rpc): drop the default-RPC-manifest guard where no default remains (#39684)
RpcDispatcher now requires its method list, so these suites can no longer consume
the default manifest. Suites that still reach the registry through the RPC server
keep the guard.

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-11-07 20:10:01 -08:00
Neil 27a724e303 refactor(backend): import defining owners and extract ACP startup (#39417)
* refactor(backend): use defining owners for orchestration and ACP startup

* test(orchestration): trim redundant observations and retain safety cases

* test(orchestration): seed restored sessions with one fixed operation clock
2026-11-07 04:15:38 -08:00
Neil 0314da7ed8 refactor(main): group RPC and orchestration modules by owner (#38813) 2026-11-06 08:21:33 -08:00
Neil 1c950baf33 refactor(main): group session lifecycle and journal modules by owner (#38638)
* refactor: group structured session modules by owner

* test: retire status history work-count oracle suite

* refactor(main): organize journal storage and projection owners

* Consolidate restart recovery fixture imports

* Consolidate concrete session option model imports
2026-11-06 06:00:08 -08:00
Neil 18bab90efa refactor(main): group client RPC pairing and host runtime owners (#38584)
* refactor(main): group client RPC pairing and host runtime owners

* test: target concrete mobile notification owner in headless fixture

* test: retarget dynamic quit mocks to runtime metadata owner
2026-11-05 09:16:47 -08:00
Brennan Benson 0fa5dae2f9 Read who each agent works with: its manager, workers and peers (#34890)
* Nest chat-UI orchestration workers under the agent that started them

Main's lineage projection only walked terminal panes, so workers running
in the chat UI never got a parent link and listed flat in the sidebar.
It now also projects chat workers, keyed by their visible chat row, and
resolves a chat creator or coordinator to that chat's visible row
instead of its handle or private mailbox pane key.

* Fix public worker lineage across assignment and row identities

* Complete terminal lineage provenance and bound active session selection

* Repair lineage test runtime boundaries and SQLite counters

* Provide read-only crew snapshots for each agent

* Batch public lineage verification after exact recency selection

* Derive crew relations once per meaningful host change

* Remove package manager lockfile churn from hook

* Use checked runtime values in lineage integration tests

* Document the trusted Dispatch SQL row boundary

* Keep lineage integration imports within their test programs

* Load native lineage fixtures before test execution

* test: complete lineage inputs in runtime and sidebar fixtures

* test: seed benchmark facts from active dispatches

* test: replace partial orchestration DB casts with typed fixtures

* test: complete stale-leaf mailbox fixture session input

* Derive durable crew parents from retained assignment proof

* fix(test): render worker lineage fixture with production list

* Add scoped terminal identity acquisition and preserve crew source wakes

* Share raw crew identity inputs through scoped production readers

* Exercise crew generation retirement through actual source owners

* Retire crew inputs on consumed ownership and visibility writes

* Use typed crew mutation and fixture access

* Repair crew fixture and RPC type contracts

* Align crew integration fixtures and RPC parameter contracts

* Encapsulate terminal identity mutations and captured reader contracts

* Seed terminal identity fixtures through canonical mutations

* Fix terminal identity fixture return type

* Cover terminal identity lifecycle schedules

* Align terminal fixtures with canonical identity sources

* Repair runtime fixture source contracts

* Use checked database fixtures for federation contracts

* Update crew source inventories and exit fixtures

* Preserve crew setup outcomes at existing subscription owners

* Match managed restore fixture to its nullable async return type

* Keep absent setup observations distinct from undefined failures

* Drive pending Crew questions from existing metadata

* Preserve nested pairing repair outcomes for shared callers

* Route attached workers to explicit local recipients

* Type Crew pending-question measurement receipts

* Match SSH repair fixture target absence to store contract

* Bound cold terminal layout construction and finite pane acquisition

* test: use public runtime identity in pane acquisition fixture

* Avoid repeated searches in cold workspace layout loading

* Verify cold loader fault identity and hook claimant effects

* Revert "Route attached workers to explicit local recipients"

This reverts commit f1335d1c06.

Keep attached workers on the existing Run-home routing path for this change.

* Omit crew peer addresses outside the active home route

* Keep crew retries on the verified runtime session

* Bound cold layout retirement and recorded-key construction

* Verify original cold layout failure identity

* fix: preserve surviving layout split references

* test: use typed retirement instrumentation properties

* test: preserve pane and retirement fixture contracts with checked types

* Prepare Crew record admission and wake identity changes

* Derive Crew recipient actions from current target owners

* Keep nested Crew process facts within selected acquisition

* Defer Crew legacy facts to consumers and complete raw coverage on success

* Use deterministic time in Crew command fixtures

* Document schema-owned Dispatch row assertions

* test: align Crew authority fixtures with typed project boundaries

* test: place Crew CLI integration in cross-surface project

* Type retained identity test meters with canonical source maps

* Document suffix Dispatch row boundary and preserve fixture unbinding

* test: migrate Crew authority arguments and mailbox observations

* Derive Crew attention and actions from complete recipient authority

* Fix Crew public-command fixture type contracts

* Align Crew fixtures with runtime reader and routing contracts

* Narrow receiving fixture calls to unary RPC methods

* Age pending question fixtures from every actual answer timestamp

* Complete Crew public-command workspace fixture result

* Test delegated helper follow-up through Crew advertised address

* Qualify Crew follow-up addresses through shared command admission

* Complete finite follow-up authority and lifetime evidence

* Assert exact remote authority acquisition error identity

* Consolidate Crew SQL test fixture imports

* Trim Crew read-model tests to the high-impact set

Keep the real CLI/RPC/database command paths, one regression per review-found bug, the cross-version contracts and one N vs 10N operation contract per scaling piece. Drop permutation matrices, equivalence oracles, near-duplicate scenarios and diagnostic output, and return the SSH, managed-server and WSL test files to main, since their production changes were reverted earlier in this PR.

* Drop diagnostic output and a duplicate error case from Crew tests

Remove two leftover console.log diagnostics, an unused 2x2 sample matrix and unused return fields in the follow-up cost contract, and the subscription AbortError serialization case that the subscription identity and preload tests already cover.

* Pass the recorded-leaf reader in main's check-details loader test
2026-11-01 13:28:53 -08:00