Commit Graph
3 Commits
Author SHA1 Message Date
Brennan Benson 88a266e003 Make SQLite the source of truth for native chats (#39256)
* Retire projection adapter tests and use real SQL journal fixtures

* Persist bounded history detail and conversation outline projections

* Select rewind cuts by provider indexes and delete uniform rows as a set

* Cache bounded conversation outlines after reader projection

* Remove retired journal fold and message adapters

* Add the 4a main-vs-candidate native chat comparison harness

One entry (native-chat-4a-comparison.ts) seeds the 55k-history/310-chat
profile once through the seed arm, clones it per arm at a fixed path,
launches each built checkout in the background with an isolated home,
and drives the same stub provider events through the host RPC surface
both builds serve. Envelopes come from each arm's own source bundle.
Rows cover send/queue, 1/8/16 MiB streams with a competing send,
maintenance windows, open/reopen, history/detail, outline/fork,
rewind, record operations, R3 retained-history admission and, reported
separately, first-launch conversion (optional 20 GiB source). Output is
one JSON and one markdown table keyed by both head SHAs.

* Fail the 4a harness launch fast on a non-e2e build and stop the app it launched

* Shim tsx's __name in 4a harness pages and install the probe inside the guarded launch

* Read the native chat working flag from the 4a harness's visible root itself

* Retry transient main-process evaluates in the 4a harness host-work probe

* Use the history page's itemId in the 4a harness and name what a visible-state wait timed out on

* Activate 4a harness chat tabs through their tab group and seed enough smoke filler chats for the unopened-chat row

* WIP: migrate native chat tests to SQL journal readers

* Treat refusals carried inside a successful 4a harness RPC result as refusals

* Give the 4a stub's user echo the uuid its transcript row records

* Remove retired journal writer test machinery

* Chain the 4a stub's resumed transcript to the last message, past a fork copy's title row

* Fork the 4a harness's large chat through its latest response, not its latest prompt

* Port persisted chat tests to SQL fixtures (red WIP)

* Add the 4a main-vs-candidate native chat comparison harness

One entry (native-chat-4a-comparison.ts) seeds the 55k-history/310-chat
profile once through the seed arm, clones it per arm at a fixed path,
launches each built checkout in the background with an isolated home,
and drives the same stub provider events through the host RPC surface
both builds serve. Envelopes come from each arm's own source bundle.
Rows cover send/queue, 1/8/16 MiB streams with a competing send,
maintenance windows, open/reopen, history/detail, outline/fork,
rewind, record operations, R3 retained-history admission and, reported
separately, first-launch conversion (optional 20 GiB source). Output is
one JSON and one markdown table keyed by both head SHAs.

* Fail the 4a harness launch fast on a non-e2e build and stop the app it launched

* Shim tsx's __name in 4a harness pages and install the probe inside the guarded launch

* Read the native chat working flag from the 4a harness's visible root itself

* Retry transient main-process evaluates in the 4a harness host-work probe

* Use the history page's itemId in the 4a harness and name what a visible-state wait timed out on

* Activate 4a harness chat tabs through their tab group and seed enough smoke filler chats for the unopened-chat row

* Treat refusals carried inside a successful 4a harness RPC result as refusals

* Give the 4a stub's user echo the uuid its transcript row records

* Chain the 4a stub's resumed transcript to the last message, past a fork copy's title row

* Fork the 4a harness's large chat through its latest response, not its latest prompt

* Port ACP tests to SQL journal named reads

* Port Claude tests to SQL journal named reads

* Port Codex tests to SQL journals and drop the legacy row rollback test

* Port failure and persisted-read fixtures to the active chat database

* Remove obsolete internal-work counting from chat inventory tests

* test(native-chat): repair SQL epoch fixtures and sender closure case

* fix(native-chat): own async option preparation and port SQL runtime fixtures

* test(native-chat): encode the full founding identity

* fix(native-chat): recover saved live work on SQL open

* fix(native-chat): select retained image sources before history paging

* fix(native-chat): preserve catch-up authority and scoped journal teardown

* Cover the 4a conversion, history-shape and maintenance-debt cases through outside observation

Conversion: seed a medium chat (310 chats on the full profile) and refuse a
cached seed from another recipe. Observe the candidate's existing
epoch-installed ready boundary with a read-only poll of journal_chat_state.
An idle seed copy yields launch to every chat ready and the per-size
ready-mark intervals; the families copy verifies an unconverted first open and
waits for every chat ready before any steady-state family.

History/detail: the stub emits ordinary Claude subagent frames (interleaved
and sparse-root corpora) and a shapes turn (huge completed tool originals, one
above the detail reply cap, an oversized ordinary message). Each named shape is
verified through the real history/detail route before its samples count; diff
items stay explicit missing evidence because the Claude adapter writes none.

Maintenance: write and rewind large completed rows on a disposable seeded chat
under a held outside reader, then measure other-chat sends during the hold,
after release and while freed pages are reclaimed, from read-only page/WAL
state. Retired-epoch pruning is skipped unless such rows are observed.

* fix(native-chat): preserve stopped result disclosure and async fixture ownership

* Resolve lost-session warning identity at the journal writer

* Cover the 4a conversion, history-shape and maintenance-debt cases through outside observation

Conversion: seed a medium chat (310 chats on the full profile) and refuse a
cached seed from another recipe. Observe the candidate's existing
epoch-installed ready boundary with a read-only poll of journal_chat_state.
An idle seed copy yields launch to every chat ready and the per-size
ready-mark intervals; the families copy verifies an unconverted first open and
waits for every chat ready before any steady-state family.

History/detail: the stub emits ordinary Claude subagent frames (interleaved
and sparse-root corpora) and a shapes turn (huge completed tool originals, one
above the detail reply cap, an oversized ordinary message). Each named shape is
verified through the real history/detail route before its samples count; diff
items stay explicit missing evidence because the Claude adapter writes none.

Maintenance: write and rewind large completed rows on a disposable seeded chat
under a held outside reader, then measure other-chat sends during the hold,
after release and while freed pages are reclaimed, from read-only page/WAL
state. Retired-epoch pruning is skipped unless such rows are observed.

* Remove the retired ACP warning-reader fixture field

* Validate warning writer authority before duplicate selection

* Wake queued delivery when pending requests change

* Validate authority before all resolved item selection

* Use current chat test fixtures and name the handover journal owner

* Keep warning authority validation on the semantic selection branch

* test: lose unsaved ACP creations before a successful resume

* refactor: tighten SQL journal types and publication reads

* test: use SQL chat writers and async journal observations

* Replace casts and empty type imports in the 4a harness scripts with annotations and guards

Renderer times and tab ids from waitForFunction handles go through typeof
guards, the visibility probe is a declared Window global, the runtime call
keeps its own typed response, and cached manifests, Electron paths, profiles
and families are narrowed by guards. The empty `import type {}` blocks name
AppStore and use it. Three casts remain, each with a line-specific SAFETY:
the generic RPC result and Node's performance inside main-process callbacks.

* refactor: type handover owners and finite recovery snapshots

* Name the 4a arm's bundled envelope builders by their tracked source files

The protocol bundle imported from './src/shared/agent-session/...', a folder
the path-literal ratchet reads as an unresolved import. Each module is now a
tracked-file constant joined onto the arm's own checkout, the same form
runtime-file-watcher-resource-probe.mjs uses; the bundle and its exports are
unchanged.

* Replace casts and empty type imports in the 4a harness scripts with annotations and guards

Renderer times and tab ids from waitForFunction handles go through typeof
guards, the visibility probe is a declared Window global, the runtime call
keeps its own typed response, and cached manifests, Electron paths, profiles
and families are narrowed by guards. The empty `import type {}` blocks name
AppStore and use it. Three casts remain, each with a line-specific SAFETY:
the generic RPC result and Node's performance inside main-process callbacks.

* Name the 4a arm's bundled envelope builders by their tracked source files

The protocol bundle imported from './src/shared/agent-session/...', a folder
the path-literal ratchet reads as an unresolved import. Each module is now a
tracked-file constant joined onto the arm's own checkout, the same form
runtime-file-watcher-resource-probe.mjs uses; the bundle and its exports are
unchanged.

* Port cross-version wire tests to SQL journal reads

Hosts take the shared domain database, reads go through named async
journal reads and catch-up history pages, and released-build fixtures
type the release's own synchronous journal API.

* Align cross-version downgrade tests with one-way conversion

A downgraded build sees the chat and records as converted and its
later writes are not carried forward. Drop tests whose subject was
this build's legacy row and outcome writes; seed newer legacy rows
before the first conversion.

* Port cross-version wire tests to SQL journal reads

Hosts take the shared domain database, reads go through named async
journal reads and catch-up history pages, and released-build fixtures
type the release's own synchronous journal API.

* Align cross-version downgrade tests with one-way conversion

A downgraded build sees the chat and records as converted and its
later writes are not carried forward. Drop tests whose subject was
this build's legacy row and outcome writes; seed newer legacy rows
before the first conversion.

* test: open runtime fixtures with SQL journal workers

* test: combine runtime fixture imports

* test: prune resolved compatibility timeout finding

* test: use bundled SQL workers for runtime admission fixtures

* test: await SQL reader delivery and complete real rewind cuts

* test: include the medium chat in comparison profile metadata

* test: cover snapshot holds after replacement rollback and read failure

* test: keep clear history assertions on saved and rendered results

* Inject the deletion scenarios' record removal at the SQL record store's reads

The SQL record store has no transact or record-deletion command, so the
authoritative-deletion cases make the store's public record reads answer as
for a missing record and restore them before teardown. Assertions unchanged.

* Align cross-version fixtures with async SQL publication

Wait for frames that follow committed reads, await the async
completion feed, route the outline and item calls to their current
host methods, and expect the restart unreconciled latch on loaded
records.

* test: delete automation records with their command receipts

* Seed completed diff originals through the baseline writer and measure them

The 4a stub speaks Claude stream-json, whose adapter writes no diff items, so
completed diff originals were a skipped row. The seed now appends known
diffs once into its last filler chat through the seed arm's own original
journal writer, after that arm stops and before the seed is renamed or
copied, so every arm starts from the same rows and item IDs. Both arms verify
kind, original length, a short page head and the exact patch from detail
before timing the tail page and the detail read, and the conversion sweep
reports the diff chat apart from the small chats.

* Pin each 4a arm to a full SHA and prove its build came from it

Each --arm now names name=<checkout>@<full sha>. The harness refuses an arm whose HEAD
differs, whose product sources are dirty, whose renderer build-commit stamp is not that
HEAD, or whose sources changed after the build, and records that provenance in every
result. A cached seed from another seed-arm commit is refused, and report-only refuses to
mix results from different pins, seeds or recipes. Tables name full SHAs and disclose that
writer BEGIN-to-COMMIT spans are not observed.

* Establish restart recovery debt in the record writer

* Add active-chat resident memory comparison to 4a harness

* Commit complete provider dispatches and preserve Stop sources

* Coalesce pending publications across provider dispatches

* Capture CI watch actions at SQL writer publication

* Restore Claude task linkage in SQL acquisition facts

* Lift restarted queue holds through SQL admission

* Forward brief clear through Claude close

* Isolate SQL test worker lifetimes and prove fork durability

* Preserve saved queue order and recover restart custody on SQL open

* Keep saved queue contracts explicit and type fixtures accurately

* Run startup chat workspace conversion through its SQL owner

* Make journal command classifications exhaustive and remove unchecked casts

* Temporarily reverse quality cleanup to isolate translation regression

* Restore verified behavior-neutral journal quality cleanup

* Initialize real saved snapshot routes in Claude translation tests

* Repair native chat campaign readiness and contention measurements

* Measure fork contention under isolated retired history pressure

* Verify contention still targets a live stream at dispatch

* Run chat maintenance within the record-store lifetime

* Bound chat activity reads to work transitions

* test: adopt saved snapshots in direct Claude sinks

* Cover maintenance activity during registered starts

* test: initialize saved facts for Claude permission and resume fixtures

* test: await durable Claude resume and chain publications

* Complete provider dispatch identities in maintenance fixtures

* Preserve journal body admission and text close finality

* Scope optional permission backfill failure to its record change

* test: seed SQL reader fixtures through their owning contract

* test: assert saved Codex outcomes through SQL readers

* test: seed replay-budget withdrawals through command turns

* fix: refuse chat access on newer journal schemas

* test: migrate Claude echo wait; fails on known product finding

The retained Working assertion still fails after awaiting initial publication and accepted echo. The product finding is handed to the author.

* test: await SQL transcript and command publications

* test: migrate public rewind cursors and teardown warning scopes

* test: await child-work and status re-entry publications

* test: bind operation fixtures to SQL current-work reads

* test: establish recovery fixtures through owner restart reconciliation

* Keep child work owned across null status publications

* test: await subscriber and late-proof SQL observations

* test: resolve the profile transfer fixture from the repository root

* test: assert restart recovery capsules instead of internal call counts

* test: migrate Claude echo wait; fails on known product finding

The retained Working assertion still fails after awaiting initial publication and accepted echo. The product finding is handed to the author.

* test: await runtime session publications and final writer acknowledgement

* test: migrate coordinator mail SQL APIs; known product failures remain

* test: persist fork sources and use SQL RPC fixture contracts

* test: observe conversation publications in worker abandonment

* test: admit downgrade fixture owners through the SQL store

* test: await ordered attention prompt relay observations

* Keep status observer failures outside provider ingress

* Order provider facts before late delivery receipts

* test: submit replay fixture turns through the general writer command

* test: await durable worker delivery and lifecycle outcomes

* test: follow current provider ownership in coordinator mail

* fix(chat): settle mail custody from writer outcomes

* fix(perf): format retired fixture row count explicitly

* fix(chat): admit automatic worker close from current work

* Require live tool contention witness at send dispatch

* Await attention fixture updates and prune the fixed echo wait baseline

* Preserve the Pi startup exit verdict after recording process identity

* Check restored image deletion through the transcript view

* Adapt history pressure fixture and probes to current item storage

* Repair CI import paths and asynchronous lint findings

* Keep journal storage and projections acyclic

* Preserve proven startup facts across acquisition settlement

* Wait for the owning chat boundary in recovery and delivery tests

* Observe admitted delivery work in native chat fixtures

* Make native chat comparison measurements fail closed

* Isolate SQL worker fixtures across test files

* fix(sqlite): avoid retaining large bindings in cached statements

* perf(native-chat): reuse the captured queue publication

* test(orchestration): await refused mail settlement before recovery

* fix(orchestration): reconcile owned mail before consuming checks

* fix(orchestration): preserve each mail check response

* test(orchestration): diagnose surviving timers through public SQL reads

* perf(native-chat): place item metadata before large forms

* test(native-chat): require the octet-length column flag

* fix(native-chat): version persisted item presentations

* fix(native-chat): bound queued-image history admission

* test(orchestration): update shared mail source support

* fix(native-chat): preserve Stop transitions and admitted forms

* perf(native-chat): select the existing latest-turn index

* perf(native-chat): reuse history admission facts

* perf(native-chat): reuse shortened history wire forms

* perf(native-chat): batch bounded fork metadata reads

* fix(native-chat): synchronize CI fixtures and type mailbox receipts

* perf(native-chat): bound fork scratch presentation work

* test(native-chat): await subscriber catch-up completion

* fix(native-chat): retire readers after fork setup failure

* perf(native-chat): batch bounded fork metadata and output reads

* Reuse validated journal publication inputs on demand

* fix(chat): cancel reader requests on requested shutdown

* perf(native-chat): reuse narrow SQLite work reads

Read canonical current work and queue eligibility in one snapshot, and reuse validated work through the existing journal observation entry. Preserve full status publication, live authority validation, optional queue failures and final writer claims.

Frozen700 matches the reviewed 27-file source manifest and measured tree 6423b1cfbb. Cold and changed-head reads meet the one-point budget; unchanged publication frames improve.

Disclose report698 strict-rule misses accepted by the coordinator as noise: the approximately 4us warm in-memory hit has paired p50 ratio 1.047; one writer-ack cell (empty / 2 subscribers / changed) is +5.5% at p50, 0.259 to 0.274 ms, while completion is approximately twice as fast.

* test(native-chat): synchronize Stop with interrupt entry

The request note is written before the separate dispatch read and native cancel call, so observing the note cannot prove cancelTurn has started. Await the mock interrupt entry, then inspect the request note while its response stays unanswered.

Remove eventual polling and verify the single status note before and after the response without asserting its wording. No production ordering change, sleep, retry or timeout increase.

* fix(native-chat): refresh invalid full work projections

Use the existing SQL authority and context validation before preserving a
full projection, so refreshed narrow work can be reused after row-free
invalidation. Preserve a valid full projection arriving during a narrow
read and extend the existing owner scenario for both transitions.

Correct the description in b98a2544bf: the report698 strict misses are
small measured overhead, not proven noise. Empty warm currentWork p50 is
3.542 to 3.709 us (+0.167 us, 60/80 pairs slower, ratio 1.04715, 95% CI
1.02291-1.07259). Empty/2-subscriber/changed writer ack p50 rises 5.5%,
0.259459 to 0.273833 ms, with its CI excluding parity, on an unchanged
writer path; completion falls 1.546834 to 0.657125 ms. The coordinator
accepts these marginal costs; the strict FAILs remain disclosed.

Frozen714 patch 6a4ded1ee72e8c0134a9a2121fb39bde618213e3f740b4888fd13c8900274620.
Owning 53 tests pass on both configured entries; static gates pass. Private
point attribution verifies repeated narrow reuse and all 92 serving/frame
outputs match. Focused714 against pre-P7 base still measures a small warm
currentWork median overhead; report716 preserves raw samples and CIs.

* fix(native-chat): retain full work projections with saved Stop

Reuse the existing work-answer validity lookup when a narrow read returns.
A full status projection includes the saved Stop, while narrow work leaves
that overlay unrequested. Comparing only the authority work actually read
preserves a valid arriving full projection and retires stale authority or
context without a second cache or comparator.

Extend the held-read owner scenario with a real saved Stop. Private point
attribution shows unchanged full demand going from one reader point to
zero, with full and work outputs equal; invalidation reuse remains one
read followed by three cache hits. Owning 53 tests pass on both entries and
static gates pass. Frozen721 is parent plus two W3 files only; paused SQL
diagnostics work is excluded. Historical P7 benchmark costs remain small
measured overhead as corrected in 12d479e640 and report716.

* refactor(native-chat): remove obsolete work-state read plumbing

Remove the test-only combined work/queue wrapper and its queue callback after the work-state RPC removal. Delete unused projection and queue-list wrappers, keep internal functions private, and point the hydration overlap assertion at surviving current-work and queue owners.

Retain the narrow SQL work path, work-only projection entry and invalidation/Stop reuse guards: subscriber frames, history/current requests and background-task publication still demand them. Combined work and queue eligibility plus resolved-work reuse remain in production.

Validation: 7 owning files, 26 cases on direct Node and configured Bun entry; full configured Node typecheck and static gates pass. Focused parent-versus-change micro preserves exact output equality and point shape across 36 live queue/frame cells and 80 pairs; no paired interval excludes parity. Removed RPC work-state measurements are historical only.

Prior P7 strict misses remain disclosed small measured overhead: warm current-work median 3.542 to 3.709 us (+0.167 us, ratio 1.047; 60/80 pairs slower), and empty / 2 subscribers / changed writer acknowledgment 0.259459 to 0.273833 ms (+5.5%, CI excludes 1) on an unchanged writer path; completion 1.546834 to 0.657125 ms. Neither cost is established as noise.

* test(native-chat): preserve released chat queue publication coverage

Use the surviving queue publication owner after rewind and reopen,
preserving visible card persistence checks across both providers and all
three release cohorts without a production wrapper used only by tests.

Validation: all 49 cross-version and owning files pass on Node and the
configured Bun entry (148 passed, 1 skipped each); the e2e typecheck
ratchet has zero test errors and the owning static gates pass.

* Register native chat SQL workers with the main-blocking ratchet

The journal SQL writer and reader entries run on worker threads. Their
dependencies are grandfathered in the SQLite baseline as worker-only, the
same way existing worker dependencies are; none run SQL on Electron main.
Prune baseline rows the branch no longer needs.

* test(orchestration): hold native admission in mail reconciliation busy chats

The file-local busyChat answered turn/start, so main's automatic mail
steering (#41476) handed each queued mail card to the accepted turn about
80 ms after send. Cases that wait for a visible card then timed out (or
found it gone), and teardown logged the closed SQL reader. Reuse the
custody fixture's busyChat and settleTurn, which keep admission
unanswered until the scenario settles the turn.

Bun: 4/7 failed before; 35/35 passed across 5 runs after. Node: passes
with the custody file (18/18).

* test(agent-session): wait for the at-rest opening snapshot on the open stream

Subscribers queue the opening snapshot behind an asynchronous SQL work
read, so agentSession.subscribe can reply before the snapshot frame; the
queued-image wire contract depends on that ordering. The at-rest case
read frames only up to the reply and lost the race under CI load.

With a 50 ms opening delay injected locally: old assertion 3/3 failed
(expected undefined), new assertion 3/3 passed.

* Correct subscribe comment: the opening snapshot follows the reply

* test(agent-session): assert no agent start after the at-rest opening snapshot

* test(agent-session): wait for the status summary's own publication in the child-records switch

The session-list summary publishes after an async journal projection read that
neither flushStreamedEvents nor the strip's batch covers, so asserting it right
after either raced the SQL worker.

* fix(explorer): resync a recovered remote watch only after the host watcher is installed

The contact-regained resync fired when the files.watch subscription was accepted, before the
restarted host reported its watcher ready, so a file created in that gap was neither in the
re-read listing nor reported by the watcher. The managed-orcad explorer recovery spec now waits
for that resync before writing its probe file.

* fix(explorer): re-read a recovered remote explorer once when its host watcher fails to install

* test(explorer): the recovered watch resyncs after its ready signal

* test(agent-session): await the queued-operation results instead of polling a settled flag

The operation-order cases polled a settled flag and the compact spy with
vi.waitFor's 1 s default. Under SQL worker contention the goal or rewind
answers after more than a second, so the poll gave up while the operation
was still on its way (CI: "expected false to be true", then "closed by
quit" from the unhandled result). Await the result itself and a promise the
compact stub resolves.

With a 30 ms SQL reply delay injected locally plus CPU load, 3 repeats: the
old file failed "releases a startup goal" (expected false to be true) and
"answers a goal" (compact called 0 times); the awaited version passed all.

* fix(agent-session): reserve queued sends before a waiting operation's own place

DeliveryOrder.reconcile became async when the first-queued read moved to the
SQL readers, but the ready-operation path still called it without awaiting
and reserved the operation's place right after. The operation then took its
place before the queued send it was meant to follow, so a goal, rewind or
other operation requested while the agent starts could run ahead of a
message sent earlier; a failed read also became an unhandled rejection.
Await it, restoring main's order (reconcile, then reserve).

Found by typescript/no-floating-promises over src/main/native-chat.

* test(agent-session): let delivery park before the attention case's idle sweep

The single idle-sweep tick keeps a chat whose delivery loop is still
running, and the loop parks only after SQL reads that follow the second
send's completion event. When the tick ran first it touched the session
instead of retiring it, and the 1 s hasSession poll failed. Wait for the
loop to park (wake resolves when it does), as goal-steering does.

With an 8 ms SQL reply delay injected locally plus CPU load: old 2/5
failed (expected true to be false), new 5/5 passed.

* test(agent-session): await the fake Codex child's requests in compact-exit

Restarting after a mid-command exit runs the exit settlement, a fresh
acquire and the queued send's handover through the SQL workers before
turn/start, which outran vi.waitFor's 1 s poll under load. The shared fake
Codex now resolves nextRequest(method) when a child receives that request;
compact-exit awaits it instead of polling the calls list.

Seen once in a contended local sweep (8 ms SQL reply delay plus CPU load):
"expected [] to have a length of 1" at the turn/start poll. The new
version passed 5/5 under the same delay.

* fix(agent-session): keep a settled send's frame over an older work restatement

Subscriber delivery reads current work before the history page, then lets
that work restate unanswered sends over the page's rows. When a send's
settlement committed between the two reads, the page carried it accepted
but the earlier work restated it pending, so the frame at the new cursor
published the stale state. Later caught-up frames restate only still
unanswered sends, so the client never learned of the acceptance: the
Claude echo case kept a "pending" bubble although the journal held it
accepted. A restatement only describes sends its read saw unanswered, so
an accepted or rejected page row is newer and now wins.

structured-agent-session-claude-echo-working (failed once in CI, never
reproduced) with an 8 ms SQL reply delay injected locally plus CPU load:
before, most repeats stayed pending for 15 s while the stored row was
accepted at the subscriber's own cursor; after, 7/7 passed in ~100 ms.

* fix(native-chat): record the original journal copy identity

* chore(native-chat): grandfather the writer worker's one-shot source receipt

The receipt is captured once inside domain initialization, which runs only in
the SQL writer worker, so its synchronous stat and SQLite write never run on
Electron main or the orcad loop.

* test(native-chat): await current-request reads instead of polling for one second

The child-request case waited 1 s for the SQL reader's current-request
answer and failed on a loaded CI shard. Wake on each real state change
from the subscription and the request reader instead.

* fix(native-chat): re-answer current work when a catch-up page passes its cursor

A subscriber catch-up answered current work at one journal cursor, then read
the history page in a later reader transaction. The restated copy of a send
still pending at the work cursor replaced the page's accepted row, so the
renderer kept showing the send as sending.

Re-answer work when the page cursor has passed the work answer, refresh the
fan-out's shared work answer once the journal adopts a newer commit, and keep
the page's own rows when the page holds a commit not yet adopted.

* fix(native-chat): read subscriber page rows and current work in one reader snapshot

The catch-up overlaid a work answer from one journal cursor on a history page
read at another. Re-answering work afterwards still left a page that ran ahead
of the adopted cursor without its work, so a send shown as unknown could
revert to pending under an unchanged work fingerprint.

The history read now answers current work in its own transaction. The main
thread validates that answer against live owners and handover with the
existing currentness check, and reads the frame again until it has adopted
the frame's cursor. A cached answer is reused only when the page head is that
same cursor. The fan-out's queue now follows the work answer it was read
with.

* fix(native-chat): capture each subscriber frame at one held command-lane position

A subscriber frame paired rows, current work and the queue from different
positions, and waiting for the main thread to adopt a newer head re-read the
page until it did.

Each frame now captures its page and work (one reader transaction) and its
queue inside a single readInOrder on the chat's command lane, then releases
the lane before emitting. The queue reaches the frame through the explicit
queuePublication option, so emission never reads a newer one. A capture
whose live authority changed mid-read is discarded without advancing the
subscriber; the change's own publication delivers its replacement. Inactive
subscribers, closed journals and replaced session handles abandon the frame.
The resume path no longer decorates stop notes with work read before catch-up;
they restate the work of the frame the subscriber caught up to.

* fix(native-chat): validate whole captured frames and capture every work overlay

The queue gate inside a frame capture re-read work until it matched live
authority, holding the command lane, and the capture stopped validating
before its queue read. Stop-note pages, history replies, attach hydration
and queued-image decoration still paired rows and work from different
positions.

The queue gate now reads once at the frame's own authority, and outside a
capture once at the live one; a moved authority abandons the frame for the
change's own redelivery. The capture re-validates authority, handle and
cursor after its queue read. Stop-note pages are read on the lane only
while the journal is still at the delivered frame, otherwise the resume
catches up first. History replies and attach hydration use the same held
capture, and image decoration that finds the journal past its frame is
abandoned in favour of the newer commit's delivery. With every overlay
coherent, a settled page row can no longer meet an older restatement, so
the settled-send guard is removed.

* fix(native-chat): owe restatements after an emitted replacement and refuse exhausted captures as unavailable history

* test(native-chat): follow captured history frames in cross-version and queued-image wire tests

* Remove the converted legacy chat journal after its grace period (#42024)

* fix(native-chat): retire certified legacy journals after the grace period

* perf(native-chat): pace source proof through existing maintenance

* test(native-chat): exercise retirement with the native source lock

* fix(native-chat): account for queued card epochs during source proof

* fix(native-chat): settle retained journals and cancel stalled source leases

* fix(native-chat): keep the prepared retirement state narrowed

* fix(native-chat): apply retirement backoff before writer admission

* fix(native-chat): retry restored legacy retirement eligibility

* fix(native-chat): retain pending legacy eligibility helpers

* chore(native-chat): grandfather the writer worker's one-shot source receipt

The receipt is captured once inside domain initialization, which runs only in
the SQL writer worker, so its synchronous stat and SQLite write never run on
Electron main or the orcad loop.

* fix(native-chat): make retirement filesystem checks asynchronous

The four maintenance SQLite uses run only in the SQL writer worker and the 28 proof-page SQLite uses run only in the SQL reader worker; main-side proof coordination dispatches worker requests.

* fix(native-chat): build lock helper at the server glibc floor

Build Linux server helpers with the existing pinned glibc 2.28 slot images when the host is newer. Keep native floor and musl builds, and select the separate desktop toolchain explicitly. This prevents fast Ubuntu prebuilds from linking GLIBC_2.33/2.34 into a 2.28 payload.

Bound the helper PR workflow to 20 minutes and cover compiler selection without compiling native code locally.

* fix(native-chat): honor read-only retirement ownership

Do not acquire a writable retirement lifecycle for a newer-schema database, so shutdown never sends its read-only writer a retirement-credit command. Keep the existing preservation and empty-diagnostics regression unchanged.

Select the native exclusion test with the dedicated workflow flag instead of generic CI, and merge duplicate database imports in the retirement test.

* fix(native-chat): remove source proof table assertion

Annotate the content table inventory as readonly strings so membership checks accept schema table names without a type assertion.

* fix(native-chat): bound legacy retirement proof work

Persist source-intrinsic refusals for the exact copy, source and destination descriptor. Resume completed proof work after transient failures, and let the existing conversion owner finish pending imports.

Use covering-index key pages for history membership and exit terminal retirement states before writer admission. Preserve originals and prepared unlink manifests.

* fix(native-chat): recover legacy proof without repeating source work

Use the fixed profile-copy columns for source certification and keep damaged destination inventory retryable. Advance the shared refusal policy so earlier false inventory results are re-proved.

Complete verified empty heads after transient conversion failure, and carry proven original-source diagnoses through SQL errors without changing their ordinary identity. Persist only source-intrinsic refusals.

Validate in-place schema recovery, bounded empty completion, and malformed/newer histories at 128/1280 rows with zero retirement proof, conversion, restore or helper work across subsequent owners. Focused tests, tc:node, scoped checks and all 27 ratchets pass.

* test(native-chat): trace cursor resume frames on failure

Capture synthetic subscriber frame payloads and emission stacks when the existing restart assertion fails in CI. Keep the two-frame expectation and production behavior unchanged.

* test(native-chat): allow held-cursor roster frames on resume

A restarted host can publish an empty task and command roster before catch-up when its initial coherent capture is superseded. Validate that optional metadata frame, preserve the two history/work frames and no-replay checks, and await the work restatement instead of polling packet counts.

Remove temporary per-frame stack tracing now that CI identified the publisher. Production wire behavior is unchanged.

* test(native-chat): assert empty resume batches independently

Use literal empty arrays in the resume contract so a defect in the host's batch constructor cannot also change the expected value. Keep the optional held-cursor roster prefix and the two required catch-up/work frames.

* test(native-chat): expect the current conversation fence on resume

The roster publisher reads the conversation fence after restart and send, which can advance beyond the earlier runtime lease fence. Derive the expected fence from that current record and preserve every other resume assertion.

* test(native-chat): hold the domain database lock while a revoked history step is dispatched

* test(native-chat): accept held-cursor state frames before a resumed catch-up and fold them in a released client

* Start the chat journal writer earlier and defer maintenance until serving (#42045)

* Boot the chat journal writer during startup waits

Start the existing record-store slot after login env and before the daemon
wait in orcad, and right after runtime construction on Electron, through one
shared location builder that probes for saved chats once. Startup restoration
consumes that same attempt, so a refused journal is not reopened before RPC.
orcad's push-service start moves to its own module to keep the entry small.

* Start journal maintenance after startup restoration

The CPU profile showed the background sweep's reader spawn and writer
work competing with pre-RPC startup for the disk and CPU. A startup
open now holds maintenance until its restoration settles; other openers
start it on open as before. Reads still convert saved chats on demand.

* Move the headless-serve browser check to its own module

Behavior-neutral; keeps main-process-runtime-launch.ts within the line limit.

* Collect saved chat ids once and start journal maintenance after serving

orcad passes the ids it collected for the early open into runtime
construction instead of collecting them again. Background journal
maintenance now waits for the host's serving milestone (orcad readiness
published, Electron serve ready, desktop RPC start) rather than
restoration settling, so a stalled renderer cannot hold it. The desktop
early open starts after the first-window services fence. The refusal
test now observes that the journal is opened once.

* Keep the deferred desktop chat open inside shutdown, and start CI-line watches after serving

A first-window fence that settles after quit or a failed startup has
committed no longer opens the chat journal past its teardown; the
Electron startup-failure path now commits host shutdown as orcad's does.
Host installation re-arms CI-line watches only after a teardown stopped
them, so the first run starts at each host's serving milestone (orcad
after readiness, Electron after startup settles) instead of during
restoration.

* Admit CI-line watches at each host's serving milestone and let installs retry them

The first watch start no longer depends on renderer load: orcad admits it
after readiness, Electron serve after the serve-ready line, desktop once
the window and RPC start settle. Before admission, chat installation
leaves watches to that milestone; after it, installation's idempotent
start covers a failed first start as well as a teardown. Starts are
skipped once host shutdown has committed.

* Fence desktop watch admission, make warmup retry-only, and release failed watch starts

Desktop admits CI-line watches after the bounded first-window fence, so
automatic journal work never precedes it. The startup warmup timer only
retries a failed start; admission stays with each host's serving
milestone. A watch service whose start rejects after registering its
outcome observer and job hooks is stopped before a retry replaces it.

* Retry a failed CI-line watch start on the journal's maintenance pass

After admission, a saved-chat host whose first watch start failed had no
automatic retry. The journal's existing periodic maintenance pass now
calls the idempotent start when watches are admitted but neither running
nor starting; it does nothing after committed shutdown or while a service
runs. No new timer or loop, and nothing moves before readiness.

* Wire PR-watch recovery into every record-store opener

The chat host's install could open the once-slot first, without the
maintenance-tick hook, leaving a failed first watch start with no automatic
retry. The hook is now a required location field, set through one shared
builder by every production opener.

* Keep runtime identity state within the file-length limit

The saved-chat startup open's import now uses one namespace line, so the
file stays at the 300-line cap after the early-open field and method.

* Load the real chat host once before the PR-watch tests

The host-install-first regression loaded the real chat host module inside
the test, which CI's in-test vi.importActual gate refuses. It now loads in
beforeAll; the test body and its assertions are unchanged.

* Mark the desktop serving test's runtime stand-in as a reasoned cast

CI's changed-code gate requires the lint suppression form for a test cast;
the stand-in and its SAFETY reason are unchanged.

* Share journal database open options so the file stays within its line limit

* test(chat): restore released-client resume projection coverage

* test(chat): type the released-client fold accumulator

* test: fail the first opening subscription data frame

* test: update benchmark module paths after main sync

* test: resolve benchmark modules within each checkout

* test(native-chat): inject late receipts without a history read
2026-11-14 05:51:16 -08:00
Neil 7d76028a5c refactor: group agent session journal and request modules (#36882) 2026-10-31 22:33:23 -07:00
Brennan BensonandCodex 2cc33b8d89 fix(native-chat): save each chat change as keyed records and reopen from the newest (#36024)
* Save native chat state and restore only its journal tail

Publish versioned reducer checkpoints atomically, checkpoint through the chat write queue, and backfill closed chats without retiring journal rows.

Co-Authored-By: Codex <noreply@openai.com>

* Keep checkpoint recovery durable and skip unchanged captures

* Keep journal timer and restart fixtures compatible with checkpoints

* Keep checkpoint performance tests focused on operation contracts

* Remove unused benchmark import

* Restore journal append interception and control Stop fixture time

* Compare saved reducer state without a key assertion

* Supply thread scope in checkpoint test appends

* Prove restored journal consumer paths

* test: exercise restored consumer backward pagination

* fix: pace journal backfill with a monotonic clock

* refactor: keep journal opening cursor checks together

* refactor: project chat journal state in the append transaction

Track isolated reducer mutations and persist keyed changes under a savepoint before adopting committed state. Keep appends working through projection lag and repair coverage on the write lane, including legacy bootstrap.

Co-Authored-By: Codex <noreply@openai.com>

* fix: keep reducer objects live across journal commits

The draft isolates every write until COMMIT, then writes each dirty copy back
onto its original, so submissions, items and queue cards handed out before a
commit keep reading current values as they did with in-place folding. A failed
COMMIT still leaves live state untouched. Also replaces banned Reflect.get uses
and adds braces the lint rule requires.

* fix: bound projection repair and check the invariant after notification

Failed publications and savepoints back off from 1 s up to an hour, reset by a
healthy incremental write or a reopen. open() no longer waits for a legacy
chat's publication; the scheduled repair publishes it on the write lane. A
failed open closes its handle. The equality check runs after commit
notification, only when projection tests opt in or, log-only and sampled, in
development, and it requires the saved coverage to match. Removes the unused
streamed commit argument.

* fix: keep the background projection pass proportional to chats

The orphan sweep reads one state row per chat and clears orphans by session
key. A chat whose fold cannot publish stops at its first damage and is skipped
until its epoch or newest row changes.

* fix: version the keyed projection layout as codec 2

This branch's earlier whole-state layout used version 1; it now decodes to null
and the chat replays.

* test: follow the append-with-fold contract and deferred publication

A submission whose fold throws now rolls back with its receipt before any
dispatch, so those sends are refused and nothing is recorded. The unconfirmed
continuation outcome is pinned through its real cause: the agent admits the
message and no delivery proof arrives. Tests that need saved state right after a
chat opens publish it explicitly, the history reload clears the saved state
covering its edited row, and the codex stream bound matches its capture.

* test: pin projection repair backoff, background pass and N/10N costs

Covers the production repair timer under a persistent failure, a legacy open
that publishes after it resolves, a failed open leaving no timer, a damaged
closed chat skipped until a new row, and equal per-write, reopen, sweep and
closed-chat publication work at N and 10N. Projection tests opt into the
per-commit equality check.

* refactor: inline draft write-back helpers

The repo lint bans object-typed parameters; the copy and write-back now run in
the closure that already holds the wrapped value.

* fix: write a new epoch's saved state in its own transaction

New chats, rollovers and rewinds publish the new epoch's projection beside its
rows, inside the same statement-only savepoint the append path uses, so they
start covered instead of waiting a second for a repair. A projection failure
there still keeps the epoch and leaves the scheduled repair to publish it.

* fix: keep the sampled development check log-only on any failure

A read failing inside the check must not reject an append that already
committed and notified.

* fix: typecheck the draft copy without closure narrowing

Assign the lazily created copy through a local so strict programs that do not
narrow a captured variable after ??= still see an object.

* test: catch whole-table sweeps and an undecodable projection

The sweep contract now records exec statements too and pins its exact keyed
statement list, so a NOT EXISTS scan through exec fails it. A projection whose
item stops decoding fails the per-commit check on coverage although a full
replay still matches. Also fixes two test lint findings.

* refactor: drop draft casts the types can carry

Overlay maps and sets finish from their own typed touched keys, and present map
entries are read through one helper. The two remaining generic boundaries, a
draft standing for its object and finishing back to it, carry the repo's SAFETY
directive.

* test: show a parked damaged chat publishes after a clean epoch replaces it

The replacement ends at the parked chat's newest sequence, so only the epoch in
the skip key re-arms it.

* test: give the seeded queued card its enqueue cursor

Production records queuedAt as the cursor before the queue row; the fixture
now does too.

* fix: catch lagging saved chat state up in atomic pages

Repair of a legacy, lagging or undecodable chat, open or closed, no longer
republishes the whole chat. One runner folds the rows after the saved coverage
onto a private copy of the saved prefix, at most 32 rows or about 1 MiB of row
text per page, and commits each page's changed entries with the new coverage in
one FULL transaction. That transaction first checks the chat's epoch and the
exact saved row the page was folded from, so a page from a retired epoch or a
stale base writes nothing. Pages rest between slices, and the saved prefix is
decoded in pages too.

An open chat runs the pages on its write lane; sends keep committing between
them and skip the projection until coverage reaches the tip, then project
incrementally again. The background pass hands a chat over to its open owner at
a page boundary. Damage parks the chat after the pages before it, and a failed
page keeps the committed ones and retries from them on the existing backoff.
Whole-chat publication now runs only inside epoch transactions.

* test: pin the draft layer's tracked nodes per streamed revision and acceptance

A transition now reports how many objects and collections its fold reached. The
scaling chat gains accepted sends (receipts and aliases) and waiting, dispatched
and returned queue cards at N and 10N. A streamed revision's only growing term
is the card scan, one node per card plus each returned card's body; an echo
also reads every item with its body and every submission.

* test: append the between-pages send by item identity

appendItem takes the provider identity, not its key.

* test: check every catch-up prefix of the recorded Claude and Codex streams

A runner can page single rows, so the recorded streams' aliases, Stop settlement
and per-row timestamps are each checked at a committed prefix against a full
replay.

* test: measure each backfill pass's row reads on its own

The parked-chat test counted its own replay checks: 64 pass rows plus 68, 68
and 100 oracle rows made the 300 CI saw. Each timer window is now measured
alone, and coverage is read from its saved row, so the unchanged pass reads 0
rows and the re-armed one reads one page.

* fix: bound catch-up reads, release the finished prefix, and keep unreadable views

- A catch-up page now stops reading rows at the row whose text reaches about
  1 MiB, so rows past the budget are never fetched and fetched again; saved-view
  item and entry pages use the same budget.
- An open chat drops its private prefix once coverage reaches the tip and keeps
  it only across a failed or parked page.
- A present current-epoch view this build cannot read (another codec version or
  corrupt content) is no longer cleared by the first page. Its whole history
  must fold cleanly through the tip first; otherwise it parks untouched. A
  storage error while reading the view fails with backoff instead of reading
  as a missing view.
- The pace charges only the runner's own work units, so time spent waiting on
  an open chat's catch-up or the write lane no longer lengthens the next rest.

* test: cover takeover, epoch replacement and stop in the middle of a pass

A chat opened while the pass rests is caught up by its own runner from the
committed page; the pass pages it no further and does not charge the wait to
its pace. A replacement leaves its new view untouched, and a stopped pass
resumes after its last committed page.

* test: fail the takeover test without the handover break, and the replacement test without the stale-runner guard

The takeover test asserts the pass asks the open owner only twice for the
handed-over chat, and pins performance.now so the open runner's pace never
waits on a fake timer. The replacement now lands while the open runner is
resting in its saved-view decode, so its page reaches the lane after the
replacement; the test asserts no transaction opens.

* test: read saved-view pages with the page budget, and await database closes

The paged-decode round trip passes the row-and-text budget and checks the
view decoded. Projection, scaling and ACP launch tests await the now-async
test database close, so a closing checkpoint worker cannot close the next
test's database.

* Persist native chat outcomes with a rebuildable cache

Store immutable keyed transition results with canonical ordering, fence outcome epochs, and convert legacy inputs through paced invisible generations. Rebuild disposable caches from validated outcomes and read sparse logical sequence groups without changing the wire.

Co-Authored-By: Codex <noreply@openai.com>

* test: repair the split suites so they parse, lint clean and find their setup

Restores the describe blocks the split truncated, the loop variable it renamed, the
fixture references and imports it dropped, and a stray argument pair. Moves the
grandfathered flaky-pattern counts with the code that moved.

* test: read outcome epochs as outcomes where the adapted assertions read input rows

A replaced epoch restates its Stop in the saved pause marks, the sparse-group batch
test needs an outcome epoch rather than the input-format fixture, and a revised item
keeps its first place, so the held-write order is read from the replayed groups.

* test: expect saved queue cards as whole values in the queue-history assertions

The adapted helpers read card outcomes, which store each card's whole state, so an
import or refusal shows as the card's own state change rather than as an input row.

* style: format the files this PR changed

* test: import each module once in two journal suites

* fix: decode saved outcomes with a frozen format-1 schema, independent of the cache codec

Outcome values decode through a versioned format-1 schema that never follows the live reducer type; an unknown field reads as saved by a newer Orca, and the cache codec only delegates to it, so a codec bump rebuilds the cache and nothing else.

* fix: key outcomes by an integer generation and keep the cache stamp in its state row

Each outcome row names its generation, the attempt's id, instead of repeating the chat id, epoch and a UUID; the cache stamp is a column of the cache's own state row, so there is one stamp to keep in step; cache ordinals are text like the outcomes they copy. Orphaned caches are swept from the per-chat state row with indexed per-chat deletes. The WSL census lists both schema files.

* fix: finalize queue ownership before publishing a new epoch, and reject unreadable outcome rows

Epoch replacement and rollover adopt the committed state, keep attachment claims and finalize the queue receipt before the fallible publication step, as before outcomes. Every persisted outcome row is read back through the row reader, so a row it rejects fails as journal_row_rejected. Publishing an epoch checks the epoch it replaces is still current. A key deleted and re-created in one write moves to the end of the pending changes, matching its new ordinal.

* fix: open a chat synchronously and unpaced, and classify read failures by cause

A user's open reads the cache or rebuilds from outcomes in one synchronous pass, like replay on main, so no background rebuild can interleave with it and nothing paces or bills it. Only an outcome decode failure is damage; a value from a newer format is refused as newer; storage errors propagate as unavailable. Writes derive cache currency from the stamp instead of a latch and ask the maintenance runner when it lags.

* fix: run all journal maintenance through one paced runner per database

One runner serves requested chats first, then lagging ones, one catch-up at a time, open or closed, so an open chat keeps its conversion's progress. A busy or replaced chat is skipped and the pass continues; content that cannot convert parks until the chat's rows change; storage failures back off. Background work alone pays the duty cycle. Conversion verifies the whole input once, then only the outcome tail; cache rebuilds copy outcome pages to the moving tip and stamp from the scalar outcomes.

* test: adapt journal suites to generation-keyed outcomes, the merged cache stamp and the publication order

Queries read outcomes by generation; the bootstrap test expects the queue receipt to finalize before publication; queue-import-scaling states the reopen's constant two key visits when it retries a failed import; append-delivery reads the outcome change index's item ids.

* test: pin outcome format 1 with checked-in rows, and cover open consistency, transient reads and re-created keys

A golden fixture of format-1 outcome rows covering every field and scalar must restore to the pinned chat; a cache codec bump only rebuilds the cache; an unknown field reads as newer. An open between two pages of a cache rebuild reads the whole chat from outcomes. A transient read error during open is unavailable, not damage, and the next open loads the chat. A key deleted and re-created around a new key in one row keeps live order, written or converted.

* refactor: check a converting chat's epoch once, in the catch-up step that runs each page

The catch-up reads the chat's epoch in the same synchronous step as the page's transaction, for conversion and cache rebuild alike, so the conversion's second copy of the check is dropped.

* test: drive the maintenance runner over a real database, each guard killed by a mutation

Startup sweeps, closed chats converted one at a time with a busy chat skipped, an open chat writing between conversion pages, a stop between pages, an epoch replaced mid-conversion, content parked until the chat writes again and then delivered, and a storage failure backed off rather than parked.

* test: N/10N contracts for outcome appends, opens, conversion, publication, idle passes, sweeps, tail verify and wire catch-up

Each counts statements through prepare and exec, rows read per table and rows and bytes written, never time, and requires the same work for a chat ten times larger: a streamed revision, a Codex plan update and an acceptance; a cached open reads no outcome; a conversion page reads one page of input and publication none; an idle background pass reads no history; a retired generation and an orphaned cache are deleted one indexed page at a time; a second verify reads only the new outcomes; a wire catch-up reads only the groups asked for.

* test: shipped builds keep a new or converted outcome chat untouched

The newest release refuses the chat as a newer Orca's and leaves every byte of the database as it was; a build from before that reason opens it read-only with nothing shown, refuses writes, and changes no saved row.

* test: an epoch publication over an epoch the caller did not see is refused, and damage at row 1 still yields to a newer row

Publishing a new epoch checks, inside its transaction, the epoch the caller last saw, and writes nothing when it changed. A chat whose first row is unreadable keeps reading for a newer build's row, as main does, and is refused as a newer Orca's.

* test: old-build probes fold input rows only, and the interrupted-call probe writes inputs

* test: import the state layout once in the cache test support

* fix: type the tail verify's live lookups per collection, and two test call shapes

The tail verify looked keys parsed from stored JSON up in a union of live maps, which does not typecheck; each format-1 collection now has a typed lookup that narrows the parsed key first, and the table is checked complete against the format's collection list.

* fix: wake the maintenance runner for a request or a due retry ahead of the periodic pass

One timer now wakes the runner at the earlier of the next periodic pass and the first due request or storage retry, re-derived from the waiting chats after each pass, so a request after the first pass is served within a second and a backoff retry is no longer replaced by the six-hour pass. A chat whose load swept a retired page keeps its turn instead of waiting for the next pass. Tests drive start(), requests and timers on fake time.

* test: pin the draft's tracked nodes per streamed revision and acceptance at N and 10N

The scaling chat gains dispatched and returned queue cards. A streamed revision's only growing term is the queue card scan, one node per card plus each returned card's body; an acceptance echo also reads every item with its body and every submission. Main's reducer makes the same scans on these rows; the draft adds one node per value reached.

* test: cross-version probes read a new chat's turn end as one outcome sequence, and the kept-card downgrade uses an unconverted chat

* fix: store every new write as format-tagged outcome records after an existing chat's rows, with no conversion

An existing chat keeps its input rows byte for byte; its first write appends one marker row that
shipped builds refuse, and its records follow. New chats hold only their marker and records. Each
record carries its own format and the epoch keeps the highest format it holds, checked before a
cache is used or anything is written. Opening a chat with an input prefix folds the prefix as main
does and applies the records over it, deletions included, keeping prefix keys in place; catch-up
pages the prefix rows, then the record groups. Background conversion and its attempts, staging,
verification and publication are gone; the maintenance runner rebuilds only caches of chats without
a prefix, parks a rebuild on its epoch's records rather than the newest sequence, and looks a
requested chat up by its key.

* test: pin the marker, prefix-plus-records reopen, record admission before the cache, cache-only runner parking and N/10N work

Input fixtures stand for a chat an older build wrote: their input epoch keeps taking input rows, so
the input reader still meets writer-made rows; epochs this build founds are written as it writes
them. Conversion tests give way to a mixed chat whose prefix keys are deleted, recreated and updated
in place, the runner's rebuild parking that later writes never re-arm, and contracts for writing
to, opening and rebuilding chats at N and 10N.

* fix: refuse a chat whose outcome marker is a newer version as a newer Orca's, not as damage

* test: shipped builds refuse an existing chat at its appended marker, its input rows unchanged, and this build reads it whole again

The probe writes three items as input rows the way an older build did, then this build's writes
append one marker and outcome records; the newest release refuses the chat as a newer Orca's with
every byte unchanged, the writable baseline latches read-only at the marker, and the re-upgraded
build reads all six items with the prefix rows as they were.

* test: read the runner's rebuild instances as rebuilds, so the parked-chat test typechecks

* fix: admit saved outcome records and the cache only through a strict storage schema

A saved record or cached value holding a key or union tag this build does not declare, at any depth, now refuses the chat as a newer Orca's before the cache is used or anything is written. The storage schema reuses the wire's shared definitions, completes the keys they leave out (failure facts, senders, edit patches and the rest), and closes every object and open union; string vocabularies and opaque payloads keep the wire's meaning. A compile-time check fails CI when a persisted type gains a key storage does not declare, and a checked-in corpus of every declared shape replaces the schema fingerprint, so an added optional key needs no new format.

* fix: refuse an input row or legacy card holding a key this build does not declare before folding it

Restating a chat on a new epoch is safe only if the open chat holds nothing this build half understands. An input row of a known kind (an existing chat's frozen prefix, with or without outcome records after it) is now checked key by key against what this build declares before main's reader normalizes it, and refuses the chat as a newer Orca's; values keep main's checks and execution options their legacy meaning. A legacy queued card whose saved body holds an undeclared key stays on disk, unadopted, as an unreadable card does. Client sends already arrive through strict request schemas, and their options through a reader that drops choices this build does not use.

Main's tests that kept a newer build's unknown body kind, subject or key readable and writable now expect the refusal; the three that drove a newer Orca's approval or item through this build's own writers are removed, since this build can no longer hold one. A checked-in input prefix pins how this build folds older rows, and every value it folds into is one storage admits.

* test: a persisted type holding a key storage does not declare, nested included, fails to compile

* test: this build refuses a newer build's body or subject kind as a newer Orca's, keeping the row an older build deletes

* fix: fold each outcome write from the row a reload reads, so an annotation the reader drops never fails the strict record write

An ACP usage report with size 0 (allowed by the protocol) reached the model as an unusable window, the strict encoder refused it, and the agent was stopped mid-turn. Main writes that row and drops the annotation on reload; folding the read-back row keeps live state equal to that reload for every annotation the reader drops.

* test: an echo declared without a known original is stored as null, the shape a reload reads

* test: probe the downgrade refusal on both releases in the downgrade window, not only the newest

* test: both releases in the downgrade window write a Claude and a Codex chat through their own writers, and this build admits every row and opens the chat as that release had it

* test: rewind, replacement and rollover of a chat holding old rows, the marker and records keep what the user kept and delete the retired epoch whole

* test: phone and paired catch-up over a chat holding old rows then records, from a cursor inside the old rows, draws as the clients render it

* test: ACP launch decisions read an outcome chat from its cache and from its records alone, and a newer one as unreadable

* Open new and mixed chats from each key's newest record and drop the saved-state cache

The cache copied each key's newest record into a second set of tables and needed stamps, a paced
rebuild, a savepoint fallback and a repair runner to stay usable. The records' key index already
answers the same read: an open seeks from one key to the next, taking each key's newest record
(deletions included) and never reading superseded revisions. A mixed chat still folds its frozen
input prefix first. A send now writes only its records, the epoch's coverage and its receipt, so
any failure among them refuses the send as main's row insert does.

The background runner keeps only the paced sweep of retired generations, its one remaining job.
The cache tables are no longer created; they never shipped.

* Derive an outcome chat's last sequence from its coverage instead of recording it

Every write recorded the sequence scalar beside its own records, about half the records on
streamed paths. The epoch's coverage already holds it, so a read supplies it from there and a write
no longer stores it. Records an earlier build stored still decode.

* test: count strict admission per open, catch-up page and write of an existing chat at N and 10N

Admission walks each input row it reads once (on top of main's parse), and each outcome record it
reads or writes once: the open of a mixed chat walks exactly its prefix rows and admits one record
per key, a catch-up page walks one row per input row it returns, and a write walks its row's
read-back and admits each record it inserts. Linear in what is read, with a constant of one.

* Keep a record that updated a prefix key when a later prefix fold no longer yields that key

A tail record with the prefix ordinal is the key's newest value. If a later
build's reducer reads the old rows differently and no longer produces that
key, the value is appended instead of refusing the whole chat as damaged;
only that key's place can change.

* Resume each sweep page at the generation it is deleting

After a page of records the sweep went back to its start position, so every
later page rescanned every live chat's generation numbered below the retired
one. It now resumes at that generation; a contract at N and 10N live chats
pins that only a pass's first page walks them.

* test: the sweeper rests nine times each page's work

* Start the retired-record sweep from the chat database the attachments already take

The sweep was wired through an optional parameter, so dropping it at the one
production caller would have left retired records forever with every test
passing. It now runs on the required journal database, and a wiring test
checks that installing starts it and stopping ends it.

* test: an existing chat whose first write fails keeps no marker, and the newest release still opens and writes it

* test: chats the window's releases and the oldest release with this chat database wrote are edited, sent to and rewound by this build as the newest release does

Each release writes a Claude and a Codex chat through its own writers. This
build opens it, edits a card the old rows hold (its first write: the marker
and an in-place update), sends, rewinds (every kept message and card saved
again through the strict encoder) and reopens, with every commit restoring
to the live chat. The result equals the newest release's after the same
steps. The oldest release is resolved from tags, as the window is.

* test: the released-chat writer resumes the queue after every Stop it records, and the probe reads each step's result once

* test: the released-chat probe checks this build's edit, send and rewind on its own result

Comparing the carried chat with the newest release's would fail the required
cross-version check for any legitimate main change to those flows until the
next release ships. The probe still admits every released row, opens the chat
as the release and the newest release do, keeps the release's rows byte for
byte with one marker, and restores to live after every commit; after that it
checks the edited card, the accepted send and the rewind's survivors. The
oldest release with the per-host database is named directly: it never changes.

* test: the golden input prefix holds data in every keyed collection

Six collections folded empty, so a change to how old rows derive their keys
went unnoticed there. Eight rows v1.4.245 wrote through its own journal (an
item with an execution owner, a declared echo, a removed turn, and a queued
card handed over with carried history, then cancelled in doubt) follow the
existing rows unchanged, and the test now requires every collection to be
non-empty.

* Say once, beside the marker version, that the reading of old rows under it is frozen

The fold comment claimed an unheld prefix-ordinal value only moves that key;
a prefix read differently can also show a key twice or bring back one the
tail deleted. The marker version now states the rule: a build that reads old
rows into different keys or values keys that reading off a new marker
version, and duplication or resurrection is never intended.

* test: the released-chat probe compares what the release wrote, not fields this build adds

The open check compared whole snapshots, so a field main starts setting would
fail the required cross-version job until a release ships it. It now compares
the release's items, sends and cards, in order and by value, allowing added
fields; the byte-identity and restore-equals-live checks are unchanged.

* test: say why the first-write-failure downgrade test needs a longer timeout

* Type the open-union fallback check by schema, not a bare object

* Check each outcome-format key set separately so no check reads as a redundant union member

* Read an open or branded string as a value in the storage key check, not as an object

* Decode each newest outcome record from the value admission parsed, without parsing or walking it again or keeping its text

* Match each encoded record to its change by position, so a settlement batch's encode stays linear in its mutations

* Encode each changed value as the draft holds it, without reading it through the draft's tracking

* test: rewind, replacement and rollover restate a chat with fixed statements and work linear in what they restate

---------

Co-authored-by: Codex <noreply@openai.com>
2026-10-31 00:40:23 -07:00