2 Commits

Include renames

Author	SHA1	Message	Date
Neil	7d76028a5c	refactor: group agent session journal and request modules (#36882)	2026-10-31 22:33:23 -07:00
Brennan BensonandCodex	2cc33b8d89	fix(native-chat): save each chat change as keyed records and reopen from the newest (#36024) ... * Save native chat state and restore only its journal tail Publish versioned reducer checkpoints atomically, checkpoint through the chat write queue, and backfill closed chats without retiring journal rows. Co-Authored-By: Codex <noreply@openai.com> * Keep checkpoint recovery durable and skip unchanged captures * Keep journal timer and restart fixtures compatible with checkpoints * Keep checkpoint performance tests focused on operation contracts * Remove unused benchmark import * Restore journal append interception and control Stop fixture time * Compare saved reducer state without a key assertion * Supply thread scope in checkpoint test appends * Prove restored journal consumer paths * test: exercise restored consumer backward pagination * fix: pace journal backfill with a monotonic clock * refactor: keep journal opening cursor checks together * refactor: project chat journal state in the append transaction Track isolated reducer mutations and persist keyed changes under a savepoint before adopting committed state. Keep appends working through projection lag and repair coverage on the write lane, including legacy bootstrap. Co-Authored-By: Codex <noreply@openai.com> * fix: keep reducer objects live across journal commits The draft isolates every write until COMMIT, then writes each dirty copy back onto its original, so submissions, items and queue cards handed out before a commit keep reading current values as they did with in-place folding. A failed COMMIT still leaves live state untouched. Also replaces banned Reflect.get uses and adds braces the lint rule requires. * fix: bound projection repair and check the invariant after notification Failed publications and savepoints back off from 1 s up to an hour, reset by a healthy incremental write or a reopen. open() no longer waits for a legacy chat's publication; the scheduled repair publishes it on the write lane. A failed open closes its handle. The equality check runs after commit notification, only when projection tests opt in or, log-only and sampled, in development, and it requires the saved coverage to match. Removes the unused streamed commit argument. * fix: keep the background projection pass proportional to chats The orphan sweep reads one state row per chat and clears orphans by session key. A chat whose fold cannot publish stops at its first damage and is skipped until its epoch or newest row changes. * fix: version the keyed projection layout as codec 2 This branch's earlier whole-state layout used version 1; it now decodes to null and the chat replays. * test: follow the append-with-fold contract and deferred publication A submission whose fold throws now rolls back with its receipt before any dispatch, so those sends are refused and nothing is recorded. The unconfirmed continuation outcome is pinned through its real cause: the agent admits the message and no delivery proof arrives. Tests that need saved state right after a chat opens publish it explicitly, the history reload clears the saved state covering its edited row, and the codex stream bound matches its capture. * test: pin projection repair backoff, background pass and N/10N costs Covers the production repair timer under a persistent failure, a legacy open that publishes after it resolves, a failed open leaving no timer, a damaged closed chat skipped until a new row, and equal per-write, reopen, sweep and closed-chat publication work at N and 10N. Projection tests opt into the per-commit equality check. * refactor: inline draft write-back helpers The repo lint bans object-typed parameters; the copy and write-back now run in the closure that already holds the wrapped value. * fix: write a new epoch's saved state in its own transaction New chats, rollovers and rewinds publish the new epoch's projection beside its rows, inside the same statement-only savepoint the append path uses, so they start covered instead of waiting a second for a repair. A projection failure there still keeps the epoch and leaves the scheduled repair to publish it. * fix: keep the sampled development check log-only on any failure A read failing inside the check must not reject an append that already committed and notified. * fix: typecheck the draft copy without closure narrowing Assign the lazily created copy through a local so strict programs that do not narrow a captured variable after ??= still see an object. * test: catch whole-table sweeps and an undecodable projection The sweep contract now records exec statements too and pins its exact keyed statement list, so a NOT EXISTS scan through exec fails it. A projection whose item stops decoding fails the per-commit check on coverage although a full replay still matches. Also fixes two test lint findings. * refactor: drop draft casts the types can carry Overlay maps and sets finish from their own typed touched keys, and present map entries are read through one helper. The two remaining generic boundaries, a draft standing for its object and finishing back to it, carry the repo's SAFETY directive. * test: show a parked damaged chat publishes after a clean epoch replaces it The replacement ends at the parked chat's newest sequence, so only the epoch in the skip key re-arms it. * test: give the seeded queued card its enqueue cursor Production records queuedAt as the cursor before the queue row; the fixture now does too. * fix: catch lagging saved chat state up in atomic pages Repair of a legacy, lagging or undecodable chat, open or closed, no longer republishes the whole chat. One runner folds the rows after the saved coverage onto a private copy of the saved prefix, at most 32 rows or about 1 MiB of row text per page, and commits each page's changed entries with the new coverage in one FULL transaction. That transaction first checks the chat's epoch and the exact saved row the page was folded from, so a page from a retired epoch or a stale base writes nothing. Pages rest between slices, and the saved prefix is decoded in pages too. An open chat runs the pages on its write lane; sends keep committing between them and skip the projection until coverage reaches the tip, then project incrementally again. The background pass hands a chat over to its open owner at a page boundary. Damage parks the chat after the pages before it, and a failed page keeps the committed ones and retries from them on the existing backoff. Whole-chat publication now runs only inside epoch transactions. * test: pin the draft layer's tracked nodes per streamed revision and acceptance A transition now reports how many objects and collections its fold reached. The scaling chat gains accepted sends (receipts and aliases) and waiting, dispatched and returned queue cards at N and 10N. A streamed revision's only growing term is the card scan, one node per card plus each returned card's body; an echo also reads every item with its body and every submission. * test: append the between-pages send by item identity appendItem takes the provider identity, not its key. * test: check every catch-up prefix of the recorded Claude and Codex streams A runner can page single rows, so the recorded streams' aliases, Stop settlement and per-row timestamps are each checked at a committed prefix against a full replay. * test: measure each backfill pass's row reads on its own The parked-chat test counted its own replay checks: 64 pass rows plus 68, 68 and 100 oracle rows made the 300 CI saw. Each timer window is now measured alone, and coverage is read from its saved row, so the unchanged pass reads 0 rows and the re-armed one reads one page. * fix: bound catch-up reads, release the finished prefix, and keep unreadable views - A catch-up page now stops reading rows at the row whose text reaches about 1 MiB, so rows past the budget are never fetched and fetched again; saved-view item and entry pages use the same budget. - An open chat drops its private prefix once coverage reaches the tip and keeps it only across a failed or parked page. - A present current-epoch view this build cannot read (another codec version or corrupt content) is no longer cleared by the first page. Its whole history must fold cleanly through the tip first; otherwise it parks untouched. A storage error while reading the view fails with backoff instead of reading as a missing view. - The pace charges only the runner's own work units, so time spent waiting on an open chat's catch-up or the write lane no longer lengthens the next rest. * test: cover takeover, epoch replacement and stop in the middle of a pass A chat opened while the pass rests is caught up by its own runner from the committed page; the pass pages it no further and does not charge the wait to its pace. A replacement leaves its new view untouched, and a stopped pass resumes after its last committed page. * test: fail the takeover test without the handover break, and the replacement test without the stale-runner guard The takeover test asserts the pass asks the open owner only twice for the handed-over chat, and pins performance.now so the open runner's pace never waits on a fake timer. The replacement now lands while the open runner is resting in its saved-view decode, so its page reaches the lane after the replacement; the test asserts no transaction opens. * test: read saved-view pages with the page budget, and await database closes The paged-decode round trip passes the row-and-text budget and checks the view decoded. Projection, scaling and ACP launch tests await the now-async test database close, so a closing checkpoint worker cannot close the next test's database. * Persist native chat outcomes with a rebuildable cache Store immutable keyed transition results with canonical ordering, fence outcome epochs, and convert legacy inputs through paced invisible generations. Rebuild disposable caches from validated outcomes and read sparse logical sequence groups without changing the wire. Co-Authored-By: Codex <noreply@openai.com> * test: repair the split suites so they parse, lint clean and find their setup Restores the describe blocks the split truncated, the loop variable it renamed, the fixture references and imports it dropped, and a stray argument pair. Moves the grandfathered flaky-pattern counts with the code that moved. * test: read outcome epochs as outcomes where the adapted assertions read input rows A replaced epoch restates its Stop in the saved pause marks, the sparse-group batch test needs an outcome epoch rather than the input-format fixture, and a revised item keeps its first place, so the held-write order is read from the replayed groups. * test: expect saved queue cards as whole values in the queue-history assertions The adapted helpers read card outcomes, which store each card's whole state, so an import or refusal shows as the card's own state change rather than as an input row. * style: format the files this PR changed * test: import each module once in two journal suites * fix: decode saved outcomes with a frozen format-1 schema, independent of the cache codec Outcome values decode through a versioned format-1 schema that never follows the live reducer type; an unknown field reads as saved by a newer Orca, and the cache codec only delegates to it, so a codec bump rebuilds the cache and nothing else. * fix: key outcomes by an integer generation and keep the cache stamp in its state row Each outcome row names its generation, the attempt's id, instead of repeating the chat id, epoch and a UUID; the cache stamp is a column of the cache's own state row, so there is one stamp to keep in step; cache ordinals are text like the outcomes they copy. Orphaned caches are swept from the per-chat state row with indexed per-chat deletes. The WSL census lists both schema files. * fix: finalize queue ownership before publishing a new epoch, and reject unreadable outcome rows Epoch replacement and rollover adopt the committed state, keep attachment claims and finalize the queue receipt before the fallible publication step, as before outcomes. Every persisted outcome row is read back through the row reader, so a row it rejects fails as journal_row_rejected. Publishing an epoch checks the epoch it replaces is still current. A key deleted and re-created in one write moves to the end of the pending changes, matching its new ordinal. * fix: open a chat synchronously and unpaced, and classify read failures by cause A user's open reads the cache or rebuilds from outcomes in one synchronous pass, like replay on main, so no background rebuild can interleave with it and nothing paces or bills it. Only an outcome decode failure is damage; a value from a newer format is refused as newer; storage errors propagate as unavailable. Writes derive cache currency from the stamp instead of a latch and ask the maintenance runner when it lags. * fix: run all journal maintenance through one paced runner per database One runner serves requested chats first, then lagging ones, one catch-up at a time, open or closed, so an open chat keeps its conversion's progress. A busy or replaced chat is skipped and the pass continues; content that cannot convert parks until the chat's rows change; storage failures back off. Background work alone pays the duty cycle. Conversion verifies the whole input once, then only the outcome tail; cache rebuilds copy outcome pages to the moving tip and stamp from the scalar outcomes. * test: adapt journal suites to generation-keyed outcomes, the merged cache stamp and the publication order Queries read outcomes by generation; the bootstrap test expects the queue receipt to finalize before publication; queue-import-scaling states the reopen's constant two key visits when it retries a failed import; append-delivery reads the outcome change index's item ids. * test: pin outcome format 1 with checked-in rows, and cover open consistency, transient reads and re-created keys A golden fixture of format-1 outcome rows covering every field and scalar must restore to the pinned chat; a cache codec bump only rebuilds the cache; an unknown field reads as newer. An open between two pages of a cache rebuild reads the whole chat from outcomes. A transient read error during open is unavailable, not damage, and the next open loads the chat. A key deleted and re-created around a new key in one row keeps live order, written or converted. * refactor: check a converting chat's epoch once, in the catch-up step that runs each page The catch-up reads the chat's epoch in the same synchronous step as the page's transaction, for conversion and cache rebuild alike, so the conversion's second copy of the check is dropped. * test: drive the maintenance runner over a real database, each guard killed by a mutation Startup sweeps, closed chats converted one at a time with a busy chat skipped, an open chat writing between conversion pages, a stop between pages, an epoch replaced mid-conversion, content parked until the chat writes again and then delivered, and a storage failure backed off rather than parked. * test: N/10N contracts for outcome appends, opens, conversion, publication, idle passes, sweeps, tail verify and wire catch-up Each counts statements through prepare and exec, rows read per table and rows and bytes written, never time, and requires the same work for a chat ten times larger: a streamed revision, a Codex plan update and an acceptance; a cached open reads no outcome; a conversion page reads one page of input and publication none; an idle background pass reads no history; a retired generation and an orphaned cache are deleted one indexed page at a time; a second verify reads only the new outcomes; a wire catch-up reads only the groups asked for. * test: shipped builds keep a new or converted outcome chat untouched The newest release refuses the chat as a newer Orca's and leaves every byte of the database as it was; a build from before that reason opens it read-only with nothing shown, refuses writes, and changes no saved row. * test: an epoch publication over an epoch the caller did not see is refused, and damage at row 1 still yields to a newer row Publishing a new epoch checks, inside its transaction, the epoch the caller last saw, and writes nothing when it changed. A chat whose first row is unreadable keeps reading for a newer build's row, as main does, and is refused as a newer Orca's. * test: old-build probes fold input rows only, and the interrupted-call probe writes inputs * test: import the state layout once in the cache test support * fix: type the tail verify's live lookups per collection, and two test call shapes The tail verify looked keys parsed from stored JSON up in a union of live maps, which does not typecheck; each format-1 collection now has a typed lookup that narrows the parsed key first, and the table is checked complete against the format's collection list. * fix: wake the maintenance runner for a request or a due retry ahead of the periodic pass One timer now wakes the runner at the earlier of the next periodic pass and the first due request or storage retry, re-derived from the waiting chats after each pass, so a request after the first pass is served within a second and a backoff retry is no longer replaced by the six-hour pass. A chat whose load swept a retired page keeps its turn instead of waiting for the next pass. Tests drive start(), requests and timers on fake time. * test: pin the draft's tracked nodes per streamed revision and acceptance at N and 10N The scaling chat gains dispatched and returned queue cards. A streamed revision's only growing term is the queue card scan, one node per card plus each returned card's body; an acceptance echo also reads every item with its body and every submission. Main's reducer makes the same scans on these rows; the draft adds one node per value reached. * test: cross-version probes read a new chat's turn end as one outcome sequence, and the kept-card downgrade uses an unconverted chat * fix: store every new write as format-tagged outcome records after an existing chat's rows, with no conversion An existing chat keeps its input rows byte for byte; its first write appends one marker row that shipped builds refuse, and its records follow. New chats hold only their marker and records. Each record carries its own format and the epoch keeps the highest format it holds, checked before a cache is used or anything is written. Opening a chat with an input prefix folds the prefix as main does and applies the records over it, deletions included, keeping prefix keys in place; catch-up pages the prefix rows, then the record groups. Background conversion and its attempts, staging, verification and publication are gone; the maintenance runner rebuilds only caches of chats without a prefix, parks a rebuild on its epoch's records rather than the newest sequence, and looks a requested chat up by its key. * test: pin the marker, prefix-plus-records reopen, record admission before the cache, cache-only runner parking and N/10N work Input fixtures stand for a chat an older build wrote: their input epoch keeps taking input rows, so the input reader still meets writer-made rows; epochs this build founds are written as it writes them. Conversion tests give way to a mixed chat whose prefix keys are deleted, recreated and updated in place, the runner's rebuild parking that later writes never re-arm, and contracts for writing to, opening and rebuilding chats at N and 10N. * fix: refuse a chat whose outcome marker is a newer version as a newer Orca's, not as damage * test: shipped builds refuse an existing chat at its appended marker, its input rows unchanged, and this build reads it whole again The probe writes three items as input rows the way an older build did, then this build's writes append one marker and outcome records; the newest release refuses the chat as a newer Orca's with every byte unchanged, the writable baseline latches read-only at the marker, and the re-upgraded build reads all six items with the prefix rows as they were. * test: read the runner's rebuild instances as rebuilds, so the parked-chat test typechecks * fix: admit saved outcome records and the cache only through a strict storage schema A saved record or cached value holding a key or union tag this build does not declare, at any depth, now refuses the chat as a newer Orca's before the cache is used or anything is written. The storage schema reuses the wire's shared definitions, completes the keys they leave out (failure facts, senders, edit patches and the rest), and closes every object and open union; string vocabularies and opaque payloads keep the wire's meaning. A compile-time check fails CI when a persisted type gains a key storage does not declare, and a checked-in corpus of every declared shape replaces the schema fingerprint, so an added optional key needs no new format. * fix: refuse an input row or legacy card holding a key this build does not declare before folding it Restating a chat on a new epoch is safe only if the open chat holds nothing this build half understands. An input row of a known kind (an existing chat's frozen prefix, with or without outcome records after it) is now checked key by key against what this build declares before main's reader normalizes it, and refuses the chat as a newer Orca's; values keep main's checks and execution options their legacy meaning. A legacy queued card whose saved body holds an undeclared key stays on disk, unadopted, as an unreadable card does. Client sends already arrive through strict request schemas, and their options through a reader that drops choices this build does not use. Main's tests that kept a newer build's unknown body kind, subject or key readable and writable now expect the refusal; the three that drove a newer Orca's approval or item through this build's own writers are removed, since this build can no longer hold one. A checked-in input prefix pins how this build folds older rows, and every value it folds into is one storage admits. * test: a persisted type holding a key storage does not declare, nested included, fails to compile * test: this build refuses a newer build's body or subject kind as a newer Orca's, keeping the row an older build deletes * fix: fold each outcome write from the row a reload reads, so an annotation the reader drops never fails the strict record write An ACP usage report with size 0 (allowed by the protocol) reached the model as an unusable window, the strict encoder refused it, and the agent was stopped mid-turn. Main writes that row and drops the annotation on reload; folding the read-back row keeps live state equal to that reload for every annotation the reader drops. * test: an echo declared without a known original is stored as null, the shape a reload reads * test: probe the downgrade refusal on both releases in the downgrade window, not only the newest * test: both releases in the downgrade window write a Claude and a Codex chat through their own writers, and this build admits every row and opens the chat as that release had it * test: rewind, replacement and rollover of a chat holding old rows, the marker and records keep what the user kept and delete the retired epoch whole * test: phone and paired catch-up over a chat holding old rows then records, from a cursor inside the old rows, draws as the clients render it * test: ACP launch decisions read an outcome chat from its cache and from its records alone, and a newer one as unreadable * Open new and mixed chats from each key's newest record and drop the saved-state cache The cache copied each key's newest record into a second set of tables and needed stamps, a paced rebuild, a savepoint fallback and a repair runner to stay usable. The records' key index already answers the same read: an open seeks from one key to the next, taking each key's newest record (deletions included) and never reading superseded revisions. A mixed chat still folds its frozen input prefix first. A send now writes only its records, the epoch's coverage and its receipt, so any failure among them refuses the send as main's row insert does. The background runner keeps only the paced sweep of retired generations, its one remaining job. The cache tables are no longer created; they never shipped. * Derive an outcome chat's last sequence from its coverage instead of recording it Every write recorded the sequence scalar beside its own records, about half the records on streamed paths. The epoch's coverage already holds it, so a read supplies it from there and a write no longer stores it. Records an earlier build stored still decode. * test: count strict admission per open, catch-up page and write of an existing chat at N and 10N Admission walks each input row it reads once (on top of main's parse), and each outcome record it reads or writes once: the open of a mixed chat walks exactly its prefix rows and admits one record per key, a catch-up page walks one row per input row it returns, and a write walks its row's read-back and admits each record it inserts. Linear in what is read, with a constant of one. * Keep a record that updated a prefix key when a later prefix fold no longer yields that key A tail record with the prefix ordinal is the key's newest value. If a later build's reducer reads the old rows differently and no longer produces that key, the value is appended instead of refusing the whole chat as damaged; only that key's place can change. * Resume each sweep page at the generation it is deleting After a page of records the sweep went back to its start position, so every later page rescanned every live chat's generation numbered below the retired one. It now resumes at that generation; a contract at N and 10N live chats pins that only a pass's first page walks them. * test: the sweeper rests nine times each page's work * Start the retired-record sweep from the chat database the attachments already take The sweep was wired through an optional parameter, so dropping it at the one production caller would have left retired records forever with every test passing. It now runs on the required journal database, and a wiring test checks that installing starts it and stopping ends it. * test: an existing chat whose first write fails keeps no marker, and the newest release still opens and writes it * test: chats the window's releases and the oldest release with this chat database wrote are edited, sent to and rewound by this build as the newest release does Each release writes a Claude and a Codex chat through its own writers. This build opens it, edits a card the old rows hold (its first write: the marker and an in-place update), sends, rewinds (every kept message and card saved again through the strict encoder) and reopens, with every commit restoring to the live chat. The result equals the newest release's after the same steps. The oldest release is resolved from tags, as the window is. * test: the released-chat writer resumes the queue after every Stop it records, and the probe reads each step's result once * test: the released-chat probe checks this build's edit, send and rewind on its own result Comparing the carried chat with the newest release's would fail the required cross-version check for any legitimate main change to those flows until the next release ships. The probe still admits every released row, opens the chat as the release and the newest release do, keeps the release's rows byte for byte with one marker, and restores to live after every commit; after that it checks the edited card, the accepted send and the rewind's survivors. The oldest release with the per-host database is named directly: it never changes. * test: the golden input prefix holds data in every keyed collection Six collections folded empty, so a change to how old rows derive their keys went unnoticed there. Eight rows v1.4.245 wrote through its own journal (an item with an execution owner, a declared echo, a removed turn, and a queued card handed over with carried history, then cancelled in doubt) follow the existing rows unchanged, and the test now requires every collection to be non-empty. * Say once, beside the marker version, that the reading of old rows under it is frozen The fold comment claimed an unheld prefix-ordinal value only moves that key; a prefix read differently can also show a key twice or bring back one the tail deleted. The marker version now states the rule: a build that reads old rows into different keys or values keys that reading off a new marker version, and duplication or resurrection is never intended. * test: the released-chat probe compares what the release wrote, not fields this build adds The open check compared whole snapshots, so a field main starts setting would fail the required cross-version job until a release ships it. It now compares the release's items, sends and cards, in order and by value, allowing added fields; the byte-identity and restore-equals-live checks are unchanged. * test: say why the first-write-failure downgrade test needs a longer timeout * Type the open-union fallback check by schema, not a bare object * Check each outcome-format key set separately so no check reads as a redundant union member * Read an open or branded string as a value in the storage key check, not as an object * Decode each newest outcome record from the value admission parsed, without parsing or walking it again or keeping its text * Match each encoded record to its change by position, so a settlement batch's encode stays linear in its mutations * Encode each changed value as the draft holds it, without reading it through the draft's tracking * test: rewind, replacement and rollover restate a chat with fixed statements and work linear in what they restate --------- Co-authored-by: Codex <noreply@openai.com>	2026-10-31 00:40:23 -07:00