fix(native-chat): preserve large structured command results (#17707)

* fix(native-chat): preserve large structured command results

* chore: place native chat validation artifacts under docs

* chore: drop stale root package config

* fix(native-chat): enforce rebuilt lifecycle append slots

---------

Co-authored-by: Merge Sim <sim@local>
This commit is contained in:
Brennan Benson
2026-08-31 12:37:20 -07:00
committed by Merge Sim
co-authored by Merge Sim
parent 894ed75abb
commit 32b47ecf14
142 changed files with 14469 additions and 2444 deletions
@@ -0,0 +1,11 @@
# Native structured chat large-result Electron evidence
- Date: 2026-08-31 (local dev build)
- App identity: `Orca: brennanb2025/fix-native-chat-large-results`
- Worktree: `/Users/brennanbenson/orca/workspaces/orca/fix-native-chat-large-results`
- Rendered surface: native Codex chat tab in the Electron app, attached through CDP on port 9340.
- Prompt 1: requested a shell command printing exactly 1,100,000 characters.
- Visible result: tool activity settled and the chat rendered `Confirmed: exactly 1,100,000 characters were printed.` without killing the tab or provider. The large payload was not dumped into the rendered transcript.
- Prompt 2 (same chat): `Now reply with exactly: follow-up succeeded`.
- Visible result: `follow-up succeeded` rendered in the same tab, proving the provider/session remained usable after the >1 MiB item completion.
+130
View File
@@ -0,0 +1,130 @@
# Native structured chat large-result lifecycle plan
## Outcome and scope
Keep a native structured chat session alive when a valid provider `item/completed` JSONL frame carries a multi-megabyte command result, preserve bounded memory/disk/replay behavior, settle the tool and turn truthfully, and make a proven unexpected provider exit visible and recoverable without automatically redispatching an unknown user message.
This is host-local. It does not add or change an RPC method, stream opcode, capability, or published wire shape, so mixed-version clients and hosts continue to use the current structured-session contract. Structured native chat currently admits runtime-local worktrees and local folder workspaces; direct WSL and SSH locations remain explicitly refused. Framing, persistence, owner release, and reacquisition stay on the execution host, so this fix does not weaken those location boundaries.
## Evidence and precedent
- The persistent provider connection currently retains one UTF-8 string and kills the provider tree as soon as the unterminated line exceeds 1 MiB. The observed valid frames were 1,090,188 and 2,900,090 bytes, so the limit is below the protocol's real serialized envelope.
- The synced provider protocol maps execution output deltas to `item/commandExecution/outputDelta`, then maps execution end to an authoritative `item/completed`. Its item builder copies retained `aggregated_output` into that completed item. The turn shell-execution path caps final aggregated raw command output at 1 MiB total, while JSON escaping and the surrounding item/RPC object make the serialized line larger. A receiver limit equal to the raw payload cap is therefore invalid. A separate direct execution API can use full-buffer capture, but the host does not call that API.
- The host already has a byte-counted, chunk-safe 16 MiB NDJSON envelope with bounded unterminated-line discard. Reuse/generalize that parser rather than maintaining a second quadratic string accumulator. For the live turn methods in scope, 16 MiB is a derived admission envelope: two capped raw streams, worst-case six-byte JSON escaping, and protocol metadata still fit. It is not a claim that every record the provider can ever emit is globally bounded.
- The same connection carries thread/resume responses, whose history arrays scale with thread length and can validly exceed a per-record admission envelope. An oversized response must therefore be refused visibly and settle its pending request without killing the provider or leaving acquisition waiting forever. This is distinct from accepting the bounded live completion that triggered this fix.
- The item-stream layer already coalesces deltas and uses geometric checkpoints; item translation already treats the completed item as authoritative and routes command output through bounded inline text.
- Journal payload bounds keep 16 KiB inline, record original byte length and digest, and write the complete accepted payload to content-addressed storage. Its current 256 MiB session guard counts journal rows, not referenced blob bytes; therefore repeated large results can exceed the stated disk bound. Compaction prunes blobs with no retained state references. The fix must generalize this path into one durable quota rather than inline full results or create an unaccounted store.
- Runtime construction currently omits the event callback. Consequently session close can journal an `ended` event but cannot tell the host that its attached session no longer has a provider child. Existing observed-exit lease transition and held-session resume behavior are the precedents to extend.
## Design
### 1. Use one bounded, non-fatal JSONL framer with a method-sized admission envelope
Generalize the existing NDJSON parser into a concretely named shared/main framing module usable by both daemon traffic and the persistent provider connection, retaining its current daemon exports for compatibility. Feed decoded UTF-8 chunks through it so byte accounting is incremental and a partial multi-byte character is handled by the stream decoder. Configure the provider connection with a 16 MiB admission envelope derived for the bounded live turn methods in scope, rather than a raw-payload-sized 1 MiB limit.
The parser must:
- dispatch complete object records in arrival order;
- keep at most the configured line ceiling plus the current input chunk and avoid duplicate full-line copies before parse;
- accept frames on both sides of the old 1 MiB boundary, including a realistic 2.9 MiB escaped command completion;
- reject non-object JSON as an unhandled invalid frame, preserving current behavior;
- treat malformed complete JSON as an unhandled frame without killing a healthy provider;
- when a line exceeds the envelope, report exactly one structured rejected-frame result, discard through its newline with constant additional memory, and then resume parsing later lines;
- settle any pending request that could have owned the rejected line with an explicit size error. If correlation cannot be proven from the bounded prefix, fail all currently pending requests as unknown rather than guessing which request was delivered; never leave one waiting for its timeout solely because the frame was discarded;
- when the bounded prefix safely proves an oversized provider-initiated JSON-RPC request id, send a bounded JSON-RPC size-error response so the provider can unblock that request. If neither request/response/notification class nor correlation can be proven, escalate the rejected frame into explicit protocol-failure recovery rather than leaving a possibly blocked provider attached;
- avoid full-history resume responses in the first place: request metadata-only resume (`excludeTurns`). Treat the reduced journal as authoritative for already imported history and derive the next turn ordinal from it, so unexpected-exit reacquisition needs no provider-history scan. A first structured attach to an existing thread may hydrate only facts absent from the journal: stable turn/item identities and bounded summary/not-loaded bodies, stopping at the first identity already present or the provider end cursor. Bound the whole hydration, not just each page, by the same 16 MiB acquisition-memory envelope, the existing maximum reduced-history item count, and the existing acquisition deadline; derive the maximum page count from that item cap and the requested provider page size. Check aggregate encoded bytes and item count before accepting each page. Hitting any byte/item/page/time bound before an existing identity or end cursor is a visible acquisition refusal, not partial publication. Cache a narrowly detected unsupported-method/parameter result per connection and retain the current single-response bounded full-history fallback for older provider builds;
- surface any still-oversized acquisition/history response or single history page as a bounded, user-visible refusal. Acquisition cleanup may deliberately close the now-unused child, but the framer itself must not destroy stdout or terminate the provider tree;
- surface an oversized notification to the translator so active streamed tool state is explicitly terminalized as incomplete and later frames, including `turn/completed` and a next turn, remain processable;
- avoid repeated whole-buffer `Buffer.byteLength` scans and repeated prefix slicing.
This is not an unbounded parser and not a larger arbitrary buffer: the live method's raw payload ceilings, worst-case JSON escaping, protocol overhead, and the existing NDJSON ceiling define admission. Valid but larger response classes get a deterministic refusal, while malformed or adversarial unbounded input has bounded memory and cannot kill the provider or wedge the request/session. Framing work is synchronous and bounded; the connection processes records one at a time and can stop between records when the durable sink reaches its high-water mark.
### 2. Keep streamed and authoritative output on the existing bounded persistence path
Do not create a parallel result store or publish full command output to renderers. Continue to coalesce `outputDelta` notifications, checkpoint snapshots geometrically, and upsert one tool row. Generalize the existing journal payload-bound code with a byte-counted streamed-text accumulator: retain no more than the accepted per-item payload budget, track total observed bytes, and append an explicit truncation marker after saturation. Stop growing or checkpointing that item once the retained representation stops changing. This bounds the delta coalescer, `latestText`, checkpoint blob sizes, and disk churn even if the provider sends its maximum 10,000 deltas per stream.
On `item/completed`, forget stale coalesced text before appending the authoritative item, then let `boundInlineText` retain a 16 KiB head and write the complete accepted payload to the content-addressed blob store before the journal row is appended/published. For the supported turn path the authoritative completion remains within the derived frame/payload admission envelope. If the completion frame itself is rejected, terminalize the streamed row as incomplete with observed byte/truncation evidence; do not present it as a successful authoritative result.
Make the asynchronous event sink an explicit bounded disk-backpressure boundary. Charge every queued closure before enqueue by its encoded row/blob byte estimate and one operation; permit only one in-flight operation plus configured queued-byte and queued-operation high-water marks derived from the live-frame envelope and reserved lifecycle capacity. Coalesce/replace pending same-item checkpoints before blob creation and coalesce repeated diagnostic/error-surface frames into one bounded per-turn/session count-and-last-digest row; distinct ordinary events never bypass the charge. When accepting the next parsed record would cross either high-water mark, pause provider stdout before dispatching it, retain at most the framer envelope plus the current input chunk, and resume below a fixed low-water mark after ordered writes drain. Node and OS pipe backpressure then bound unread provider output without blocking the main process. Authoritative completion and terminal lifecycle operations use their pre-reserved capacity and are never dropped or coalesced away.
If a write fails, the sink enters one bounded failed state: reject/coalesce further ordinary operations into its single failure diagnostic, keep stdout paused, report the failed lifecycle barrier to the host, and run the explicit cancellation/observed-exit recovery path rather than accumulating retries or closures. A transient write can resume only after the serialized store has revalidated quota and the failed operation has either committed idempotently or settled through its reserved fallback. A permanent failure leaves attach refused and the owner verdict honest; it cannot grow memory or silently keep a dead/actionable session. Count estimates are checked against actual encoded bytes before write, and any underestimate consumes only the operation's reservation or produces its bounded fallback.
Generalize the journal's 256 MiB per-session limit into a total physical durable-storage quota covering every host-owned file in the session journal directory: the journal log/snapshot, unique referenced blob bytes, retained corruption/newer-schema quarantine files, and temporary staging copies used by blob writes, log rewrites, snapshot compaction, and quarantine. Blob admission and row append must run on the journal's serialized write path as one preflighted operation: account for digest deduplication and post-upsert live references, reserve worst-case staging bytes before writing, compact/prune and recompute before refusal, write a new blob before its row, and remove a newly staged blob if the row fails. Compaction, replacement, or quarantine starts only when the peak old-plus-temp-plus-final physical footprint fits the hard limit; otherwise refuse the mutation without creating another copy. On open, count known staging and quarantine artifacts before admission. Only a writable current-schema open may sweep artifacts it can prove stale; preserve and continue charging any quarantine file that cannot be removed, including on a read-only or newer-schema open, then recompute. Refuse safely if retained physical state already exceeds the limit. This keeps blob-before-row and evidence-preserving quarantine crash safety without letting restart, corruption handling, or temporary double writes exceed the budget. Tests must fill a session with many maximum-size results, include repeated identical payloads, inject crashes leaving known staging files, reopen it, and prove the physical directory footprint stays within the configured bound.
Quota admission must reserve lifecycle headroom inside that same hard limit at every transition that creates unsettled durable state. Maintain a deterministic per-turn terminal-batch accumulator alongside the individual ownership tokens. Before admitting another running tool, pending approval/question, or terminal assistant candidate, pre-encode and preflight the whole possible terminal settlement for that turn: the outer row envelope, every nested identity and bounded body, bounded statuses, one physical append, projected item/removal count, and the forward/tail/backward history-page row and byte budgets. Refuse the new running/pending state before either the physical batch cap or any projected page cap would be exceeded. This aggregate reservation, rather than the sum of independently usable fallback tokens alone, is the proof that one unexpected exit can settle the complete turn.
Reserve at each transition:
- before `performSend` calls the provider, reserve both the worst-case accepted/rejected/unknown dispatch-settlement row and a tentative whole-turn terminal accumulator/rate slot. The provider may synchronously emit `turn/started` before dispatch settles, so bind that tentative reservation to the reported turn id before publishing its running lifecycle row. Release the turn portion only after a proven rejection with no started turn; retain it across accepted or unknown delivery until terminal evidence, and refuse dispatch before provider contact if either reservation cannot be made;
- before `turn/started` publishes a running lifecycle row, reserve one bounded terminal batch containing its tombstone, turn-level status, and either a bounded final assistant row or an explicit assistant-output-unavailable row;
- before persisting any `running` tool or pending approval/question, reserve the worst-case bytes and one batch append slot for its whole no-new-blob fallback, including terminal/cancelled upsert plus bounded status.
Each reservation token carries independent byte and append-slot budgets and is consumed by exactly one idempotent settlement batch/row before release. Hold it until that exact submission, turn, tool, or prompt settles; settling one item must not release the turn or submission reservation. The per-turn accumulator claims the aggregate maximum rather than double-counting its component tokens, and is recalculated deterministically after every admitted or settled item. Keep a small fixed session emergency token for rejecting an event that cannot be admitted at all. Rebuild all outstanding tokens and aggregate accumulators from pending submissions and running/pending reduced state on reopen. Normal payload/blob writes may consume only unreserved capacity, while terminal settlement consumes and releases the matching tokens and accumulator. This makes truthful settlement possible for tool-less turns, already-settled tools inside an active turn, in-flight sends, and authoritative completion arriving exactly at saturation. If a legacy/recovered turn is already too large for one admitted row, split it by a deterministic stable ordering into bounded idempotent lifecycle batches that each fit ordinary page limits, terminalize tools/prompts/status first, and append the turn tombstone strictly last; attach/reacquisition remains barred until every batch succeeds.
Mirror byte reservations in the append-rate guard. Reservation-consuming lifecycle batches and the fixed emergency settlement path get a narrowly scoped rate slot that ordinary provider rows cannot consume; do not generally exempt lifecycle-shaped rows. The number of exempt settlements remains bounded by previously admitted reservations plus one emergency path, so a delta flood at `maxAppendsPerWindow` cannot strand a tool, prompt, turn, or dispatch row and cannot create an unbounded bypass. Reconstruct rate reservations with byte reservations on reopen.
Make authoritative completion plus quota fallback one ordered journal operation. If a completed tool's blob/row cannot be admitted after compaction, consume only that tool's reservation to upsert it as `completed` with no output blob and append a bounded status explaining that the successful provider result could not be retained. Keep the provider turn and unrelated prompts running: output-persistence failure is not evidence that the turn ended, and `turn/completed` may still deliver an assistant answer. If a pending provider prompt cannot obtain its own reservation, send the provider's bounded cancel/error response and persist only its cancelled/non-actionable representation plus status; again, do not tombstone the turn merely for storage pressure. If an announced running tool cannot be reserved, never publish it as running; record the bounded quota status from emergency capacity and continue processing later lifecycle frames.
Only a positively observed terminal turn notification or provider exit may consume the turn reservation and tombstone its lifecycle row. A raw cancellation RPC acknowledgement, including `cancelTurn()` returning `{ cancelled: true }`, means only that cancellation was requested: return and journal bounded acknowledgement-only wording such as `Cancellation requested`, do not append `Turn cancelled.`, and keep the active-turn send gate engaged until `turn/completed` (including its cancelled/interrupted status) or process exit arrives. Once terminality is observed, cancellation/exit may atomically terminalize every still-running tool, cancel prompts, append status, and tombstone. If storage pressure requires abandoning the whole turn rather than one result, request cancellation but keep the spinner and send gate truthful until that terminal evidence arrives.
Represent any multi-mutation settlement as one bounded `lifecycle-batch` journal row whose reducer applies only existing item/tombstone semantics together. Give every nested mutation the outer row's sequence, timestamp, and fence for replay/gap purposes while retaining the reducer's existing item-creation ordering rule. Cap the batch and its projected nested effects below the existing history-page content budget. Extend `agent-session-journal-batch` projection to expand the row's nested touches: an after-cursor read must publish every resulting current item and removal together at the outer cursor, or return a bounded reset, never advance past only part of the settlement. Client-facing item/tombstone shapes stay unchanged.
Tail and backward history paging must also treat items sharing one creation sequence as an indivisible group for row-count and byte limits. Never split a sequence group and then issue a strict-less-than cursor that skips its omitted peers. The admitted lifecycle-batch cap guarantees its group fits a normal page; if a legacy/corrupt group exceeds the page budget, return a bounded reset/stand-in without advancing a cursor through a partial group.
A torn final JSONL append is discarded as one incomplete row, never as a partially applied settlement; a successfully appended row makes every terminal mutation visible at the same sequence. Give each lifecycle batch a stable settlement id derived from durable session/turn or exited-fence identity, and preserve the bounded applied-settlement-id set in journal snapshot/compaction metadata. An unacknowledged successful append replay becomes a no-op instead of a second row. Cover write faults before, within, and after the batch append, reopen the journal, and prove reduced state is either wholly pre-settlement or wholly terminal.
Extend the deferred sink with a lifecycle barrier that reports whether the queued primary-or-fallback write succeeded; `drained()` must no longer make recovery infer success from errors that were merely reported and swallowed. Provider callbacks remain non-throwing. A failed lifecycle barrier is carried into host recovery, but it must not suppress release of an owner whose exit was positively observed.
Before publishing or reacquiring any writable session whose prior lifecycle barrier failed—or whose reduced journal still contains running tools, pending prompts, or a running turn while durable owner state proves that generation exited—the normal attach/dead-owner recovery path rebuilds and appends the same idempotent no-new-blob lifecycle batch. It must complete that batch and its sink barrier before snapshot publication or provider acquisition. If the journal is read-only or the reserved fallback still cannot land, release remains honest but attach refuses; it must not publish stale controls or start a replacement writer behind them.
Only a fully readable, current-schema journal may reconstruct quota state or prune blobs. Its referenced set is the union of digests in the reduced live snapshot and every retained replay-tail revision, because an older cursor can still request a superseded payload before compaction. Physical quota accounting includes that union plus the current log/snapshot. For a newer/unknown schema or any read-only open, preserve every blob and perform no quota mutation, pruning, compaction, or write; readable-state recovery must not turn forward compatibility into data loss.
### 3. Make unexpected provider exit a generation-fenced host lifecycle event
Make the adapter's `ended` event discriminated as `unexpected-exit` versus `requested-close`, and carry both the acquisition fence and a freshly minted acquisition-generation token on it (host-local TypeScript only). The token also travels in the adapter acquisition result and is retained by the host session; a fence alone cannot distinguish two children acquired under the same durable generation. Only the connection `onExit` path emits `unexpected-exit`; eviction, handoff, superseding acquisition, and shutdown emit `requested-close`. The adapter already suppresses stale-child exit callbacks; the host must compare both fence and generation with its attached session before mutating state. Expected closes continue their existing quiet teardown and must never enter recovery.
Wire production `onEvent` during runtime construction to a new host lifecycle entry point, but route only `unexpected-exit` into recovery. The adapter/translator owns journal settlement because it alone retains item identities, current turn ids, and last streamed text. Before disposing that translator it must:
1. flush delta checkpoints;
2. terminalize every still-running command item as interrupted/failed with its bounded last output;
3. cancel every unresolved approval and question row so the journal does not retain controls for requests the provider can no longer answer;
4. append one visible bounded status item containing the provider-exit reason;
5. tombstone running turn lifecycle rows only after those terminal rows.
The translator must retain the identities/bodies required to upsert pending prompts as `cancelled`. Perform those journal updates before clearing the adapter's live prompt registry on both unexpected and requested close; requested close remains quiet in the sense that it adds no exit-error status and triggers no recovery, but it must not leave actionable stale prompts.
The host callback then serializes with session mutations, revalidates the session id, native owner, event fence, child generation, and handoff state, and:
1. awaits the lifecycle barrier and records whether terminal publication succeeded;
2. regardless of barrier success, releases the native owner through the existing observed-exit transition while explicit current-generation exit evidence is still in hand, producing `claimStatus: released`, `exit-observed`, and fence + 1;
3. only after the durable release succeeds marks the in-memory session as having no provider child and updates its fence;
4. returns a recovery ticket containing session id, released fence, dead acquisition generation, stable settlement id, whether settlement retry is required, and the requirement for a resume-capable holder.
End the serialized callback at that point. The existing attach path also enqueues on the non-reentrant per-session promise chain, so invoking it from inside this callback would deadlock. Only after the callback promise resolves may recovery call the normal attach path. Attach must revalidate the ticket against the then-current released fence, dead generation, resume-capable holder set, null handoff stage, and terminal-settlement state; when retry is required it lands the idempotent lifecycle batch before publication/acquisition. A hold disappearing or a handoff queued between release and reattach cancels automatic recovery without dispatching or leaking an owner.
Do not reuse the mutable `hasProviderChild` guard as the proof for this transition: add a narrowly named observed-unexpected-exit release entry point that requires the expected fence/generation and reuses `releaseStoredAgentSessionOwnerAfterSurfaceClose`. A test must pin the released claim, evidence kind, and fence increment. Extend holder bookkeeping to retain the existing `resume` capability per holder and expose `hasResumeCapableHolder(sessionId)`; a subscription-only hold must never cause exit recovery to spawn a child.
Reacquisition is safe because process exit is positively observed on the execution host and the old fence is retired. It must never replay a journal submission. If a send was in flight when the provider died, its adapter error remains `unknown`; recovery may restore a live session, but only an explicit client retry with `retryUnknown` may redispatch that message. A later new user message can dispatch on the replacement child.
Expected close/eviction remains quiet and follows the existing ordered teardown. A stale exit event is ignored. If the serialized callback finds an active handoff stage, it must not force a surface release or parallel reacquisition: the handoff coordinator owns the already-started transition. The callback records that the native child is gone and lets the handoff finish, roll back, or enter its existing explicit recovery state; tests cover an exit racing `preparing` and prove no second owner or dead native `live` lease remains. A failed ordinary reacquisition leaves the durable journal readable, the visible exit item present, and the lease/refusal state honest for a later hold or attach retry.
### 4. Tests
Add focused regression coverage at four layers:
- Framing/connection: split and coalesced chunks, UTF-8 boundaries, just below/at/above 1 MiB, the 1,090,188-byte case, the 2,900,090-byte escaped completion case, just below/at/above the admission envelope, malformed JSON followed by a valid frame, and an unbounded unterminated frame followed by recovery at newline. Valid admitted frames must not destroy stdout, terminate the tree, fail pending calls, or emit `onExit`; rejected frames report once, use bounded memory, leave the provider alive when safely classifiable, settle affected requests, and allow a later valid frame. Prove an oversized provider request with an early id receives one bounded size-error response. Add metadata-only/paged resume coverage plus an oversized history response/page fixture representative of the pinned provider history mode and assert an immediate visible acquisition refusal rather than a timeout or deterministic retry loop. Feed many individually legal history pages and independently hit cumulative byte, item, page, and deadline bounds; prove hydration stops before accepting the crossing page, publishes no partial import, closes/refuses acquisition visibly, and performs no deterministic retry loop. Prove recovery with an existing journal derives its ordinal and stops without paging once a retained stable identity is reached.
- Translation/persistence: realistic `item/started` + many deltas + authoritative large `item/completed` + `turn/completed`. Assert one terminal successful tool item, a 16 KiB-bounded projection with explicit byte length/digest, full blob persistence, no live turn marker, and bounded replay/page payloads. Drive deltas past the accumulator budget and assert bounded retained memory/checkpoint count/blob bytes plus an explicit truncation marker. With slow and failing disk, flood same-item deltas, distinct ordinary notifications, error-surface frames, tools, and lifecycle frames; assert queued bytes/operations and framer retention never exceed their high-water envelopes, stdout pauses/resumes at the watermarks, diagnostics remain countable and bounded, reserved terminal events settle in order, and permanent failure enters one visible/refused recovery state without closure or retry growth. Saturate byte and append-rate quota after a pending submission, running turn/checkpoint, and pending prompts, then refuse the authoritative blob: assert the tool's full fallback batch consumes its own byte/slot token, the tool becomes completed-with-output-unavailable, and the turn remains live. Saturate capacity before send and prove `performSend` refuses without provider contact; at the exact remaining boundary, prove the tentative dispatch plus whole-turn reservation is acquired before provider contact, a synchronous `turn/started` binds it and publishes truthful running/send-gate state, and accepted or unknown dispatch retains it until terminal evidence. At the exact aggregate batch/page boundary admit the final pending item, reject the next one before it becomes actionable, and prove an unexpected exit settles every admitted tool/prompt/status plus the turn within the preflighted physical and projection limits. Repeat at byte, item-count, and append-slot saturation; reopen between admission and exit to prove reconstruction preserves the same decision. Exercise the legacy/recovered overflow path and prove deterministic idempotent multi-batch settlement with the turn tombstone in the final successful batch. Deliver an assistant item that cannot use ordinary capacity, then `turn/completed`; assert the turn token publishes bounded assistant-output-unavailable/status/tombstone together. Separately make `cancelTurn()` acknowledge cancellation without a terminal notification and assert the API result, journal text, and UI say only `Cancellation requested`, the running turn and active-turn send gate remain engaged, and no terminal wording/state appears until a later terminal notification or exit consumes the turn token and settles all remaining tools/prompts/status/turn. From a cursor immediately before the batch, assert one forward page contains every nested current item/removal at the batch cursor (or one bounded reset), never a subset; use low row/byte limits to prove tail/backward paging never splits multiple new items sharing that creation sequence. Fault the batch JSONL write at several byte offsets and prove reopen never exposes a partial batch; replay an unacknowledged successful settlement across snapshot/compaction and prove its stable id deduplicates. Exercise tool-less active turns, already-settled tools with a still-running turn, append-window saturation, many distinct maximum results, duplicate-digest results, quota refusal, compaction, staged blob/log/snapshot crash artifacts, retained-tail-only blob references, writable reopen accounting/sweep, and newer-schema read-only reopen without pruning. Near quota, corrupt the journal and prove quarantine preflight includes simultaneous source/temp/final copies; if it cannot fit, mutation is refused without exceeding quota. Repeat with an unsweepable quarantine file and a read-only/newer-schema combination, proving the retained file remains preserved and charged across reopen.
- Host lifecycle: inject a current provider death during an active tool, with unresolved approval/question rows, and during an in-flight send. Assert terminal tool/prompt/status rows precede lifecycle tombstones, sink-barrier success, observed-exit `claimStatus: released` with fence + 1, no stale-fence or stale-generation effect, no automatic redispatch of an unknown message, successful reacquisition for a resume-capable hold, no reacquisition for subscription-only holds, and successful dispatch of a distinct next message on the same chat. Make the recovery test time-bounded to detect keyed-queue reentrancy deadlock. Queue a handoff after release but before attach and prove ticket revalidation cancels auto-reattach without creating a second owner. Inject a failed barrier and an absent/torn final batch: owner release must still complete, attach/dead-owner recovery must write exactly one wholly terminal batch before publication/reacquisition, and persistent retry failure must refuse attach. Separately assert deliberate close/eviction/superseding acquisition emits no recovery, cancels prompts, and cover an unexpected exit racing a handoff without creating a second owner or leaving a dead native owner live.
- Runtime integration: construct the production runtime path (not a test-only adapter callback), deliver a valid frame above 1 MiB, complete the turn, then send another turn. Separately prove an unexpected death reaches host recovery through the production callback while a requested close does not.
Use temporary profile/journal directories and scripted child processes so no test depends on a developer's account or global runtime.
### 5. Validation and evidence
Run the focused connection, translation, journal, lifecycle, and runtime integration tests; then `pnpm tc:node`, `pnpm run check:code-quality:changed`, and the structured agent-session cross-version test. The row shapes remain compatible, but newly published terminal/status/prompt-cancellation content must be exercised against the mixed-version contract.
Finally use Playwright CDP through the Electron skill against the dev build launched from this exact worktree. In a real native structured chat, run a command that emits more than 1 MiB, observe the rendered tool reach a terminal successful state with bounded output, and send a second user message in the same chat that receives a normal assistant completion. Also exercise a controlled unexpected provider exit, observe terminal tool/prompt/turn state and visible recovery, then send a new message through the reacquired child. Record the worktree identity, command/result byte evidence, visible state, backing journal/session state, and any remaining gap in a concise worktree evidence file. Do not use desktop-control automation and do not create a PR.
## Scope guard
Do not change provider output semantics, add a remote capability/opcode, redesign all provider transports, or automatically retry unknown messages. Limit adjacent refactors to the shared JSONL framer, bounded command-result persistence needed by this failure, and current-generation provider-exit recovery required to keep the attached session truthful and usable. Do not broaden the frame envelope to accommodate unbounded history; refuse that response class explicitly and recoverably.
@@ -22,6 +22,10 @@ export type CodexAppServerConnection = {
notify: (method: string, params?: Record<string, unknown>) => void
respond: (id: number | string, result: unknown) => void
respondWithError: (id: number | string, code: number, message: string) => void
/** Stops provider stdout at a record boundary while a durable sink drains. */
pauseReading?: () => void
/** Continues with any records retained from the chunk that triggered the pause. */
resumeReading?: () => void
/** Resolves true only after the child emitted `exit` or `close`; false is unproven. */
close: () => Promise<boolean>
}
@@ -5,6 +5,8 @@ import { PassThrough } from 'node:stream'
import { afterEach, describe, expect, it, vi } from 'vitest'
import type { spawnProcess } from '../../shared/child-process/run-process'
import {
CODEX_APP_SERVER_MAX_RECORD_BYTES,
CodexAppServerFrameSizeError,
isCodexAppServerRequestError,
openCodexAppServerConnection,
type CodexAppServerConnection,
@@ -128,6 +130,67 @@ function rejection(promise: Promise<unknown>): Promise<Error> {
)
}
function commandCompletionFixture(
targetBytes: number,
itemId = 'item-large'
): { line: string; output: string } {
const frame = {
method: 'item/completed',
params: {
turnId: 'turn-large',
item: { id: itemId, type: 'commandExecution', aggregated_output: '' }
}
}
const emptyBytes = Buffer.byteLength(JSON.stringify(frame), 'utf8')
const remaining = targetBytes - emptyBytes
if (remaining < 0) {
throw new Error(`target ${targetBytes} is smaller than fixture envelope ${emptyBytes}`)
}
const output = `${'\n'.repeat(Math.floor(remaining / 2))}${remaining % 2 ? 'x' : ''}`
frame.params.item.aggregated_output = output
const line = JSON.stringify(frame)
expect(Buffer.byteLength(line, 'utf8')).toBe(targetBytes)
return { line: `${line}\n`, output }
}
function commandCompletionLine(targetBytes: number): string {
return commandCompletionFixture(targetBytes).line
}
function responseLine(targetBytes: number, id: number): string {
const frame = { id, result: { data: '' } }
const emptyBytes = Buffer.byteLength(JSON.stringify(frame), 'utf8')
frame.result.data = 'x'.repeat(targetBytes - emptyBytes)
const line = JSON.stringify(frame)
expect(Buffer.byteLength(line, 'utf8')).toBe(targetBytes)
return `${line}\n`
}
function resultFirstResponseLine(targetBytes: number, id: number, resultKey: 'result' | 'error') {
const response =
resultKey === 'result'
? `{"result":{"turn":{"id":"turn-large"}},"id":${id},"padding":"`
: `{"error":{"code":-32000,"message":"too large"},"id":${id},"padding":"`
const suffix = '"}'
const padding = targetBytes - Buffer.byteLength(response + suffix, 'utf8')
if (padding < 0) {
throw new Error(`target ${targetBytes} is smaller than fixture envelope`)
}
const line = `${response}${'x'.repeat(padding)}${suffix}`
expect(Buffer.byteLength(line, 'utf8')).toBe(targetBytes)
return `${line}\n`
}
function giantContainerBeforeIdResponseLine(targetBytes: number, id: number): string {
const giantResult = `{"result":{"payload":"${'x'.repeat(62_000)}"},"id":${id},"padding":"`
const suffix = '"}'
const padding = targetBytes - Buffer.byteLength(giantResult + suffix, 'utf8')
if (padding < 0) {
throw new Error(`target ${targetBytes} is smaller than giant response envelope`)
}
return `${giantResult}${'x'.repeat(padding)}${suffix}\n`
}
describe('openCodexAppServerConnection', () => {
it('advertises the experimental API required for rollout-path resume', async () => {
const { child, spawnImpl, written } = stubChild()
@@ -413,25 +476,255 @@ describe('openCodexAppServerConnection', () => {
await expect(connection.close()).resolves.toBe(true)
})
it('ends the connection rather than buffering an oversized line', async () => {
it.each([1_090_188, 2_900_090])(
'accepts a realistic %i-byte escaped command completion and keeps processing',
async (frameBytes) => {
const { child, spawnImpl } = stubChild()
answerInitialize(child)
const completed: unknown[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{
onNotification: (method, params) => {
if (method === 'item/completed') {
completed.push(params)
}
}
},
spawnImpl
)
const line = Buffer.from(commandCompletionLine(frameBytes), 'utf8')
const split = Math.floor(line.length / 3)
child.stdout.write(line.subarray(0, split))
child.stdout.write(line.subarray(split, split * 2))
child.stdout.write(line.subarray(split * 2))
child.stdout.write('{"method":"turn/completed","params":{"turn":{"id":"turn-large"}}}\n')
await vi.waitFor(() => expect(completed).toHaveLength(1))
expect(
(completed[0] as { item: { aggregated_output: string } }).item.aggregated_output.length
).toBeGreaterThan(500_000)
expect(connection.closed).toBe(false)
await connection.close()
}
)
it('accepts two realistic large command completions without losing either payload', async () => {
const { child, spawnImpl } = stubChild()
answerInitialize(child)
const completed: { item: { id: string; aggregated_output: string } }[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{
onNotification: (method, params) => {
if (method === 'item/completed') {
completed.push(params as { item: { id: string; aggregated_output: string } })
}
}
},
spawnImpl
)
const fixtures = [
commandCompletionFixture(1_090_188, 'item-large-a'),
commandCompletionFixture(2_900_090, 'item-large-b')
]
child.stdout.write(fixtures[0]!.line)
child.stdout.write(fixtures[1]!.line)
await vi.waitFor(() => expect(completed).toHaveLength(2))
expect(completed.map((entry) => entry.item.id)).toEqual(['item-large-a', 'item-large-b'])
expect(
completed.map((entry) => Buffer.byteLength(entry.item.aggregated_output, 'utf8'))
).toEqual(fixtures.map((fixture) => Buffer.byteLength(fixture.output, 'utf8')))
expect(connection.closed).toBe(false)
await connection.close()
})
it('accepts the 16 MiB boundary and settles one byte above without killing the provider', async () => {
const { child, spawnImpl } = stubChild({ exitOnStdinEnd: false })
answerInitialize(child)
const exits: string[] = []
const frames: { kind: string; payload: unknown }[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{ onExit: (error) => exits.push(error.message) },
{
onExit: (error) => exits.push(error.message),
onUnhandledFrame: (kind, payload) => frames.push({ kind, payload })
},
spawnImpl
)
child.kill.mockImplementation(() => {
child.emit('exit', null, 'SIGKILL')
return true
})
const below = connection.request('thread/resume')
child.stdout.write(responseLine(CODEX_APP_SERVER_MAX_RECORD_BYTES - 1, 2))
expect(((await below) as { data: string }).data.length).toBeGreaterThan(
CODEX_APP_SERVER_MAX_RECORD_BYTES - 40
)
const at = connection.request('thread/resume')
child.stdout.write(responseLine(CODEX_APP_SERVER_MAX_RECORD_BYTES, 3))
expect(((await at) as { data: string }).data.length).toBeGreaterThan(
CODEX_APP_SERVER_MAX_RECORD_BYTES - 40
)
const inFlight = rejection(connection.request('turn/start'))
child.stdout.write('x'.repeat(1024 * 1024 + 1))
child.stdout.write(responseLine(CODEX_APP_SERVER_MAX_RECORD_BYTES + 1, 4))
expect(await inFlight).toBeInstanceOf(CodexAppServerFrameSizeError)
expect(frames).toEqual([
{
kind: 'frame:oversized-response',
payload: expect.objectContaining({ classification: 'response', id: 4 })
}
])
expect(exits).toEqual([])
expect(connection.closed).toBe(false)
const later = connection.request('turn/start')
child.stdout.write('{"id":5,"result":{"turn":{"id":"turn-next"}}}\n')
await expect(later).resolves.toEqual({ turn: { id: 'turn-next' } })
child.emit('exit', 0, null)
await connection.close()
})
it.each(['result', 'error'] as const)(
'classifies oversized responses with %s before id',
async (resultKey) => {
const { child, spawnImpl } = stubChild({ exitOnStdinEnd: false })
answerInitialize(child)
const frames: { kind: string; payload: unknown }[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{ onUnhandledFrame: (kind, payload) => frames.push({ kind, payload }) },
spawnImpl
)
const inFlight = rejection(connection.request('thread/resume'))
child.stdout.write(
resultFirstResponseLine(CODEX_APP_SERVER_MAX_RECORD_BYTES + 1, 2, resultKey)
)
expect(await inFlight).toBeInstanceOf(CodexAppServerFrameSizeError)
expect(frames).toEqual([
{
kind: 'frame:oversized-response',
payload: expect.objectContaining({ classification: 'response', id: 2 })
}
])
expect(connection.closed).toBe(false)
child.emit('exit', 0, null)
await connection.close()
}
)
it('classifies an oversized response when a giant result container precedes id', async () => {
const { child, spawnImpl } = stubChild({ exitOnStdinEnd: false })
answerInitialize(child)
const frames: { kind: string; payload: unknown }[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{ onUnhandledFrame: (kind, payload) => frames.push({ kind, payload }) },
spawnImpl
)
const inFlight = rejection(connection.request('thread/resume'))
child.stdout.write(giantContainerBeforeIdResponseLine(CODEX_APP_SERVER_MAX_RECORD_BYTES + 1, 2))
await expect(inFlight).resolves.toBeInstanceOf(CodexAppServerFrameSizeError)
expect(frames).toEqual([
{
kind: 'frame:oversized-response',
payload: expect.objectContaining({ classification: 'response', id: 2 })
}
])
child.emit('exit', 0, null)
await connection.close()
})
it('answers an oversized provider request once and resumes after its newline', async () => {
const { child, spawnImpl, written } = stubChild()
answerInitialize(child)
const frames: string[] = []
const notifications: string[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{
onUnhandledFrame: (kind) => frames.push(kind),
onNotification: (method) => notifications.push(method)
},
spawnImpl
)
child.stdout.write(
`{"id":"approval-1","method":"item/requestApproval","params":{"data":"${'x'.repeat(
CODEX_APP_SERVER_MAX_RECORD_BYTES
)}"}}\n{"method":"turn/completed","params":{}}\n`
)
await vi.waitFor(() => expect(notifications).toEqual(['turn/completed']))
expect(frames).toEqual(['frame:oversized-request'])
expect(written.at(-1)).toEqual({
id: 'approval-1',
error: {
code: -32001,
message: `request exceeds ${CODEX_APP_SERVER_MAX_RECORD_BYTES} byte limit`
}
})
expect(connection.closed).toBe(false)
await connection.close()
})
it('keeps malformed and non-object JSON non-fatal and processes the next record', async () => {
const { child, spawnImpl } = stubChild()
answerInitialize(child)
const frames: { kind: string; payload: unknown }[] = []
const notifications: string[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{
onUnhandledFrame: (kind, payload) => frames.push({ kind, payload }),
onNotification: (method) => notifications.push(method)
},
spawnImpl
)
child.stdout.write('not json\n[]\n{"method":"turn/completed","params":{}}\n')
await vi.waitFor(() => expect(notifications).toEqual(['turn/completed']))
expect(frames).toEqual([
{ kind: 'frame:invalid-json', payload: 'not json' },
{ kind: 'frame:invalid-json', payload: '[]' }
])
expect(connection.closed).toBe(false)
await connection.close()
})
it('pauses between coalesced records and resumes the retained remainder', async () => {
const { child, spawnImpl } = stubChild()
answerInitialize(child)
const notifications: string[] = []
let connection: CodexAppServerConnection
connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{
onNotification: (method) => {
notifications.push(method)
if (notifications.length === 1) {
connection.pauseReading?.()
}
}
},
spawnImpl
)
child.stdout.write(
'{"method":"item/started","params":{}}\n{"method":"item/completed","params":{}}\n'
)
await vi.waitFor(() => expect(notifications).toEqual(['item/started']))
connection.resumeReading?.()
await vi.waitFor(() => expect(notifications).toEqual(['item/started', 'item/completed']))
expect((await inFlight).message).toContain('oversized')
expect(exits[0]).toContain('oversized')
await connection.close()
})
@@ -485,8 +778,8 @@ describe('openCodexAppServerConnection', () => {
spawnImpl
)
// The oversized line kills the child, so its own `close` lands afterwards.
child.stdout.write('x'.repeat(1024 * 1024 + 1))
// An unclassifiable oversized line initiates recovery, then child exit lands afterwards.
child.stdout.write('x'.repeat(CODEX_APP_SERVER_MAX_RECORD_BYTES + 1))
child.stderr.write('killed\n')
await flushStreams()
child.emit('exit', null, 'SIGKILL')
@@ -498,6 +791,28 @@ describe('openCodexAppServerConnection', () => {
await connection.close()
})
it('does not report recovery for a protocol failure until child exit is observed', async () => {
const { child, spawnImpl } = stubChild({ exitOnStdinEnd: false })
answerInitialize(child)
const exits: string[] = []
const connection = await openCodexAppServerConnection(
{ command: 'codex', args: ['app-server'] },
{ onExit: (error) => exits.push(error.message) },
spawnImpl
)
const inFlight = rejection(connection.request('turn/start'))
child.stdout.write('x'.repeat(CODEX_APP_SERVER_MAX_RECORD_BYTES + 1))
await flushStreams()
expect(exits).toHaveLength(0)
expect((await inFlight).message).toContain('oversized')
child.emit('exit', null, 'SIGKILL')
expect(exits).toHaveLength(1)
await connection.close()
})
it('treats a broken stdin pipe as the end of the transport', async () => {
const { child, spawnImpl } = stubChild({ exitOnStdinEnd: false })
answerInitialize(child)
+52 -103
View File
@@ -4,16 +4,16 @@ import { createProviderSpawnSpec } from './codex-app-server-posix-supervisor'
import { buildCodexAppServerExitError } from './codex-app-server-exit-error'
import { initializeCodexAppServerConnection } from './codex-app-server-handshake'
import { CodexAppServerHandshakeExitUnprovenError } from './codex-app-server-handshake-exit-proof'
import { isAppServerRecord, parseCodexAppServerJsonLine } from './codex-app-server-jsonl'
import { terminateCodexAppServerProcessTree } from './codex-app-server-process-teardown'
import { CodexAppServerRequestError } from './codex-app-server-request-error'
import { CODEX_SPAWN_TOKEN_ENV } from './codex-structured-owner-identity'
import { waitForProcessExitUntil } from './codex-process-exit-deadline'
import { NDJSON_MAX_LINE_BYTES } from '../../shared/main-process-ndjson-framer'
import {
CodexAppServerTimeoutError,
CodexAppServerUnsupportedError,
isCodexMethodNotFoundError
CodexAppServerUnsupportedError
} from './codex-app-server-session'
import { createCodexAppServerRecordDispatcher } from './codex-app-server-record-dispatch'
import { createCodexAppServerRecordReader } from './codex-app-server-record-reader'
import type {
CodexAppServerConnection,
CodexAppServerConnectionHandlers
@@ -28,6 +28,7 @@ export {
CodexAppServerRequestError,
isCodexAppServerRequestError
} from './codex-app-server-request-error'
export { CodexAppServerFrameSizeError } from './codex-app-server-frame-size-error'
// Structured chat needs a persistent bidirectional child and per-request deadlines;
// the request-scoped app-server runner cannot carry approvals or streamed turns.
@@ -47,14 +48,7 @@ const DEFAULT_REQUEST_TIMEOUT_MS = 30_000
const GRACEFUL_EXIT_MS = 1_500
const FORCED_EXIT_MS = 1_000
const STDERR_TAIL_MAX_BYTES = 8192
const STDOUT_LINE_MAX_BYTES = 1024 * 1024
type PendingRequest = {
method: string
resolve: (result: unknown) => void
reject: (error: Error) => void
timer: ReturnType<typeof setTimeout>
}
export const CODEX_APP_SERVER_MAX_RECORD_BYTES = NDJSON_MAX_LINE_BYTES
/**
* Spawns `codex app-server`, completes the initialize handshake, and returns a
@@ -80,12 +74,12 @@ export async function openCodexAppServerConnection(
return terminateCodexAppServerProcessTree(child, spawnToken)
}
const pending = new Map<number, PendingRequest>()
let stderrTail = ''
let nextRequestId = 1
let exited = false
let exitObserved = false
let closing = false
let exitReported = false
const exitProof = new RetryableProcessExitProof()
/** First terminal cause, or null while the transport is still usable. Set once:
* a child that dies reaches us through several listeners, and the specific
@@ -103,31 +97,38 @@ export async function openCodexAppServerConnection(
resolveExit()
}
child.on('exit', observeExit)
child.on('exit', () => {
observeExit()
handleUnexpectedEnd()
})
function buildExitError(cause?: Error): Error {
return buildCodexAppServerExitError(stderrTail, cause)
}
function failPending(error: Error): void {
for (const waiter of pending.values()) {
clearTimeout(waiter.timer)
waiter.reject(error)
const dispatcher = createCodexAppServerRecordDispatcher({
handlers,
writeResponse,
onProtocolFailure: (error) => {
handleUnexpectedEnd(error)
void terminateProcessTree()
}
pending.clear()
}
})
/** A death nobody asked for kills every in-flight call AND tells the owner,
* which is the only signal the session has that its lease is now worthless.
* Once only: an oversized line kills the child and its `close` arrives after,
* and a spawn failure arrives as both `error` and `close`. */
function handleUnexpectedEnd(cause?: Error): void {
if (terminalError) {
return
if (!terminalError) {
terminalError = buildExitError(cause)
dispatcher.failPending(terminalError)
}
terminalError = buildExitError(cause)
failPending(terminalError)
if (!closing) {
// Transport/protocol failures make the connection unusable immediately so
// callers do not hang, but recovery must not treat that as a child exit
// until the execution host has observed `exit`/`close`.
if (exitObserved && !closing && !exitReported) {
exitReported = true
handlers.onExit?.(terminalError)
}
}
@@ -149,87 +150,33 @@ export async function openCodexAppServerConnection(
// close the reap is already under way and `exited` must stay honest, or
// `close` would skip the kill it still owes.
if (closing) {
failPending(error)
dispatcher.failPending(error)
return
}
void terminateProcessTree()
handleUnexpectedEnd(error)
void terminateProcessTree()
})
function dispatchMessage(message: Record<string, unknown>): void {
const hasMethod = typeof message.method === 'string'
const hasId = typeof message.id === 'number' || typeof message.id === 'string'
if (hasMethod && hasId) {
handlers.onServerRequest?.({
id: message.id as number | string,
method: message.method as string,
params: message.params
})
return
}
if (hasMethod) {
handlers.onNotification?.(message.method as string, message.params)
return
}
if (typeof message.id !== 'number') {
handlers.onUnhandledFrame?.('frame:unclassified', message)
return
}
const waiter = pending.get(message.id)
if (!waiter) {
handlers.onUnhandledFrame?.('response:unmatched', message)
return
}
pending.delete(message.id)
clearTimeout(waiter.timer)
const error = message.error
if (isAppServerRecord(error)) {
const detail = typeof error.message === 'string' ? error.message : 'unknown error'
waiter.reject(
isCodexMethodNotFoundError(error)
? new CodexAppServerUnsupportedError(
`codex app-server does not support ${waiter.method}: ${detail}`
)
: new CodexAppServerRequestError(
waiter.method,
typeof error.code === 'number' ? error.code : null,
`codex app-server ${waiter.method} failed: ${detail}`
)
)
return
}
waiter.resolve(message.result)
}
let stdoutBuffer = ''
child.stdout.setEncoding('utf8').on('data', (chunk: string) => {
stdoutBuffer += chunk
if (Buffer.byteLength(stdoutBuffer) > STDOUT_LINE_MAX_BYTES) {
child.stdout.destroy()
void terminateProcessTree()
handleUnexpectedEnd(new Error('codex app-server emitted an oversized JSONL line'))
return
}
let newlineIndex: number
while ((newlineIndex = stdoutBuffer.indexOf('\n')) !== -1) {
const line = stdoutBuffer.slice(0, newlineIndex).trim()
stdoutBuffer = stdoutBuffer.slice(newlineIndex + 1)
if (!line) {
continue
}
const parsed = parseCodexAppServerJsonLine(line)
if (!parsed) {
const recordReader = createCodexAppServerRecordReader({
stdout: child.stdout,
maxRecordBytes: CODEX_APP_SERVER_MAX_RECORD_BYTES,
onRecord: (parsed, line) => {
if (typeof parsed !== 'object' || parsed === null || Array.isArray(parsed)) {
handlers.onUnhandledFrame?.('frame:invalid-json', line)
continue
}
try {
dispatchMessage(parsed)
} catch (error) {
child.stdout.destroy()
void terminateProcessTree()
handleUnexpectedEnd(error instanceof Error ? error : new Error(String(error)))
return
}
dispatcher.dispatch(parsed as Record<string, unknown>)
},
onRejected: (rejected) => {
if (rejected.kind === 'invalid-json') {
handlers.onUnhandledFrame?.('frame:invalid-json', rejected.line)
} else {
dispatcher.rejectOversized(rejected)
}
},
onFatal: (error) => {
handleUnexpectedEnd(error)
void terminateProcessTree()
}
})
@@ -268,14 +215,14 @@ export async function openCodexAppServerConnection(
// Why: per request, not per session — a chat session outlives every call,
// so only the individual call can carry a deadline.
const timer = setTimeout(() => {
pending.delete(id)
dispatcher.deletePending(id)
reject(new CodexAppServerTimeoutError(`codex app-server ${method} exceeded ${timeoutMs}ms`))
}, timeoutMs)
pending.set(id, { method, resolve, reject, timer })
dispatcher.addPending(id, { method, resolve, reject, timer })
try {
sendLine(params === undefined ? { method, id } : { method, id, params })
} catch (error) {
pending.delete(id)
dispatcher.deletePending(id)
clearTimeout(timer)
reject(error instanceof Error ? error : new Error(String(error)))
}
@@ -309,13 +256,13 @@ export async function openCodexAppServerConnection(
if (!exited) {
const treeExited = await terminateProcessTree()
if (!treeExited) {
failPending(new Error('codex app-server process-tree exit was not proven'))
dispatcher.failPending(new Error('codex app-server process-tree exit was not proven'))
return false
}
await waitForProcessExitUntil(exitPromise, FORCED_EXIT_MS)
}
}
failPending(new Error('codex app-server connection closed'))
dispatcher.failPending(new Error('codex app-server connection closed'))
return exitObserved
})
}
@@ -331,6 +278,8 @@ export async function openCodexAppServerConnection(
notify,
respond: (id, result) => writeResponse({ id, result }),
respondWithError: (id, code, message) => writeResponse({ id, error: { code, message } }),
pauseReading: recordReader.pause,
resumeReading: recordReader.resume,
close
}
@@ -0,0 +1,12 @@
export class CodexAppServerFrameSizeError extends Error {
constructor(
readonly method: string | null,
readonly observedBytes: number,
readonly maxBytes: number
) {
super(
`codex app-server${method ? ` ${method}` : ''} response exceeds ${maxBytes} byte limit (${observedBytes} bytes received)`
)
this.name = 'CodexAppServerFrameSizeError'
}
}
@@ -0,0 +1,166 @@
import type { NdjsonRejectedRecord } from '../../shared/main-process-ndjson-framer'
import type { CodexAppServerConnectionHandlers } from './codex-app-server-connection-types'
import { CodexAppServerFrameSizeError } from './codex-app-server-frame-size-error'
import { isAppServerRecord } from './codex-app-server-jsonl'
import { CodexAppServerRequestError } from './codex-app-server-request-error'
import {
CodexAppServerUnsupportedError,
isCodexMethodNotFoundError
} from './codex-app-server-session'
import { classifyJsonRpcPrefix } from './codex-app-server-record-prefix'
const OVERSIZED_REQUEST_ERROR_CODE = -32001
export type CodexPendingRequest = {
method: string
resolve: (result: unknown) => void
reject: (error: Error) => void
timer: ReturnType<typeof setTimeout>
}
export function createCodexAppServerRecordDispatcher(input: {
handlers: CodexAppServerConnectionHandlers
writeResponse: (payload: Record<string, unknown>) => void
onProtocolFailure: (error: Error) => void
}): {
addPending: (id: number, waiter: CodexPendingRequest) => void
deletePending: (id: number) => void
failPending: (error: Error) => void
dispatch: (message: Record<string, unknown>) => void
rejectOversized: (rejected: NdjsonRejectedRecord & { kind: 'line-too-long' }) => void
} {
const pending = new Map<number, CodexPendingRequest>()
const failPending = (error: Error): void => {
for (const waiter of pending.values()) {
clearTimeout(waiter.timer)
waiter.reject(error)
}
pending.clear()
}
const failPendingForOversizedUnknown = (
record: NdjsonRejectedRecord & { kind: 'line-too-long' }
): void => {
for (const waiter of pending.values()) {
clearTimeout(waiter.timer)
waiter.reject(
new CodexAppServerFrameSizeError(waiter.method, record.observedBytes, record.maxLineBytes)
)
}
pending.clear()
}
const dispatch = (message: Record<string, unknown>): void => {
const hasMethod = typeof message.method === 'string'
const hasId = typeof message.id === 'number' || typeof message.id === 'string'
if (hasMethod && hasId) {
input.handlers.onServerRequest?.({
id: message.id as number | string,
method: message.method as string,
params: message.params
})
return
}
if (hasMethod) {
input.handlers.onNotification?.(message.method as string, message.params)
return
}
if (typeof message.id !== 'number') {
input.handlers.onUnhandledFrame?.('frame:unclassified', message)
return
}
const waiter = pending.get(message.id)
if (!waiter) {
input.handlers.onUnhandledFrame?.('response:unmatched', message)
return
}
pending.delete(message.id)
clearTimeout(waiter.timer)
const error = message.error
if (isAppServerRecord(error)) {
const detail = typeof error.message === 'string' ? error.message : 'unknown error'
waiter.reject(
isCodexMethodNotFoundError(error)
? new CodexAppServerUnsupportedError(
`codex app-server does not support ${waiter.method}: ${detail}`
)
: new CodexAppServerRequestError(
waiter.method,
typeof error.code === 'number' ? error.code : null,
`codex app-server ${waiter.method} failed: ${detail}`
)
)
return
}
waiter.resolve(message.result)
}
const rejectOversized = (rejected: NdjsonRejectedRecord & { kind: 'line-too-long' }): void => {
const classification = classifyJsonRpcPrefix(rejected.prefix)
const payload = {
reason: 'record-too-large',
observedBytes: rejected.observedBytes,
maxBytes: rejected.maxLineBytes,
classification: classification.kind,
...('id' in classification ? { id: classification.id } : {}),
...('method' in classification ? { method: classification.method } : {})
}
if (classification.kind === 'response') {
const waiter = pending.get(classification.id)
if (waiter) {
pending.delete(classification.id)
clearTimeout(waiter.timer)
waiter.reject(
new CodexAppServerFrameSizeError(
waiter.method,
rejected.observedBytes,
rejected.maxLineBytes
)
)
} else {
input.handlers.onUnhandledFrame?.('frame:oversized-response', payload)
failPendingForOversizedUnknown(rejected)
input.onProtocolFailure(
new Error(
`codex app-server oversized response ${classification.id} had no pending request`
)
)
return
}
input.handlers.onUnhandledFrame?.('frame:oversized-response', payload)
return
}
if (classification.kind === 'server-request') {
input.writeResponse({
id: classification.id,
error: {
code: OVERSIZED_REQUEST_ERROR_CODE,
message: `request exceeds ${rejected.maxLineBytes} byte limit`
}
})
input.handlers.onUnhandledFrame?.('frame:oversized-request', payload)
return
}
if (classification.kind === 'notification') {
input.handlers.onUnhandledFrame?.('frame:oversized-notification', payload)
return
}
input.handlers.onUnhandledFrame?.('frame:oversized-unclassified', payload)
input.onProtocolFailure(
new Error(
classification.kind === 'response-unknown'
? 'codex app-server emitted an oversized response with an unknown shape'
: 'codex app-server emitted an oversized unclassifiable JSONL record'
)
)
}
return {
addPending: (id, waiter) => pending.set(id, waiter),
deletePending: (id) => pending.delete(id),
failPending,
dispatch,
rejectOversized
}
}
@@ -0,0 +1,186 @@
export type JsonRpcPrefix =
| { kind: 'response'; id: number }
| { kind: 'server-request'; id: number | string; method: string }
| { kind: 'notification'; method: string }
| { kind: 'response-unknown' }
| { kind: 'unknown' }
type LeadingProperty = { key: string; value?: string | number }
function readJsonStringEnd(value: string, start: number): number | null {
if (value[start] !== '"') {
return null
}
let escaped = false
for (let index = start + 1; index < value.length; index += 1) {
const character = value[index]
if (escaped) {
escaped = false
} else if (character === '\\') {
escaped = true
} else if (character === '"') {
return index + 1
}
}
return null
}
function skipJsonContainer(value: string, start: number): number | null {
const opening = value[start]
const closing = opening === '{' ? '}' : opening === '[' ? ']' : null
if (!closing) {
return null
}
const stack = [closing]
let escaped = false
let inString = false
for (let index = start + 1; index < value.length; index += 1) {
const character = value[index]
if (inString) {
if (escaped) {
escaped = false
} else if (character === '\\') {
escaped = true
} else if (character === '"') {
inString = false
}
continue
}
if (character === '"') {
inString = true
continue
}
if (character === '{') {
stack.push('}')
} else if (character === '[') {
stack.push(']')
} else if (character === stack.at(-1)) {
stack.pop()
if (stack.length === 0) {
return index + 1
}
}
}
return null
}
function skipJsonLiteral(value: string, start: number): number | null {
for (const literal of ['true', 'false', 'null']) {
if (value.startsWith(literal, start)) {
return start + literal.length
}
}
return null
}
function leadingJsonRpcProperties(prefix: string): LeadingProperty[] {
const properties: LeadingProperty[] = []
let cursor = 0
const skipWhitespace = (): void => {
while (/\s/.test(prefix[cursor] ?? '')) {
cursor += 1
}
}
skipWhitespace()
if (prefix[cursor] !== '{') {
return properties
}
cursor += 1
while (properties.length < 8) {
skipWhitespace()
const keyEnd = readJsonStringEnd(prefix, cursor)
if (keyEnd === null) {
break
}
let key: unknown
try {
key = JSON.parse(prefix.slice(cursor, keyEnd))
} catch {
break
}
cursor = keyEnd
skipWhitespace()
if (prefix[cursor] !== ':') {
break
}
cursor += 1
skipWhitespace()
if (prefix[cursor] === '{' || prefix[cursor] === '[') {
properties.push({ key: String(key) })
const valueEnd = skipJsonContainer(prefix, cursor)
if (valueEnd === null) {
break
}
cursor = valueEnd
skipWhitespace()
if (prefix[cursor] !== ',') {
break
}
cursor += 1
continue
}
const literalEnd = skipJsonLiteral(prefix, cursor)
if (literalEnd !== null) {
properties.push({ key: String(key) })
cursor = literalEnd
skipWhitespace()
if (prefix[cursor] !== ',') {
break
}
cursor += 1
continue
}
const stringEnd = readJsonStringEnd(prefix, cursor)
if (stringEnd !== null) {
try {
properties.push({ key: String(key), value: JSON.parse(prefix.slice(cursor, stringEnd)) })
} catch {
break
}
cursor = stringEnd
skipWhitespace()
if (prefix[cursor] !== ',') {
break
}
cursor += 1
continue
}
const match = /^-?(?:0|[1-9]\d*)/.exec(prefix.slice(cursor))
if (!match) {
break
}
properties.push({ key: String(key), value: Number(match[0]) })
cursor += match[0].length
skipWhitespace()
if (prefix[cursor] !== ',') {
break
}
cursor += 1
}
return properties
}
export function classifyJsonRpcPrefix(prefix: string): JsonRpcPrefix {
// Only inspect complete top-level properties. Searching arbitrary quoted
// keys would let a nested result/params object impersonate JSON-RPC fields.
const properties = leadingJsonRpcProperties(prefix)
const method = properties.find((property) => property.key === 'method')?.value
const id = properties.find((property) => property.key === 'id')?.value
if (typeof method === 'string' && (typeof id === 'number' || typeof id === 'string')) {
return { kind: 'server-request', id, method }
}
if (
typeof id === 'number' &&
properties.some((property) => property.key === 'id') &&
properties.some((property) => property.key === 'result' || property.key === 'error')
) {
return { kind: 'response', id }
}
if (typeof id === 'number') {
return { kind: 'response-unknown' }
}
if (typeof method === 'string' && properties.some((property) => property.key === 'params')) {
return { kind: 'notification', method }
}
return { kind: 'unknown' }
}
@@ -0,0 +1,51 @@
import type { Readable } from 'node:stream'
import {
createIncrementalNdjsonFramer,
type NdjsonRejectedRecord
} from '../../shared/main-process-ndjson-framer'
type RecordReaderStream = Pick<Readable, 'on' | 'pause' | 'resume' | 'setEncoding'>
export type CodexAppServerRecordReader = {
pause: () => void
resume: () => void
}
export function createCodexAppServerRecordReader(input: {
stdout: RecordReaderStream
maxRecordBytes: number
onRecord: (record: unknown, line: string) => void
onRejected: (rejected: NdjsonRejectedRecord) => void
onFatal: (error: Error) => void
}): CodexAppServerRecordReader {
let paused = false
const framer = createIncrementalNdjsonFramer(input.onRecord, input.onRejected, {
maxLineBytes: input.maxRecordBytes,
shouldPause: () => paused
})
input.stdout.setEncoding('utf8').on('data', (chunk: string) => {
try {
framer.feed(chunk)
} catch (error) {
input.onFatal(error instanceof Error ? error : new Error(String(error)))
}
})
return {
pause: () => {
paused = true
input.stdout.pause()
},
resume: () => {
if (!paused) {
return
}
paused = false
framer.resume()
if (!paused) {
input.stdout.resume()
}
}
}
}
@@ -0,0 +1,108 @@
import {
boundPayload,
digestPayload
} from '../native-chat/agent-session-journal/journal-payload-bounds'
export const CODEX_JOURNAL_PROMPT_ID_COMPONENT_MAX_BYTES = 256
export const CODEX_JOURNAL_PROMPT_OPTION_ID_MAX_BYTES = 1024
export const CODEX_PROMPT_MAX_QUESTIONS = 64
export const CODEX_PROMPT_MAX_QUESTION_BYTES = 32 * 1024
export const CODEX_PROMPT_MAX_OPTIONS = 256
export const CODEX_PROMPT_MAX_OPTION_BYTES = 64 * 1024
export const CODEX_PROMPT_MAX_ANSWER_BYTES = 64 * 1024
export const MAX_CODEX_PROMPT_REGISTRY_ENTRIES = 128
export const MAX_CODEX_PROMPT_JOURNAL_BINDINGS = 256
export const MAX_CODEX_PROMPT_REGISTRY_BYTES = 4 * 1024 * 1024
export function codexJournalPromptIdPart(value: string): string {
if (Buffer.byteLength(value, 'utf8') <= CODEX_JOURNAL_PROMPT_ID_COMPONENT_MAX_BYTES) {
return value
}
const suffix = `#${digestPayload(value).slice(0, 32)}`
const bounded = boundPayload(value, {
inlineHeadBytes: CODEX_JOURNAL_PROMPT_ID_COMPONENT_MAX_BYTES - suffix.length,
maxSessionBytes: Number.MAX_SAFE_INTEGER,
maxAppendsPerWindow: Number.MAX_SAFE_INTEGER,
appendWindowMs: Number.MAX_SAFE_INTEGER
})
return `${bounded.head}${suffix}`
}
export function encodeCodexJournalQuestionOptionId(questionId: string, answer: string): string {
const exact = `${encodeURIComponent(questionId)}:${encodeURIComponent(answer)}`
if (Buffer.byteLength(exact, 'utf8') <= CODEX_JOURNAL_PROMPT_OPTION_ID_MAX_BYTES) {
return exact
}
const bounded = `${encodeURIComponent(codexJournalPromptIdPart(questionId))}:${encodeURIComponent(codexJournalPromptIdPart(answer))}`
if (Buffer.byteLength(bounded, 'utf8') <= CODEX_JOURNAL_PROMPT_OPTION_ID_MAX_BYTES) {
return bounded
}
return `#${digestPayload(questionId).slice(0, 32)}:#${digestPayload(answer).slice(0, 32)}`
}
export function readQuestionIds(params: unknown): string[] | null {
const questions = (params as { questions?: unknown } | null)?.questions
if (!Array.isArray(questions)) {
return []
}
const ids: string[] = []
let bytes = 0
for (const question of questions) {
const id = (question as { id?: unknown })?.id
if (typeof id !== 'string' || id.length === 0) {
continue
}
if (ids.length >= CODEX_PROMPT_MAX_QUESTIONS) {
return null
}
bytes += Buffer.byteLength(id, 'utf8')
if (bytes > CODEX_PROMPT_MAX_QUESTION_BYTES) {
return null
}
ids.push(id)
}
return ids
}
export function readQuestionOptionAnswers(
params: unknown
): Map<string, { questionId: string; answer: string }> | null {
const questions = (params as { questions?: unknown } | null)?.questions
const answers = new Map<string, { questionId: string; answer: string }>()
if (!Array.isArray(questions)) {
return answers
}
let optionCount = 0
let optionBytes = 0
for (const entry of questions) {
const question = typeof entry === 'object' && entry !== null ? entry : {}
const questionId = (question as { id?: unknown }).id
const options = (question as { options?: unknown }).options
if (typeof questionId !== 'string' || !Array.isArray(options)) {
continue
}
for (const option of options) {
const record = typeof option === 'object' && option !== null ? option : {}
const label = (record as { label?: unknown }).label
if (
typeof label !== 'string' ||
label.length === 0 ||
(record as { isOther?: unknown }).isOther === true
) {
continue
}
if (++optionCount > CODEX_PROMPT_MAX_OPTIONS) {
return null
}
optionBytes += Buffer.byteLength(questionId, 'utf8') + Buffer.byteLength(label, 'utf8')
if (optionBytes > CODEX_PROMPT_MAX_OPTION_BYTES) {
return null
}
answers.set(encodeCodexJournalQuestionOptionId(questionId, label), {
questionId,
answer: label
})
}
}
return answers
}
@@ -78,7 +78,7 @@ describe('Codex blocking server request dispositions', () => {
)
})
it('cancels a malformed interactive request instead of using method-not-found', () => {
it('refuses a malformed interactive request instead of inventing an answer', () => {
const { registry, connection } = harness()
disposeCodexServerRequest(registry, connection, {
@@ -87,7 +87,12 @@ describe('Codex blocking server request dispositions', () => {
params: {}
})
expect(connection.respond).toHaveBeenCalledWith(4, { decision: 'cancel' })
expect(connection.respond).not.toHaveBeenCalled()
expect(connection.respondWithError).toHaveBeenCalledWith(
4,
-32001,
'Orca could not model item/commandExecution/requestApproval as a durable prompt'
)
})
it('enumerates every server request in the negotiated stable schema', () => {
@@ -51,10 +51,12 @@ export function disposeCodexServerRequest(
switch (request.method) {
case CODEX_COMMAND_APPROVAL_METHOD:
case CODEX_FILE_CHANGE_APPROVAL_METHOD:
connection.respond(request.id, { decision: 'cancel' })
break
case CODEX_USER_INPUT_METHOD:
connection.respond(request.id, { answers: {} })
connection.respondWithError(
request.id,
-32001,
`Orca could not model ${request.method} as a durable prompt`
)
break
case CODEX_MCP_ELICITATION_METHOD:
connection.respond(request.id, { action: 'decline', content: null, _meta: null })
@@ -8,26 +8,50 @@
import type { CodexAppServerConnection } from './codex-app-server-connection'
import { CodexPromptRegistry } from './codex-structured-prompt-replies'
/** Pre-publication buffering is bounded so a provider cannot pin closures. */
export const MAX_CODEX_ACQUISITION_BUFFER_OPERATIONS = 1024
export const MAX_CODEX_ACQUISITION_BUFFER_BYTES = 4 * 1024 * 1024
export class CodexAcquisitionWindow {
readonly prompts = new CodexPromptRegistry()
/** Null until the spawn resolves; the handshake can already emit events. */
connection: CodexAppServerConnection | null = null
private readonly buffered: (() => void)[] = []
private retainedBytes = 0
private open = true
private overflowed = false
get isOverflowed(): boolean {
return this.overflowed
}
/** Returns false once the session is published, which is the caller's cue to
* deliver live rather than buffer. */
buffer(event: () => void): boolean {
buffer(event: () => void, retainedBytes = 256): boolean {
if (!this.open) {
return false
}
const bytes = Number.isFinite(retainedBytes) && retainedBytes > 0 ? Math.ceil(retainedBytes) : 1
if (
this.buffered.length >= MAX_CODEX_ACQUISITION_BUFFER_OPERATIONS ||
this.retainedBytes + bytes > MAX_CODEX_ACQUISITION_BUFFER_BYTES
) {
// Refuse the acquisition rather than dropping an event and continuing.
this.overflowed = true
this.open = false
this.buffered.length = 0
this.retainedBytes = 0
return false
}
this.buffered.push(event)
this.retainedBytes += bytes
return true
}
/** Closes the window and hands back what arrived while it was open, in order. */
drain(): (() => void)[] {
this.open = false
this.retainedBytes = 0
return this.buffered.splice(0)
}
}
@@ -0,0 +1,42 @@
export const MAX_CODEX_ITEM_STREAM_STATES = 256
export const MAX_CODEX_ITEM_STREAM_PENDING_PATCHES = 128
export const MAX_CODEX_ITEM_STREAM_RETAINED_BYTES = 32 * 1024 * 1024
export const MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES = 8 * 1024 * 1024
export const MAX_CODEX_ITEM_STREAM_ITEM_BYTES = 64 * 1024
export function codexStructuredItemKey(threadId: string, itemId: string): string {
const key = `${encodeURIComponent(threadId)}:${encodeURIComponent(itemId)}`
if (Buffer.byteLength(key, 'utf8') <= 1024) {
return key
}
let hash = 2166136261
for (const byte of Buffer.from(key, 'utf8')) {
hash ^= byte
hash = Math.imul(hash, 16777619)
}
return `${key.slice(0, 960)}:${(hash >>> 0).toString(16)}`
}
export function pendingPatchBytes(pending: {
body: unknown
blobs: readonly { payload: string }[]
}): number {
return (
Buffer.byteLength(JSON.stringify(pending.body), 'utf8') +
pending.blobs.reduce((total, blob) => total + Buffer.byteLength(blob.payload, 'utf8'), 0)
)
}
export function boundStreamItem(item: Record<string, unknown>): Record<string, unknown> {
if (Buffer.byteLength(JSON.stringify(item), 'utf8') <= MAX_CODEX_ITEM_STREAM_ITEM_BYTES) {
return item
}
return {
type: item.type,
id: item.id,
...(typeof item.command === 'string' ? { command: item.command.slice(0, 4096) } : {}),
...(typeof item.cwd === 'string' ? { cwd: item.cwd.slice(0, 4096) } : {}),
...(typeof item.status === 'string' ? { status: item.status } : {}),
...(typeof item.exitCode === 'number' ? { exitCode: item.exitCode } : {})
}
}
@@ -0,0 +1,53 @@
import type { AgentJournalItemIdentity } from '../../shared/agent-session-journal-types'
import type { AgentSessionDeltaCoalescerDeps } from '../native-chat/agent-session-wire/agent-session-delta-coalescer'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import type { codexJournalItem, CodexThreadItem } from './codex-structured-item-translation'
export type CodexItemStreamDeps = {
sink: StructuredAgentSessionEventSink
identityFor: (
threadId: string,
params: unknown,
item: CodexThreadItem
) => AgentJournalItemIdentity
coalesceMs?: number
maxRetainedBytes?: number
maxTotalRetainedBytes?: number
schedule?: AgentSessionDeltaCoalescerDeps['schedule']
}
export type CodexItemStreamState = {
identity: AgentJournalItemIdentity
item: CodexThreadItem
}
export type CodexPendingItemPatch = {
identity: AgentJournalItemIdentity
body: NonNullable<ReturnType<typeof codexJournalItem>['body']>
blobs: ReturnType<typeof codexJournalItem>['blobs']
}
export type CodexStructuredItemStreamAdmission =
| { accepted: true }
| { accepted: false; reason: 'backpressure' | 'failed' | 'closed' }
export type CodexStructuredItemStreamHandleResult = {
handled: boolean
admission: CodexStructuredItemStreamAdmission
}
export type CodexStructuredItemStreams = {
track: (threadId: string, item: CodexThreadItem, identity: AgentJournalItemIdentity) => void
handle: (
threadId: string,
method: string,
params: unknown
) => CodexStructuredItemStreamHandleResult
forget: (threadId: string, itemId: string) => void
flush: () => boolean
dispose: () => void
snapshot: (
threadId: string,
itemId: string
) => { text: string; observedBytes: number; truncated: boolean } | null
}
@@ -0,0 +1,42 @@
import { MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES } from './codex-structured-item-stream-bounds'
export const CODEX_ITEM_STREAM_TYPES = {
'item/agentMessage/delta': 'agentMessage',
'item/plan/delta': 'plan',
'item/commandExecution/outputDelta': 'commandExecution',
'item/fileChange/outputDelta': 'fileChange',
'item/reasoning/summaryTextDelta': 'reasoning',
'item/reasoning/textDelta': 'reasoning'
} as const
export const PATCH_UPDATED_METHOD = 'item/fileChange/patchUpdated'
export const REASONING_PART_METHOD = 'item/reasoning/summaryPartAdded'
export const TERMINAL_INTERACTION_METHOD = 'item/commandExecution/terminalInteraction'
export function readCodexItemStreamRecord(value: unknown): Record<string, unknown> {
return typeof value === 'object' && value !== null ? (value as Record<string, unknown>) : {}
}
export function readCodexItemStreamString(
source: Record<string, unknown>,
key: string
): string | null {
const value = source[key]
return typeof value === 'string' && value.length > 0 ? value : null
}
export function codexPatchChangeBytes(changes: readonly unknown[]): number {
let total = 0
for (const change of changes) {
const record = readCodexItemStreamRecord(change)
const path = readCodexItemStreamString(record, 'path')
const diff = readCodexItemStreamString(record, 'diff')
if (path && diff) {
total += Buffer.byteLength(path, 'utf8') + Buffer.byteLength(diff, 'utf8') + 1
if (total > MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES) {
return total
}
}
}
return total
}
+217 -92
View File
@@ -1,96 +1,142 @@
import type { AgentJournalItemIdentity } from '../../shared/agent-session-journal-types'
import {
createAgentSessionDeltaCoalescer,
type AgentSessionDeltaCoalescerDeps
} from '../native-chat/agent-session-wire/agent-session-delta-coalescer'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key'
import { createAgentSessionDeltaCoalescer } from '../native-chat/agent-session-wire/agent-session-delta-coalescer'
import {
codexJournalItem,
codexStreamingJournalItem,
type CodexThreadItem
} from './codex-structured-item-translation'
const CODEX_ITEM_STREAM_TYPES = {
'item/agentMessage/delta': 'agentMessage',
'item/plan/delta': 'plan',
'item/commandExecution/outputDelta': 'commandExecution',
'item/fileChange/outputDelta': 'fileChange',
'item/reasoning/summaryTextDelta': 'reasoning',
'item/reasoning/textDelta': 'reasoning'
} as const
const PATCH_UPDATED_METHOD = 'item/fileChange/patchUpdated'
const REASONING_PART_METHOD = 'item/reasoning/summaryPartAdded'
const TERMINAL_INTERACTION_METHOD = 'item/commandExecution/terminalInteraction'
type CodexItemStreamDeps = {
sink: StructuredAgentSessionEventSink
identityFor: (
threadId: string,
params: unknown,
item: CodexThreadItem
) => AgentJournalItemIdentity
coalesceMs?: number
schedule?: AgentSessionDeltaCoalescerDeps['schedule']
}
type StreamState = { identity: AgentJournalItemIdentity; item: CodexThreadItem }
export type CodexStructuredItemStreams = {
track: (threadId: string, item: CodexThreadItem, identity: AgentJournalItemIdentity) => void
handle: (threadId: string, method: string, params: unknown) => boolean
forget: (threadId: string, itemId: string) => void
flush: () => void
dispose: () => void
}
function readRecord(value: unknown): Record<string, unknown> {
return typeof value === 'object' && value !== null ? (value as Record<string, unknown>) : {}
}
function readString(source: Record<string, unknown>, key: string): string | null {
const value = source[key]
return typeof value === 'string' && value.length > 0 ? value : null
}
export function codexStructuredItemKey(threadId: string, itemId: string): string {
return `${encodeURIComponent(threadId)}:${encodeURIComponent(itemId)}`
}
import {
codexStructuredItemKey,
MAX_CODEX_ITEM_STREAM_PENDING_PATCHES,
MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES,
MAX_CODEX_ITEM_STREAM_RETAINED_BYTES,
MAX_CODEX_ITEM_STREAM_STATES,
boundStreamItem,
pendingPatchBytes
} from './codex-structured-item-stream-bounds'
import {
CODEX_ITEM_STREAM_TYPES,
codexPatchChangeBytes,
PATCH_UPDATED_METHOD,
readCodexItemStreamRecord,
readCodexItemStreamString,
REASONING_PART_METHOD,
TERMINAL_INTERACTION_METHOD
} from './codex-structured-item-stream-events'
import type {
CodexItemStreamDeps,
CodexItemStreamState,
CodexPendingItemPatch,
CodexStructuredItemStreamAdmission,
CodexStructuredItemStreams
} from './codex-structured-item-stream-contracts'
export type {
CodexStructuredItemStreamAdmission,
CodexStructuredItemStreamHandleResult,
CodexStructuredItemStreams
} from './codex-structured-item-stream-contracts'
export { codexStructuredItemKey } from './codex-structured-item-stream-bounds'
export {
MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES,
MAX_CODEX_ITEM_STREAM_RETAINED_BYTES,
MAX_CODEX_ITEM_STREAM_PENDING_PATCHES,
MAX_CODEX_ITEM_STREAM_STATES
} from './codex-structured-item-stream-bounds'
/** Delta-only item ids are provider input; retain only a deterministic recent window. */
export function createCodexStructuredItemStreams(
deps: CodexItemStreamDeps
): CodexStructuredItemStreams {
const states = new Map<string, StreamState>()
const latestText = new Map<string, string>()
const states = new Map<string, CodexItemStreamState>()
const checkpointLengths = new Map<string, number>()
// Patch updates are authoritative item snapshots. Keep the latest rejected
// snapshot until the journal admits it; unlike streamed deltas, there is no
// coalescer timer to retry these events for us.
const pendingPatches = new Map<string, CodexPendingItemPatch>()
let retainedPatchBytes = 0
const append = (state: StreamState, text: string): void => {
const translated = codexStreamingJournalItem(state.item, text)
if (!translated.body) {
return
const forgetState = (key: string): void => {
coalescer.forget(key)
states.delete(key)
checkpointLengths.delete(key)
const pending = pendingPatches.get(key)
if (pending) {
retainedPatchBytes = Math.max(0, retainedPatchBytes - pendingPatchBytes(pending))
pendingPatches.delete(key)
}
deps.sink.appendItem(state.identity, translated.body, translated.blobs)
deps.sink.publish()
}
const persist = (key: string, text: string, force: boolean): void => {
latestText.set(key, text)
const trimStates = (): void => {
while (states.size > MAX_CODEX_ITEM_STREAM_STATES) {
const oldest = states.keys().next().value
if (typeof oldest !== 'string') {
break
}
const pending = coalescer.snapshot(oldest)
if (pending && pending.text.length > 0 && !persist(oldest, pending.text, true)) {
// Keep the state (and its buffered text) until the sink recovers. A
// bounded map is preferable to silently losing streamed output.
break
}
forgetState(oldest)
}
}
const trimPendingPatches = (): void => {
while (pendingPatches.size > MAX_CODEX_ITEM_STREAM_PENDING_PATCHES) {
const oldest = pendingPatches.keys().next().value
if (typeof oldest !== 'string') {
break
}
const pending = pendingPatches.get(oldest)
if (pending) {
retainedPatchBytes = Math.max(0, retainedPatchBytes - pendingPatchBytes(pending))
}
pendingPatches.delete(oldest)
}
}
const append = (state: CodexItemStreamState, text: string): boolean => {
const translated = codexStreamingJournalItem(state.item, text)
if (!translated.body) {
return true
}
const options = { coalescingKey: `checkpoint:${agentJournalItemKey(state.identity)}` }
const admission = deps.sink.tryAppendItem
? deps.sink.tryAppendItem(state.identity, translated.body, translated.blobs, options)
: (deps.sink.appendItem(state.identity, translated.body, translated.blobs, options),
{ accepted: true as const })
if (!admission.accepted) {
return false
}
const published = deps.sink.tryPublish
? deps.sink.tryPublish()
: (deps.sink.publish(), { accepted: true as const })
return published.accepted
}
const persist = (key: string, text: string, force: boolean): boolean => {
const checkpointLength = checkpointLengths.get(key) ?? 0
const nextLength = Math.max(checkpointLength + 32, Math.ceil(checkpointLength * 1.125))
if (!force && checkpointLength > 0 && text.length < nextLength) {
return
return true
}
checkpointLengths.set(key, text.length)
const state = states.get(key)
if (state) {
append(state, text)
if (state && append(state, text)) {
checkpointLengths.set(key, text.length)
return true
}
return false
}
const coalescer = createAgentSessionDeltaCoalescer({
windowMs: deps.coalesceMs,
maxRetainedBytes: deps.maxRetainedBytes,
maxTotalRetainedBytes: deps.maxTotalRetainedBytes,
schedule: deps.schedule,
emit: (key, text) => persist(key, text, false)
emit: (key, text) => {
return persist(key, text, false)
}
})
const ensureState = (
@@ -98,7 +144,7 @@ export function createCodexStructuredItemStreams(
itemId: string,
type: string,
params: unknown
): StreamState => {
): CodexItemStreamState => {
const key = codexStructuredItemKey(threadId, itemId)
const existing = states.get(key)
if (existing) {
@@ -107,70 +153,149 @@ export function createCodexStructuredItemStreams(
const item = { type, id: itemId }
const state = { item, identity: deps.identityFor(threadId, params, item) }
states.set(key, state)
trimStates()
return state
}
const flush = (): void => {
coalescer.flushAll()
for (const [key, text] of latestText) {
if (checkpointLengths.get(key) !== text.length) {
persist(key, text, true)
const flush = (): boolean => {
let flushed = coalescer.flushAll()
for (const key of states.keys()) {
const snapshot = coalescer.snapshot(key)
if (snapshot && checkpointLengths.get(key) !== snapshot.text.length) {
flushed = persist(key, snapshot.text, true) && flushed
}
}
for (const [key, pending] of pendingPatches) {
const admission = deps.sink.tryAppendItem
? deps.sink.tryAppendItem(pending.identity, pending.body, pending.blobs)
: (deps.sink.appendItem(pending.identity, pending.body, pending.blobs),
{ accepted: true as const })
if (!admission.accepted) {
flushed = false
continue
}
const published = deps.sink.tryPublish
? deps.sink.tryPublish()
: (deps.sink.publish(), { accepted: true as const })
if (!published.accepted) {
flushed = false
continue
}
retainedPatchBytes = Math.max(0, retainedPatchBytes - pendingPatchBytes(pending))
pendingPatches.delete(key)
}
return flushed
}
const flushPatch = (key: string): CodexStructuredItemStreamAdmission => {
const pending = pendingPatches.get(key)
if (!pending) {
return { accepted: true }
}
const admission = deps.sink.tryAppendItem
? deps.sink.tryAppendItem(pending.identity, pending.body, pending.blobs)
: (deps.sink.appendItem(pending.identity, pending.body, pending.blobs),
{ accepted: true as const })
if (!admission.accepted) {
return admission
}
const published = deps.sink.tryPublish
? deps.sink.tryPublish()
: (deps.sink.publish(), { accepted: true as const })
if (!published.accepted) {
return published
}
retainedPatchBytes = Math.max(0, retainedPatchBytes - pendingPatchBytes(pending))
pendingPatches.delete(key)
return { accepted: true }
}
return {
track: (threadId, item, identity) => {
states.set(codexStructuredItemKey(threadId, item.id), { item, identity })
const key = codexStructuredItemKey(threadId, item.id)
states.delete(key)
states.set(key, { item: boundStreamItem(item) as CodexThreadItem, identity })
trimStates()
},
handle: (threadId, method, params) => {
const paramsRecord = readRecord(params)
const itemId = readString(paramsRecord, 'itemId')
const paramsRecord = readCodexItemStreamRecord(params)
const itemId = readCodexItemStreamString(paramsRecord, 'itemId')
if (method === PATCH_UPDATED_METHOD) {
if (!itemId || !Array.isArray(paramsRecord.changes)) {
return true
return { handled: true, admission: { accepted: true } }
}
if (
codexPatchChangeBytes(paramsRecord.changes) > MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES
) {
return { handled: true, admission: { accepted: false, reason: 'backpressure' } }
}
const key = codexStructuredItemKey(threadId, itemId)
coalescer.flush(key)
const streamFlushed = coalescer.flush(key)
const state = ensureState(threadId, itemId, 'fileChange', params)
state.item = { ...state.item, changes: paramsRecord.changes }
const translated = codexJournalItem(state.item)
if (translated.body) {
deps.sink.appendItem(state.identity, translated.body, translated.blobs)
deps.sink.publish()
const nextPending: CodexPendingItemPatch = {
identity: state.identity,
body: translated.body,
blobs: translated.blobs
}
const previous = pendingPatches.get(key)
const previousBytes = previous ? pendingPatchBytes(previous) : 0
const nextBytes = pendingPatchBytes(nextPending)
if (
nextBytes > MAX_CODEX_ITEM_STREAM_PENDING_PATCH_BYTES ||
retainedPatchBytes - previousBytes + nextBytes > MAX_CODEX_ITEM_STREAM_RETAINED_BYTES
) {
return { handled: true, admission: { accepted: false, reason: 'backpressure' } }
}
retainedPatchBytes = Math.max(0, retainedPatchBytes - previousBytes) + nextBytes
pendingPatches.set(key, nextPending)
trimPendingPatches()
if (streamFlushed) {
const admission = flushPatch(key)
if (!admission.accepted) {
return { handled: true, admission }
}
}
}
return true
return { handled: true, admission: { accepted: true } }
}
if (method === TERMINAL_INTERACTION_METHOD) {
return true
return { handled: true, admission: { accepted: true } }
}
const type = CODEX_ITEM_STREAM_TYPES[method as keyof typeof CODEX_ITEM_STREAM_TYPES]
if (!type && method !== REASONING_PART_METHOD) {
return false
return { handled: false, admission: { accepted: true } }
}
if (!itemId) {
return true
return { handled: true, admission: { accepted: true } }
}
const state = ensureState(threadId, itemId, type ?? 'reasoning', params)
const delta = method === REASONING_PART_METHOD ? '\n' : paramsRecord.delta
if (typeof delta === 'string') {
coalescer.append(codexStructuredItemKey(threadId, state.item.id), delta)
const accepted = coalescer.append(codexStructuredItemKey(threadId, state.item.id), delta)
if (!accepted) {
return { handled: true, admission: { accepted: false, reason: 'backpressure' } }
}
}
return true
return { handled: true, admission: { accepted: true } }
},
forget: (threadId, itemId) => {
const key = codexStructuredItemKey(threadId, itemId)
coalescer.forget(key)
states.delete(key)
latestText.delete(key)
checkpointLengths.delete(key)
forgetState(key)
},
flush,
dispose: () => {
coalescer.dispose()
states.clear()
latestText.clear()
checkpointLengths.clear()
pendingPatches.clear()
retainedPatchBytes = 0
},
snapshot: (threadId, itemId) => {
const key = codexStructuredItemKey(threadId, itemId)
return coalescer.snapshot(key)
}
}
}
@@ -3,8 +3,11 @@ import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key
import {
codexItemBody,
codexItemIdentity,
codexJournalItem,
codexMessageBlocks,
CodexTurnOrdinals,
MAX_CODEX_TURN_ORDINAL_BYTES,
MAX_CODEX_TURN_ORDINAL_ENTRIES,
isCodexMessageItemType,
readCodexThreadItem,
type CodexThreadItem
@@ -54,6 +57,22 @@ function keysFor(items: CodexThreadItem[]): string[] {
}
describe('codex turn ordinals', () => {
it('bounds forgotten turn tombstones while retaining the recent window', () => {
const ordinals = new CodexTurnOrdinals()
const total = MAX_CODEX_TURN_ORDINAL_ENTRIES + 12
for (let index = 0; index < total; index += 1) {
const turnId = `turn-${index}`
expect(ordinals.ordinalFor('thread-many', turnId, 'item-0')).toBe(0)
ordinals.forgetTurn('thread-many', turnId)
}
expect(ordinals.forgottenTurnCount).toBe(MAX_CODEX_TURN_ORDINAL_ENTRIES)
// The newest completed turn still keeps its counter for a late frame.
expect(ordinals.ordinalFor('thread-many', `turn-${total - 1}`, 'item-late')).toBe(1)
// The oldest turn was deterministically evicted and starts a fresh key.
expect(ordinals.ordinalFor('thread-many', 'turn-0', 'item-late')).toBe(0)
})
it('releases a forgotten turn without ever reusing an ordinal it assigned', () => {
const ordinals = new CodexTurnOrdinals()
expect(ordinals.ordinalFor('thread-1', 'turn-1', 'item-1')).toBe(0)
@@ -68,6 +87,15 @@ describe('codex turn ordinals', () => {
// Other turns are untouched.
expect(ordinals.ordinalFor('thread-1', 'turn-2', 'item-1')).toBe(0)
})
it('bounds aggregate provider identifier bytes retained by one active turn', () => {
const ordinals = new CodexTurnOrdinals()
for (let index = 0; index < 3_000; index += 1) {
ordinals.ordinalFor('thread', 'turn', `${index}:${'x'.repeat(512)}`)
}
expect(ordinals.bytes).toBeLessThanOrEqual(MAX_CODEX_TURN_ORDINAL_BYTES)
})
})
describe('codex item identity', () => {
@@ -167,6 +195,69 @@ describe('codex item bodies', () => {
})
})
it('accepts snake-case command completion output and preserves blob evidence', () => {
const output = 'x'.repeat(1_100_000)
const translated = codexJournalItem({
type: 'commandExecution',
id: 'item-large',
command: 'python big.py',
status: 'completed',
exitCode: 0,
aggregated_output: output
})
const body = translated.body
expect(body).toMatchObject({
kind: 'tool-call',
state: 'completed',
output: {
byteLength: 1_100_000,
truncated: true,
digest: expect.any(String)
}
})
if (body?.kind !== 'tool-call' || !body.output) {
throw new Error('expected bounded command output')
}
expect(body.output.head.length).toBeLessThan(20_000)
expect(translated.blobs).toEqual([
{
digest: body.output.digest,
payload: output
}
])
})
it('continues to accept camel-case command completion output', () => {
expect(
codexItemBody({
type: 'commandExecution',
id: 'item-camel',
command: 'printf ok',
status: 'completed',
aggregatedOutput: 'ok'
})
).toMatchObject({
kind: 'tool-call',
output: { head: 'ok', byteLength: 2, truncated: false }
})
})
it('aggregates assistant content parts before bounding the message body', () => {
const body = codexItemBody({
type: 'agentMessage',
id: 'assistant-parts',
content: Array.from({ length: 200 }, () => ({ type: 'text', text: 'a'.repeat(10_000) }))
})
const text =
body?.kind === 'message' && body.blocks[0]?.type === 'text' ? body.blocks[0].text : ''
expect(body).toMatchObject({ kind: 'message', role: 'assistant' })
expect(body?.kind === 'message' ? body.blocks : []).toHaveLength(1)
expect(text).toContain('output truncated')
expect(Buffer.byteLength(JSON.stringify(body), 'utf8')).toBeLessThan(20 * 1024)
})
it('calls a nonzero exit a failure even though codex calls the status completed', () => {
const body = codexItemBody({
type: 'commandExecution',
@@ -5,9 +5,16 @@ import type {
import type { NativeChatBlock } from '../../shared/native-chat-types'
import {
boundInlineText,
boundToolInput,
DEFAULT_JOURNAL_PAYLOAD_LIMITS
} from '../native-chat/agent-session-journal/journal-payload-bounds'
import { unhandledProviderFrameJournalItem } from '../native-chat/agent-session-wire/unhandled-provider-frame'
import type { CodexTurnOrdinals } from './codex-turn-ordinals'
export {
CodexTurnOrdinals,
MAX_CODEX_TURN_ORDINAL_BYTES,
MAX_CODEX_TURN_ORDINAL_ENTRIES
} from './codex-turn-ordinals'
// Codex thread items → journal item bodies and durable identities.
//
@@ -46,44 +53,6 @@ export function readCodexThreadItem(value: unknown): CodexThreadItem | null {
: null
}
/**
* Ordinals for one thread, assigned on first sight and never reassigned.
*
* Non-message items are given no ordinal at all rather than a number from a
* second counter: a counter that a resumed history cannot reproduce is worse
* than no key, because it would look reconcilable and reconcile wrongly.
*/
export class CodexTurnOrdinals {
private readonly turns = new Map<string, { assigned: Map<string, number>; next: number }>()
ordinalFor(threadId: string, turnId: string, codexItemId: string): number {
const turnKey = `${encodeURIComponent(threadId)}:${encodeURIComponent(turnId)}`
let turn = this.turns.get(turnKey)
if (!turn) {
turn = { assigned: new Map(), next: 0 }
this.turns.set(turnKey, turn)
}
const existing = turn.assigned.get(codexItemId)
if (existing !== undefined) {
return existing
}
const ordinal = turn.next
turn.assigned.set(codexItemId, ordinal)
turn.next += 1
return ordinal
}
/** Releases a finished turn's per-item map while keeping its counter, so a
* straggler frame can never be assigned an ordinal the turn already used —
* a reused slot would upsert another item's journal row. */
forgetTurn(threadId: string, turnId: string): void {
const turn = this.turns.get(`${encodeURIComponent(threadId)}:${encodeURIComponent(turnId)}`)
if (turn) {
turn.assigned = new Map()
}
}
}
function readRecord(value: unknown): Record<string, unknown> {
return typeof value === 'object' && value !== null ? (value as Record<string, unknown>) : {}
}
@@ -119,6 +88,16 @@ function readString(source: Record<string, unknown>, key: string): string | null
return typeof value === 'string' && value.length > 0 ? value : null
}
function readFirstString(source: Record<string, unknown>, keys: readonly string[]): string | null {
for (const key of keys) {
const value = readString(source, key)
if (value !== null) {
return value
}
}
return null
}
function readTextContent(source: Record<string, unknown>, key: string): string | null {
const direct = readString(source, key)
if (direct) {
@@ -143,9 +122,12 @@ function readTextContent(source: Record<string, unknown>, key: string): string |
/** `userMessage` carries structured content parts; `agentMessage` a flat text. */
export function codexMessageBlocks(item: CodexThreadItem): NativeChatBlock[] {
const text = readString(item, 'text')
const text =
item.type === 'agentMessage'
? (readString(item, 'text') ?? readTextContent(item, 'content'))
: readString(item, 'text')
if (text !== null) {
return [{ type: 'text', text }]
return [{ type: 'text', text: boundInlineText(text, DEFAULT_JOURNAL_PAYLOAD_LIMITS).text }]
}
const content = item.content
if (!Array.isArray(content)) {
@@ -158,7 +140,10 @@ export function codexMessageBlocks(item: CodexThreadItem): NativeChatBlock[] {
}
const partText = readString(part as Record<string, unknown>, 'text')
if (partText !== null) {
blocks.push({ type: 'text', text: partText })
blocks.push({
type: 'text',
text: boundInlineText(partText, DEFAULT_JOURNAL_PAYLOAD_LIMITS).text
})
continue
}
const record = part as Record<string, unknown>
@@ -192,13 +177,16 @@ export type CodexJournalItem = {
}
function commandItem(item: CodexThreadItem): CodexJournalItem {
const output = readString(item, 'aggregatedOutput')
const output = readFirstString(item, ['aggregatedOutput', 'aggregated_output'])
const bounded = output === null ? null : boundInlineText(output, DEFAULT_JOURNAL_PAYLOAD_LIMITS)
return {
body: {
kind: 'tool-call',
name: 'shell',
input: { command: item.command ?? null, cwd: item.cwd ?? null },
input: boundToolInput(
{ command: item.command ?? null, cwd: item.cwd ?? null },
DEFAULT_JOURNAL_PAYLOAD_LIMITS
),
state: commandState(item),
...(bounded === null ? {} : { output: bounded.bounded })
},
@@ -224,7 +212,7 @@ function fileChangeItem(item: CodexThreadItem): CodexJournalItem {
body: {
kind: 'tool-call',
name: 'apply_patch',
input: { changes: item.changes ?? null },
input: boundToolInput({ changes: item.changes ?? null }, DEFAULT_JOURNAL_PAYLOAD_LIMITS),
state: commandState(item)
},
blobs: [],
@@ -294,7 +282,11 @@ export function codexItemBody(item: CodexThreadItem): AgentJournalItemBody | nul
/** Snapshot body for text still streaming, before its item completes. */
export function codexStreamingMessageBody(text: string): AgentJournalItemBody {
return { kind: 'message', role: 'assistant', blocks: [{ type: 'text', text }] }
return {
kind: 'message',
role: 'assistant',
blocks: [{ type: 'text', text: boundInlineText(text, DEFAULT_JOURNAL_PAYLOAD_LIMITS).text }]
}
}
/** Snapshot body for any item-level stream, keyed onto its parent item. */
@@ -0,0 +1,33 @@
import type { AgentSessionDeltaCoalescerDeps } from '../native-chat/agent-session-wire/agent-session-delta-coalescer'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import type { CodexStructuredSessionEvent } from './codex-structured-session-adapter'
export type CodexJournalTranslatorDeps = {
sink: StructuredAgentSessionEventSink
bindPromptItemId?: (journalItemId: string, threadId: string, promptKey: string) => void
primaryThreadId?: () => string | null
coalesceMs?: number
maxRetainedBytes?: number
schedule?: AgentSessionDeltaCoalescerDeps['schedule']
}
export type CodexJournalTranslator = {
handle: (event: CodexStructuredSessionEvent) => CodexJournalTranslationAdmission
restoreThread: (
threadId: string,
thread: Record<string, unknown>
) => CodexJournalTranslationAdmission
resolvePrompt: (journalItemId: string) => void
flush: () => void
dispose: () => void
}
export type CodexJournalTranslationAdmission =
| { accepted: true }
| { accepted: false; reason: 'backpressure' | 'failed' | 'closed' | 'untranslated' }
export type CodexItemTranslation =
| { handled: false }
| { handled: true; admission: CodexJournalTranslationAdmission }
export const CODEX_JOURNAL_ADMITTED = { accepted: true } as const
@@ -0,0 +1,229 @@
import { unhandledProviderFrameJournalItem } from '../native-chat/agent-session-wire/unhandled-provider-frame'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import type {
CodexJournalTranslationAdmission,
CodexJournalTranslatorDeps
} from './codex-structured-journal-contracts'
import { CODEX_JOURNAL_ADMITTED } from './codex-structured-journal-contracts'
import {
MAX_CODEX_GENERIC_BOOKKEEPING_BYTES,
MAX_CODEX_GENERIC_BOOKKEEPING_ENTRIES,
MAX_CODEX_GENERIC_ROWS_PER_TURN,
MAX_CODEX_GENERIC_TURN_BUCKETS
} from './codex-structured-journal-limits'
import { readCodexTurnId } from './codex-structured-thread-facts'
const OVERFLOW_BUCKET = '__codex-generic-overflow__'
type SuppressedSummary = { count: number; publishedCount: number }
function boundedTurnBucket(threadId: string, turnId: string): string {
const encoded = `${encodeURIComponent(threadId)}:${encodeURIComponent(turnId)}`
if (Buffer.byteLength(encoded, 'utf8') <= 512) {
return encoded
}
let hash = 2166136261
for (const byte of Buffer.from(encoded, 'utf8')) {
hash ^= byte
hash = Math.imul(hash, 16777619)
}
return `${encoded.slice(0, 160)}:${(hash >>> 0).toString(16)}`
}
function defaultSchedule(run: () => void, ms: number): () => void {
const timer = setTimeout(run, ms)
timer.unref?.()
return () => clearTimeout(timer)
}
function publish(sink: StructuredAgentSessionEventSink): CodexJournalTranslationAdmission {
return sink.tryPublish ? sink.tryPublish() : (sink.publish(), CODEX_JOURNAL_ADMITTED)
}
export class CodexJournalGenericFrames {
private readonly genericRowsByTurn = new Map<string, number>()
private readonly suppressedRowsByTurn = new Map<string, SuppressedSummary>()
private readonly bucketOrder = new Map<string, number>()
private readonly schedule: NonNullable<CodexJournalTranslatorDeps['schedule']>
private readonly suppressionCoalesceMs: number
private nextBucketOrder = 0
private bookkeepingBytes = 0
private fallbackSequence = 0
private cancelSuppressionFlush: (() => void) | null = null
constructor(
private readonly deps: Pick<CodexJournalTranslatorDeps, 'sink' | 'schedule' | 'coalesceMs'>,
private readonly activeTurn: (threadId: string) => string | null
) {
this.schedule = deps.schedule ?? defaultSchedule
this.suppressionCoalesceMs = deps.coalesceMs ?? 60
}
appendUnhandled(
kind: string,
payload: unknown,
threadId = 'session'
): CodexJournalTranslationAdmission {
const translated = unhandledProviderFrameJournalItem('codex', kind, payload)
if (!translated) {
return { accepted: false, reason: 'untranslated' }
}
const turnId = readCodexTurnId(payload) ?? this.activeTurn(threadId) ?? 'outside-turn'
const bucket = this.bucketFor(threadId, turnId)
const rowCount = this.genericRowsByTurn.get(bucket) ?? 0
if (rowCount >= MAX_CODEX_GENERIC_ROWS_PER_TURN) {
this.addSuppressed(bucket, 1)
this.recordBucket(bucket)
this.scheduleSuppressedRows()
return CODEX_JOURNAL_ADMITTED
}
if (translated.classification === 'error-surface') {
const suppressionAdmission = this.flush()
if (!suppressionAdmission.accepted) {
return suppressionAdmission
}
}
this.fallbackSequence += 1
const admission = this.deps.sink.tryAppendItem
? this.deps.sink.tryAppendItem(
{ provider: 'orca', clientMessageId: `provider-frame:codex:${this.fallbackSequence}` },
translated.body,
translated.blobs
)
: (this.deps.sink.appendItem(
{ provider: 'orca', clientMessageId: `provider-frame:codex:${this.fallbackSequence}` },
translated.body,
translated.blobs
),
CODEX_JOURNAL_ADMITTED)
if (!admission.accepted) {
this.fallbackSequence -= 1
return admission
}
this.genericRowsByTurn.set(bucket, rowCount + 1)
this.recordBucket(bucket)
return publish(this.deps.sink)
}
suppress(threadId: string, turnId: string, count = 1): void {
const bucket = this.bucketFor(threadId, turnId)
this.addSuppressed(bucket, count)
this.recordBucket(bucket)
}
flush = (): CodexJournalTranslationAdmission => {
this.cancelSuppressionFlush?.()
this.cancelSuppressionFlush = null
let wrote = false
const ready: SuppressedSummary[] = []
let blocked: CodexJournalTranslationAdmission | null = null
for (const [bucket, summary] of this.suppressedRowsByTurn) {
if (summary.count === summary.publishedCount) {
continue
}
const text =
bucket === OVERFLOW_BUCKET
? `${summary.count} more provider notification${summary.count === 1 ? '' : 's'} not shown across evicted turns`
: `${summary.count} more provider notification${summary.count === 1 ? '' : 's'} not shown for this turn`
const admission = this.deps.sink.tryAppendItem
? this.deps.sink.tryAppendItem(
{ provider: 'orca', clientMessageId: `provider-frame-suppressed:codex:${bucket}` },
{
kind: 'status',
text
},
[],
{ coalescingKey: `provider-frame-suppressed:codex:${bucket}` }
)
: (this.deps.sink.appendItem(
{ provider: 'orca', clientMessageId: `provider-frame-suppressed:codex:${bucket}` },
{ kind: 'status', text },
[],
{ coalescingKey: `provider-frame-suppressed:codex:${bucket}` }
),
CODEX_JOURNAL_ADMITTED)
if (!admission.accepted) {
blocked ??= admission
continue
}
ready.push(summary)
wrote = true
}
if (wrote) {
const admission = publish(this.deps.sink)
if (!admission.accepted) {
blocked ??= admission
} else {
for (const summary of ready) {
summary.publishedCount = summary.count
}
}
}
if (blocked) {
this.scheduleSuppressedRows()
return blocked
}
return CODEX_JOURNAL_ADMITTED
}
dispose(): void {
this.cancelSuppressionFlush?.()
this.genericRowsByTurn.clear()
this.suppressedRowsByTurn.clear()
this.bucketOrder.clear()
this.bookkeepingBytes = 0
}
private scheduleSuppressedRows(): void {
this.cancelSuppressionFlush ??= this.schedule(() => {
this.cancelSuppressionFlush = null
this.flush()
}, this.suppressionCoalesceMs)
}
private bucketFor(threadId: string, turnId: string): string {
const requested = boundedTurnBucket(threadId, turnId)
return Buffer.byteLength(requested, 'utf8') > MAX_CODEX_GENERIC_BOOKKEEPING_BYTES
? OVERFLOW_BUCKET
: requested
}
private addSuppressed(bucket: string, count: number): void {
const summary = this.suppressedRowsByTurn.get(bucket) ?? { count: 0, publishedCount: 0 }
summary.count += count
this.suppressedRowsByTurn.set(bucket, summary)
}
private recordBucket(bucket: string): void {
if (!this.bucketOrder.has(bucket)) {
this.bucketOrder.set(bucket, this.nextBucketOrder++)
this.bookkeepingBytes += Buffer.byteLength(bucket, 'utf8')
}
while (
(this.bucketOrder.size > MAX_CODEX_GENERIC_TURN_BUCKETS ||
this.genericRowsByTurn.size + this.suppressedRowsByTurn.size >
MAX_CODEX_GENERIC_BOOKKEEPING_ENTRIES ||
this.bookkeepingBytes > MAX_CODEX_GENERIC_BOOKKEEPING_BYTES) &&
this.bucketOrder.size > 1
) {
const oldest = [...this.bucketOrder.entries()]
.filter(([id]) => id !== OVERFLOW_BUCKET && id !== bucket)
.sort((a, b) => a[1] - b[1])[0]?.[0]
if (!oldest) {
break
}
const suppressed = this.suppressedRowsByTurn.get(oldest)
this.removeBucket(oldest)
if (suppressed && suppressed.count > suppressed.publishedCount) {
this.recordBucket(OVERFLOW_BUCKET)
this.addSuppressed(OVERFLOW_BUCKET, suppressed.count - suppressed.publishedCount)
}
}
}
private removeBucket(bucket: string): void {
this.genericRowsByTurn.delete(bucket)
this.suppressedRowsByTurn.delete(bucket)
this.bookkeepingBytes = Math.max(0, this.bookkeepingBytes - Buffer.byteLength(bucket, 'utf8'))
this.bucketOrder.delete(bucket)
}
}
@@ -0,0 +1,243 @@
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../shared/agent-session-journal-types'
import { requiresTerminalSettlement } from '../native-chat/agent-session-journal/journal-lifecycle-capacity'
import {
codexItemIdentity,
codexJournalItem,
CodexTurnOrdinals,
readCodexThreadItem,
type CodexThreadItem
} from './codex-structured-item-translation'
import { createCodexStructuredItemStreams } from './codex-structured-item-streams'
import { codexStructuredItemKey } from './codex-structured-item-stream-bounds'
import type {
CodexItemTranslation,
CodexJournalTranslationAdmission,
CodexJournalTranslatorDeps
} from './codex-structured-journal-contracts'
import { CODEX_JOURNAL_ADMITTED } from './codex-structured-journal-contracts'
import {
MAX_CODEX_ACTIVE_ITEMS,
MAX_CODEX_DETAIL_BYTES,
MAX_CODEX_DETAIL_ENTRIES,
MAX_CODEX_IDENTITY_ENTRIES
} from './codex-structured-journal-limits'
import { appendCodexLifecycleItem, publishCodexLifecycle } from './codex-structured-journal-sink'
import type { CodexActiveJournalItem } from './codex-structured-journal-settlement'
import { readCodexJournalString } from './codex-structured-journal-translation-values'
import { readCodexTurnId } from './codex-structured-thread-facts'
export class CodexJournalItems {
readonly ordinals = new CodexTurnOrdinals()
readonly activeItems = new Map<string, CodexActiveJournalItem>()
readonly streams
private readonly identities = new Map<string, AgentJournalItemIdentity>()
private readonly details = new Map<string, string>()
constructor(
private readonly deps: Pick<
CodexJournalTranslatorDeps,
'sink' | 'coalesceMs' | 'maxRetainedBytes' | 'schedule'
>,
private readonly activeTurn: (threadId: string) => string | null,
private readonly suppress: (threadId: string, turnId: string) => void
) {
this.streams = createCodexStructuredItemStreams({
sink: deps.sink,
coalesceMs: deps.coalesceMs,
maxRetainedBytes: deps.maxRetainedBytes,
schedule: deps.schedule,
identityFor: (threadId, params, item) => {
const turnId = readCodexTurnId(params) ?? this.activeTurn(threadId)
return this.identityFor(threadId, turnId, item)
}
})
}
detailFor(threadId: string, itemId: string): string | null {
return this.details.get(codexStructuredItemKey(threadId, itemId)) ?? null
}
handle(event: { threadId: string; method: string; params: unknown }): CodexItemTranslation {
const params =
typeof event.params === 'object' && event.params !== null
? (event.params as Record<string, unknown>)
: {}
const item = readCodexThreadItem(params.item)
if (!item) {
return { handled: false }
}
const turnId = readCodexTurnId(event.params) ?? this.activeTurn(event.threadId)
const identity = this.identityFor(event.threadId, turnId, item)
const translated = codexJournalItem(item)
const command = readCodexJournalString(item, 'command')
if (command) {
const boundedCommand = Buffer.from(command, 'utf8')
.subarray(0, MAX_CODEX_DETAIL_BYTES)
.toString('utf8')
this.details.set(codexStructuredItemKey(event.threadId, item.id), boundedCommand)
}
const itemKey = codexStructuredItemKey(event.threadId, item.id)
if (!translated.body) {
if (event.method === 'item/completed') {
this.streams.forget(event.threadId, item.id)
this.activeItems.delete(itemKey)
} else {
this.track(event.threadId, turnId, item, identity)
const admission = this.trimActiveState()
if (!admission.accepted) {
return { handled: true, admission }
}
}
return { handled: true, admission: CODEX_JOURNAL_ADMITTED }
}
const admission = this.appendTranslated(event.method, identity, translated)
if (!admission.accepted) {
return { handled: true, admission }
}
if (event.method === 'item/completed') {
this.streams.forget(event.threadId, item.id)
this.activeItems.delete(itemKey)
} else {
this.track(event.threadId, turnId, item, identity)
const trimAdmission = this.trimActiveState()
if (!trimAdmission.accepted) {
return { handled: true, admission: trimAdmission }
}
}
return { handled: true, admission: CODEX_JOURNAL_ADMITTED }
}
dispose(): void {
this.streams.dispose()
this.identities.clear()
this.details.clear()
this.activeItems.clear()
}
private appendTranslated(
method: string,
identity: AgentJournalItemIdentity,
translated: ReturnType<typeof codexJournalItem>
): CodexJournalTranslationAdmission {
if (!translated.body) {
return CODEX_JOURNAL_ADMITTED
}
if (method === 'item/completed') {
const admission = appendCodexLifecycleItem(
this.deps.sink,
identity,
translated.body,
translated.blobs
)
return admission.accepted ? publishCodexLifecycle(this.deps.sink) : admission
}
const options = requiresTerminalSettlement(translated.body) ? { lifecycle: true } : {}
const admission = this.deps.sink.tryAppendItem
? this.deps.sink.tryAppendItem(identity, translated.body, translated.blobs, options)
: (this.deps.sink.appendItem(identity, translated.body, translated.blobs),
CODEX_JOURNAL_ADMITTED)
if (!admission.accepted) {
return admission
}
return this.deps.sink.tryPublish
? this.deps.sink.tryPublish(options)
: (this.deps.sink.publish(options), CODEX_JOURNAL_ADMITTED)
}
private track(
threadId: string,
turnId: string | null,
item: CodexThreadItem,
identity: AgentJournalItemIdentity
): void {
this.streams.track(threadId, item, identity)
this.activeItems.set(codexStructuredItemKey(threadId, item.id), {
threadId,
turnId,
identity,
item
})
}
private identityFor(
threadId: string,
turnId: string | null,
item: Parameters<typeof codexItemIdentity>[0]['item']
): AgentJournalItemIdentity {
const key = codexStructuredItemKey(threadId, item.id)
const existing = this.identities.get(key)
if (existing) {
return existing
}
const identity = codexItemIdentity({ threadId, turnId, item, ordinals: this.ordinals })
this.identities.set(key, identity)
while (this.identities.size > MAX_CODEX_IDENTITY_ENTRIES) {
const oldest = this.identities.keys().next().value
if (typeof oldest === 'string') {
this.identities.delete(oldest)
}
}
while (this.details.size > MAX_CODEX_DETAIL_ENTRIES) {
const oldest = this.details.keys().next().value
if (typeof oldest === 'string') {
this.details.delete(oldest)
}
}
return identity
}
private trimActiveState(): CodexJournalTranslationAdmission {
while (this.activeItems.size > MAX_CODEX_ACTIVE_ITEMS) {
const oldest = this.activeItems.keys().next().value
if (typeof oldest !== 'string') {
break
}
const evicted = this.activeItems.get(oldest)
if (evicted) {
const translated = codexJournalItem(evicted.item).body
if (translated) {
const admission = appendCodexLifecycleItem(
this.deps.sink,
evicted.identity,
evictedActiveBody(translated)
)
if (!admission.accepted) {
return admission
}
const published = publishCodexLifecycle(this.deps.sink)
if (!published.accepted) {
return published
}
}
this.streams.forget(evicted.threadId, evicted.item.id)
this.suppress(evicted.threadId, evicted.turnId ?? 'outside-turn')
}
this.activeItems.delete(oldest)
}
return CODEX_JOURNAL_ADMITTED
}
}
function evictedActiveBody(body: AgentJournalItemBody): AgentJournalItemBody {
if (body.kind === 'tool-call' && body.state === 'running') {
return { ...body, state: 'failed' }
}
if (
(body.kind === 'approval' || body.kind === 'question') &&
body.resolution.state === 'pending'
) {
return {
...body,
resolution: {
state: 'cancelled',
selectedOptionId: null,
resolvedBy: null,
resolvedAt: null
}
}
}
return body
}
@@ -0,0 +1,9 @@
export const MAX_CODEX_GENERIC_ROWS_PER_TURN = 8
export const MAX_CODEX_GENERIC_TURN_BUCKETS = 64
export const MAX_CODEX_GENERIC_BOOKKEEPING_ENTRIES = 128
export const MAX_CODEX_GENERIC_BOOKKEEPING_BYTES = 32 * 1024
export const MAX_CODEX_ACTIVE_ITEMS = 256
export const MAX_CODEX_PENDING_PROMPTS = 128
export const MAX_CODEX_IDENTITY_ENTRIES = 512
export const MAX_CODEX_DETAIL_ENTRIES = 512
export const MAX_CODEX_DETAIL_BYTES = 64 * 1024
@@ -0,0 +1,127 @@
import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key'
import { cancelledJournalPromptBody } from '../native-chat/agent-session-journal/journal-prompt-body-bounds'
import {
codexApprovalItem,
codexPromptIdentity,
codexQuestionItems
} from './codex-structured-prompt-items'
import { CODEX_USER_INPUT_METHOD } from './codex-structured-prompt-replies'
import type {
CodexJournalTranslationAdmission,
CodexJournalTranslatorDeps
} from './codex-structured-journal-contracts'
import { CODEX_JOURNAL_ADMITTED } from './codex-structured-journal-contracts'
import { MAX_CODEX_PENDING_PROMPTS } from './codex-structured-journal-limits'
import {
admitCodexLifecycleItems,
appendCodexLifecycleItem,
publishCodexLifecycle
} from './codex-structured-journal-sink'
import type { CodexPendingJournalPrompt } from './codex-structured-journal-settlement'
export class CodexJournalPrompts {
readonly pending = new Map<string, CodexPendingJournalPrompt>()
constructor(
private readonly deps: Pick<CodexJournalTranslatorDeps, 'sink' | 'bindPromptItemId'>,
private readonly detailFor: (threadId: string, itemId: string) => string | null
) {}
handle(event: {
threadId: string
method: string
params: unknown
codexItemId: string
promptKey: string
}): CodexJournalTranslationAdmission {
if (event.method === CODEX_USER_INPUT_METHOD) {
const questions = codexQuestionItems({
threadId: event.threadId,
promptKey: event.promptKey,
params: event.params
})
const promptItems = questions.map(({ identity, body }) => ({ identity, body }))
const admission = this.admit(event, promptItems)
if (!admission.accepted) {
return admission
}
for (const question of promptItems) {
const itemId = agentJournalItemKey(question.identity)
this.pending.set(itemId, { identity: question.identity, body: question.body })
const trimAdmission = this.trim()
if (!trimAdmission.accepted) {
return trimAdmission
}
this.deps.bindPromptItemId?.(itemId, event.threadId, event.promptKey)
}
return CODEX_JOURNAL_ADMITTED
}
const identity = codexPromptIdentity({
threadId: event.threadId,
promptKey: event.promptKey
})
const body = codexApprovalItem({
method: event.method,
params: event.params,
detail: this.detailFor(event.threadId, event.codexItemId)
})
const admission = this.admit(event, [{ identity, body }])
if (!admission.accepted) {
return admission
}
const itemId = agentJournalItemKey(identity)
this.pending.set(itemId, { identity, body })
const trimAdmission = this.trim()
if (!trimAdmission.accepted) {
return trimAdmission
}
this.deps.bindPromptItemId?.(itemId, event.threadId, event.promptKey)
return CODEX_JOURNAL_ADMITTED
}
resolve(journalItemId: string): void {
this.pending.delete(journalItemId)
}
dispose(): void {
this.pending.clear()
}
private admit(
event: { method: string; threadId: string; promptKey: string },
items: readonly CodexPendingJournalPrompt[]
): CodexJournalTranslationAdmission {
return admitCodexLifecycleItems(
this.deps.sink,
`prompt:${encodeURIComponent(event.method)}:${encodeURIComponent(
event.threadId
)}:${encodeURIComponent(event.promptKey)}`,
items
)
}
private trim(): CodexJournalTranslationAdmission {
while (this.pending.size > MAX_CODEX_PENDING_PROMPTS) {
const oldest = this.pending.keys().next().value
if (typeof oldest !== 'string') {
break
}
const evicted = this.pending.get(oldest)
if (evicted) {
const cancelled = cancelledJournalPromptBody(evicted.body)
if (cancelled) {
const admission = appendCodexLifecycleItem(this.deps.sink, evicted.identity, cancelled)
if (!admission.accepted) {
return admission
}
const published = publishCodexLifecycle(this.deps.sink)
if (!published.accepted) {
return published
}
}
}
this.pending.delete(oldest)
}
return CODEX_JOURNAL_ADMITTED
}
}
@@ -0,0 +1,299 @@
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../shared/agent-session-journal-types'
import { partitionJournalLifecycleMutations } from '../native-chat/agent-session-journal/journal-lifecycle-batch-partition'
import type { JournalLifecycleMutationInput } from '../native-chat/agent-session-journal/journal-row-builders'
import type {
StructuredAgentSessionEventSink,
StructuredAgentSessionSinkAdmission
} from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import {
boundJournalStatusText,
cancelledJournalPromptBody
} from '../native-chat/agent-session-journal/journal-prompt-body-bounds'
import {
codexJournalItem,
codexStreamingJournalItem,
type CodexThreadItem,
type CodexTurnOrdinals
} from './codex-structured-item-translation'
import type { CodexStructuredItemStreams } from './codex-structured-item-streams'
import type { CodexStructuredSessionEvent } from './codex-structured-session-adapter'
export type CodexActiveJournalItem = {
threadId: string
turnId: string | null
identity: AgentJournalItemIdentity
item: CodexThreadItem
}
export type CodexPendingJournalPrompt = {
identity: AgentJournalItemIdentity
body: AgentJournalItemBody
}
const ADMITTED: StructuredAgentSessionSinkAdmission = { accepted: true }
export function settleCodexJournalSession(input: {
event: Extract<CodexStructuredSessionEvent, { type: 'ended' }>
sink: StructuredAgentSessionEventSink
streams: CodexStructuredItemStreams
activeItems: ReadonlyMap<string, CodexActiveJournalItem>
pendingPrompts: ReadonlyMap<string, CodexPendingJournalPrompt>
currentTurnIds: ReadonlyMap<string, ReadonlySet<string>>
primaryThreadId: string | null
ordinals: CodexTurnOrdinals
}): StructuredAgentSessionSinkAdmission {
const mutations: JournalLifecycleMutationInput[] = []
const turnOrdinalsToForget: { threadId: string; turnId: string }[] = []
for (const active of input.activeItems.values()) {
const streamed = input.streams.snapshot(active.threadId, active.item.id)
const translated = streamed
? codexStreamingJournalItem(active.item, streamed.text)
: codexJournalItem(active.item)
const body = interruptedBody(translated.body)
if (body) {
mutations.push({ kind: 'item', identity: active.identity, body })
}
}
for (const prompt of input.pendingPrompts.values()) {
const body = cancelledJournalPromptBody(prompt.body)
if (body) {
mutations.push({
kind: 'item',
identity: prompt.identity,
body
})
}
}
if (!('cause' in input.event) || input.event.cause === 'unexpected-exit') {
mutations.push({
kind: 'item',
identity: { provider: 'orca', clientMessageId: exitSettlementId(input.event) },
body: {
kind: 'status',
text: boundJournalStatusText(`Provider exited: ${input.event.reason}`)
}
})
}
for (const [threadId, turnIds] of input.currentTurnIds) {
if (input.primaryThreadId !== threadId) {
continue
}
for (const turnId of turnIds) {
mutations.push({
kind: 'tombstone',
identity: {
provider: 'legacy',
agent: 'codex',
sessionId: input.event.sessionId,
recordId: `turn-lifecycle:${turnId}`
}
})
turnOrdinalsToForget.push({ threadId, turnId })
}
}
const admission = appendLifecycleMutations(input.sink, exitSettlementId(input.event), mutations)
if (!admission.accepted) {
return admission
}
for (const { threadId, turnId } of turnOrdinalsToForget) {
input.ordinals.forgetTurn(threadId, turnId)
}
return ADMITTED
}
export function settleCodexJournalTurn(input: {
sessionId: string
threadId: string
turnId: string
sink: StructuredAgentSessionEventSink
streams: CodexStructuredItemStreams
activeItems: Map<string, CodexActiveJournalItem>
}): StructuredAgentSessionSinkAdmission {
const mutations: JournalLifecycleMutationInput[] = []
const activeItemsToForget: { key: string; threadId: string; itemId: string }[] = []
for (const [key, active] of input.activeItems) {
if (active.threadId !== input.threadId || active.turnId !== input.turnId) {
continue
}
const streamed = input.streams.snapshot(active.threadId, active.item.id)
const translated = streamed
? codexStreamingJournalItem(active.item, streamed.text)
: codexJournalItem(active.item)
const body = interruptedBody(translated.body)
if (body) {
mutations.push({ kind: 'item', identity: active.identity, body })
}
activeItemsToForget.push({ key, threadId: active.threadId, itemId: active.item.id })
}
mutations.push({
kind: 'tombstone',
identity: {
provider: 'legacy',
agent: 'codex',
sessionId: input.sessionId,
recordId: `turn-lifecycle:${input.turnId}`
}
})
const admission = appendLifecycleMutations(
input.sink,
`turn-completed:${input.sessionId}:${input.threadId}:${input.turnId}`,
mutations
)
if (!admission.accepted) {
return admission
}
for (const active of activeItemsToForget) {
input.streams.forget(active.threadId, active.itemId)
input.activeItems.delete(active.key)
}
return ADMITTED
}
/** Settle streamed items whose terminal notification was rejected as oversized. */
export function settleCodexOversizedNotification(input: {
sessionId: string
threadId: string
method: string
sink: StructuredAgentSessionEventSink
streams: CodexStructuredItemStreams
activeItems: Map<string, CodexActiveJournalItem>
}): StructuredAgentSessionSinkAdmission {
const itemType = oversizedStreamItemType(input.method)
if (!itemType) {
return ADMITTED
}
const mutations: JournalLifecycleMutationInput[] = []
const activeItemsToForget: { key: string; threadId: string; itemId: string }[] = []
for (const [key, active] of input.activeItems) {
if (active.threadId !== input.threadId || active.item.type !== itemType) {
continue
}
const streamed = input.streams.snapshot(active.threadId, active.item.id)
const translated = streamed
? codexStreamingJournalItem(active.item, streamed.text)
: codexJournalItem(active.item)
const body = interruptedBody(translated.body)
if (body) {
mutations.push({ kind: 'item', identity: active.identity, body })
}
activeItemsToForget.push({ key, threadId: active.threadId, itemId: active.item.id })
}
if (mutations.length === 0) {
return ADMITTED
}
const admission = appendLifecycleMutations(
input.sink,
`oversized-notification:${input.sessionId}:${input.threadId}:${input.method}`,
mutations
)
if (!admission.accepted) {
return admission
}
for (const active of activeItemsToForget) {
input.streams.forget(active.threadId, active.itemId)
input.activeItems.delete(active.key)
}
return ADMITTED
}
function oversizedStreamItemType(method: string): CodexThreadItem['type'] | null {
if (method === 'item/agentMessage/delta') {
return 'agentMessage'
}
if (method === 'item/plan/delta') {
return 'plan'
}
if (
method === 'command/exec/outputDelta' ||
method === 'process/outputDelta' ||
method === 'item/commandExecution/outputDelta' ||
method === 'item/commandExecution/terminalInteraction'
) {
return 'commandExecution'
}
if (method === 'item/fileChange/outputDelta' || method === 'item/fileChange/patchUpdated') {
return 'fileChange'
}
if (
method === 'item/reasoning/summaryTextDelta' ||
method === 'item/reasoning/summaryPartAdded' ||
method === 'item/reasoning/textDelta'
) {
return 'reasoning'
}
return null
}
function appendLifecycleMutations(
sink: StructuredAgentSessionEventSink,
settlementId: string,
mutations: readonly JournalLifecycleMutationInput[]
): StructuredAgentSessionSinkAdmission {
const chunks = partitionJournalLifecycleMutations(settlementId, mutations)
for (const { settlementId: id, mutations: chunk } of chunks) {
let admission: StructuredAgentSessionSinkAdmission = ADMITTED
if (sink.tryAppendLifecycleBatch) {
admission = sink.tryAppendLifecycleBatch(id, chunk, { lifecycle: true })
} else if (sink.appendLifecycleBatch) {
admission = sink.appendLifecycleBatch(id, chunk, { lifecycle: true }) ?? ADMITTED
} else {
for (const mutation of chunk) {
if (mutation.kind === 'item') {
if (sink.tryAppendItem) {
admission = sink.tryAppendItem(mutation.identity, mutation.body, [], {
lifecycle: true
})
if (!admission.accepted) {
return admission
}
} else {
sink.appendItem(mutation.identity, mutation.body, [], { lifecycle: true })
}
} else {
if (sink.tryAppendTombstone) {
admission = sink.tryAppendTombstone(mutation.identity, { lifecycle: true })
if (!admission.accepted) {
return admission
}
} else {
sink.appendTombstone(mutation.identity, { lifecycle: true })
}
}
}
}
if (!admission.accepted) {
return admission
}
const publishAdmission = sink.tryPublish
? sink.tryPublish({ lifecycle: true })
: (sink.publish({ lifecycle: true }), ADMITTED)
if (!publishAdmission.accepted) {
return publishAdmission
}
}
return ADMITTED
}
function interruptedBody(body: AgentJournalItemBody | null): AgentJournalItemBody | null {
if (!body) {
return null
}
if (body.kind === 'tool-call') {
return { ...body, state: 'failed' }
}
if (body.kind === 'message') {
return body
}
return body.kind === 'diff'
? { kind: 'status', text: 'File changes were interrupted before completion.' }
: body
}
function exitSettlementId(event: Extract<CodexStructuredSessionEvent, { type: 'ended' }>): string {
const fence = 'fence' in event ? event.fence : 0
const generation = 'acquisitionGeneration' in event ? event.acquisitionGeneration : 'legacy'
return `provider-exit:${event.sessionId}:${fence}:${generation}`
}
@@ -0,0 +1,68 @@
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../shared/agent-session-journal-types'
import type {
StructuredAgentSessionEventSink,
StructuredAgentSessionJournalBlob,
StructuredAgentSessionSinkAdmission
} from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import type { CodexPendingJournalPrompt } from './codex-structured-journal-settlement'
import type { CodexJournalTranslationAdmission } from './codex-structured-journal-contracts'
import { CODEX_JOURNAL_ADMITTED } from './codex-structured-journal-contracts'
function criticalAdmission(
admission: StructuredAgentSessionSinkAdmission
): CodexJournalTranslationAdmission {
return admission.accepted ? CODEX_JOURNAL_ADMITTED : admission
}
export function appendCodexLifecycleItem(
sink: StructuredAgentSessionEventSink,
identity: AgentJournalItemIdentity,
body: AgentJournalItemBody,
blobs: readonly StructuredAgentSessionJournalBlob[] = []
): CodexJournalTranslationAdmission {
if (sink.tryAppendItem) {
return criticalAdmission(sink.tryAppendItem(identity, body, blobs, { lifecycle: true }))
}
sink.appendItem(identity, body, blobs, { lifecycle: true })
return CODEX_JOURNAL_ADMITTED
}
export function publishCodexLifecycle(
sink: StructuredAgentSessionEventSink
): CodexJournalTranslationAdmission {
if (sink.tryPublish) {
return criticalAdmission(sink.tryPublish({ lifecycle: true }))
}
sink.publish({ lifecycle: true })
return CODEX_JOURNAL_ADMITTED
}
export function admitCodexLifecycleItems(
sink: StructuredAgentSessionEventSink,
settlementId: string,
items: readonly CodexPendingJournalPrompt[]
): CodexJournalTranslationAdmission {
if (items.length === 0) {
return { accepted: false, reason: 'untranslated' }
}
if (sink.tryAppendLifecycleBatch) {
const admission = criticalAdmission(
sink.tryAppendLifecycleBatch(
settlementId,
items.map((item) => ({ kind: 'item' as const, identity: item.identity, body: item.body })),
{ lifecycle: true }
)
)
return admission.accepted ? publishCodexLifecycle(sink) : admission
}
for (const item of items) {
const admission = appendCodexLifecycleItem(sink, item.identity, item.body)
if (!admission.accepted) {
return admission
}
}
return publishCodexLifecycle(sink)
}
@@ -0,0 +1,59 @@
import type { CodexTurnOrdinals } from './codex-structured-item-translation'
import {
readCodexJournalRecord,
readCodexJournalString
} from './codex-structured-journal-translation-values'
import type { CodexJournalTranslationAdmission } from './codex-structured-journal-translation'
/** Old providers may return the complete thread from resume. Keep that fallback
* bounded before admitting any rows to the asynchronous sink. */
export const CODEX_RESTORE_MAX_OPERATIONS = 1_024
export const CODEX_RESTORE_MAX_BYTES = 16 * 1024 * 1024
export function restoreCodexJournalThread(input: {
threadId: string
thread: Record<string, unknown>
currentTurnIds: Map<string, Set<string>>
ordinals: CodexTurnOrdinals
handleItem: (event: {
threadId: string
method: string
params: unknown
}) => CodexJournalTranslationAdmission
flush: () => void
}): CodexJournalTranslationAdmission {
const turns = Array.isArray(input.thread.turns) ? input.thread.turns : []
const items = turns.flatMap((rawTurn) => {
const turn = readCodexJournalRecord(rawTurn)
const turnId = readCodexJournalString(turn, 'id')
return turnId
? (Array.isArray(turn.items) ? turn.items : []).map((item) => ({ turnId, item }))
: []
})
const encodedBytes = Buffer.byteLength(JSON.stringify(items), 'utf8')
if (items.length > CODEX_RESTORE_MAX_OPERATIONS || encodedBytes > CODEX_RESTORE_MAX_BYTES) {
return { accepted: false, reason: 'backpressure' }
}
for (const rawTurn of turns) {
const turn = readCodexJournalRecord(rawTurn)
const turnId = readCodexJournalString(turn, 'id')
if (!turnId) {
continue
}
input.currentTurnIds.set(input.threadId, new Set([turnId]))
for (const item of Array.isArray(turn.items) ? turn.items : []) {
const admission = input.handleItem({
threadId: input.threadId,
method: 'item/completed',
params: { turnId, item }
})
if (!admission.accepted) {
return admission
}
}
input.currentTurnIds.delete(input.threadId)
input.ordinals.forgetTurn(input.threadId, turnId)
}
input.flush()
return { accepted: true }
}
@@ -0,0 +1,831 @@
import { describe, expect, it, vi } from 'vitest'
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../shared/agent-session-journal-types'
import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key'
import { createJournalReducerState } from '../native-chat/agent-session-journal/journal-reducer'
import {
journalLifecycleBatchRowBuilder,
type JournalLifecycleMutationInput
} from '../native-chat/agent-session-journal/journal-row-builders'
import {
journalRowByteLength,
MAX_JOURNAL_LIFECYCLE_BATCH_BYTES,
MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS
} from '../native-chat/agent-session-journal/journal-row-schema'
import {
createDeferredStructuredAgentSessionEventSink,
type StructuredAgentSessionEventSink,
type StructuredAgentSessionEventTarget
} from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import { createCodexJournalTranslator } from './codex-structured-journal-translation'
import {
CODEX_COMMAND_APPROVAL_METHOD,
CODEX_USER_INPUT_METHOD
} from './codex-structured-prompt-replies'
import type { CodexStructuredSessionEvent } from './codex-structured-session-adapter'
const SESSION_ID = 'session-1'
const THREAD_ID = 'thread-abc'
const TURN_ID = 'turn-1'
type Row = { key: string; body: AgentJournalItemBody }
type LifecycleBatch = {
settlementId: string
mutations: JournalLifecycleMutationInput[]
}
function recorder() {
const rows: Row[] = []
const tombstones: string[] = []
const bound: [string, string, string][] = []
let publishes = 0
const sink: StructuredAgentSessionEventSink = {
appendItem: (identity: AgentJournalItemIdentity, body) =>
rows.push({ key: agentJournalItemKey(identity), body }),
appendTombstone: (identity) => tombstones.push(agentJournalItemKey(identity)),
publish: () => {
publishes += 1
}
}
return {
sink,
rows,
tombstones,
bound,
publishes: () => publishes,
bindPromptItemId: (journalItemId: string, threadId: string, promptKey: string) =>
bound.push([journalItemId, threadId, promptKey])
}
}
/** Fires the coalescing window on demand instead of on wall time. */
function manualWindow() {
const pending: (() => void)[] = []
return {
schedule: (run: () => void) => {
pending.push(run)
return () => {
const index = pending.indexOf(run)
if (index !== -1) {
pending.splice(index, 1)
}
}
},
fire: () => {
const due = pending.splice(0)
for (const run of due) {
run()
}
},
idle: () => pending.length === 0
}
}
function notification(method: string, params: unknown): CodexStructuredSessionEvent {
return { type: 'notification', sessionId: SESSION_ID, threadId: THREAD_ID, method, params }
}
const TURN_STARTED = notification('turn/started', { turn: { id: TURN_ID } })
function translatorWith(tap = recorder(), window = manualWindow()) {
const translator = createCodexJournalTranslator({
sink: tap.sink,
bindPromptItemId: tap.bindPromptItemId,
schedule: window.schedule
})
return { translator, tap, window }
}
function deferredTarget(
log: AgentJournalItemBody[],
publishes: string[] = []
): StructuredAgentSessionEventTarget {
return {
fence: 7,
journal: {
appendItem: vi.fn(async (_identity: AgentJournalItemIdentity, body: AgentJournalItemBody) => {
log.push(body)
return { cursor: { epoch: 'e', sequence: log.length } }
}),
appendTombstone: vi.fn(async () => ({ epoch: 'e', sequence: log.length })),
appendLifecycleBatch: vi.fn(
async (input: { mutations: readonly JournalLifecycleMutationInput[] }) => {
for (const mutation of input.mutations) {
if (mutation.kind === 'item') {
log.push(mutation.body)
}
}
return { epoch: 'e', sequence: log.length }
}
)
} as unknown as StructuredAgentSessionEventTarget['journal'],
publish: vi.fn(() => {
publishes.push('publish')
})
}
}
function hardWatermarkDeferred() {
return createDeferredStructuredAgentSessionEventSink({
watermarks: {
pauseQueuedBytes: 1,
maxQueuedBytes: 1,
lowQueuedBytes: 0,
pauseQueuedOperations: 1,
maxQueuedOperations: 0,
lowQueuedOperations: 0
}
})
}
function terminalExitBatches(count: number, outputBytes: number): LifecycleBatch[] {
const tap = recorder()
const batches: LifecycleBatch[] = []
tap.sink.appendLifecycleBatch = (settlementId, mutations) => {
batches.push({ settlementId, mutations: [...mutations] })
}
const translator = createCodexJournalTranslator({
sink: tap.sink,
primaryThreadId: () => THREAD_ID
})
const output = 'x'.repeat(outputBytes)
translator.handle(TURN_STARTED)
for (let index = 0; index < count; index += 1) {
const itemId = `exec-${index}`
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: itemId,
command: `run-${index}`,
status: 'inProgress'
}
})
)
translator.handle(notification('item/commandExecution/outputDelta', { itemId, delta: output }))
}
translator.handle({
type: 'ended',
sessionId: SESSION_ID,
reason: 'lost child',
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
})
return batches
}
function expectLifecycleBatchBounds(batches: readonly LifecycleBatch[]): void {
for (const [index, batch] of batches.entries()) {
expect(batch.mutations.length).toBeLessThanOrEqual(MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS)
const state = createJournalReducerState(SESSION_ID, 'epoch-test')
const row = journalLifecycleBatchRowBuilder(() => state, batch.settlementId, batch.mutations, {
fence: 7
})(index + 1, index + 1)
expect(journalRowByteLength(row)).toBeLessThanOrEqual(MAX_JOURNAL_LIFECYCLE_BATCH_BYTES)
}
}
describe('codex journal translation', () => {
it('admits turn start and turn settlement publications across the hard watermark', async () => {
const bodies: AgentJournalItemBody[] = []
const publishes: string[] = []
const deferred = hardWatermarkDeferred()
const translator = createCodexJournalTranslator({
sink: deferred.sink,
primaryThreadId: () => THREAD_ID
})
expect(translator.handle(TURN_STARTED)).toEqual({ accepted: true })
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: 'exec-hard-settlement',
command: 'run',
status: 'inProgress'
}
})
)
expect(translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))).toEqual({
accepted: true
})
expect(deferred.state()).toMatchObject({ queuedOperations: 4, backpressured: true })
deferred.bind(deferredTarget(bodies, publishes))
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
expect(bodies).toEqual([
expect.objectContaining({
kind: 'status',
turnLifecycle: { turnId: TURN_ID, state: 'running' }
}),
expect.objectContaining({ kind: 'tool-call', state: 'running' }),
expect.objectContaining({ kind: 'tool-call', state: 'failed' })
])
expect(publishes).toHaveLength(1)
})
it('admits terminal session settlement publication across the hard watermark', async () => {
const bodies: AgentJournalItemBody[] = []
const publishes: string[] = []
const deferred = hardWatermarkDeferred()
const translator = createCodexJournalTranslator({
sink: deferred.sink,
primaryThreadId: () => THREAD_ID
})
translator.handle(TURN_STARTED)
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'exec-1',
promptKey: 'approval-before-exit'
})
expect(
translator.handle({
type: 'ended',
sessionId: SESSION_ID,
reason: 'lost child',
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
})
).toEqual({ accepted: true })
expect(deferred.state()).toMatchObject({ queuedOperations: 4, backpressured: true })
deferred.bind(deferredTarget(bodies, publishes))
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
expect(bodies).toEqual([
expect.objectContaining({
kind: 'status',
turnLifecycle: { turnId: TURN_ID, state: 'running' }
}),
expect.objectContaining({
kind: 'approval',
resolution: expect.objectContaining({ state: 'pending' })
}),
expect.objectContaining({
kind: 'approval',
resolution: expect.objectContaining({ state: 'cancelled' })
}),
{ kind: 'status', text: 'Provider exited: lost child' }
])
expect(publishes).toHaveLength(1)
})
it('retries a rejected terminal admission without losing tool, prompt, turn, or session truth', () => {
const batches: LifecycleBatch[] = []
let rejected = false
const sink: StructuredAgentSessionEventSink = {
appendItem: vi.fn(),
appendTombstone: vi.fn(),
publish: vi.fn(),
tryAppendLifecycleBatch: (settlementId, mutations) => {
if (settlementId.startsWith('provider-exit:') && !rejected) {
rejected = true
return { accepted: false, reason: 'backpressure' as const }
}
batches.push({ settlementId, mutations: [...mutations] })
return { accepted: true }
},
tryPublish: () => ({ accepted: true })
}
const translator = createCodexJournalTranslator({
sink,
primaryThreadId: () => THREAD_ID
})
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: 'exec-retry-settlement',
command: 'run',
status: 'inProgress'
}
})
)
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'exec-retry-settlement',
promptKey: 'approval-retry-settlement'
})
const ended = {
type: 'ended' as const,
sessionId: SESSION_ID,
reason: 'lost child',
cause: 'unexpected-exit' as const,
fence: 7,
acquisitionGeneration: 'generation-retry'
}
expect(translator.handle(ended)).toEqual({ accepted: false, reason: 'backpressure' })
expect(translator.handle(ended)).toEqual({ accepted: true })
const mutations = batches.at(-1)?.mutations ?? []
expect(mutations).toEqual(
expect.arrayContaining([
expect.objectContaining({
body: expect.objectContaining({ kind: 'tool-call', state: 'failed' })
}),
expect.objectContaining({
body: expect.objectContaining({
kind: 'approval',
resolution: expect.objectContaining({ state: 'cancelled' })
})
}),
expect.objectContaining({
body: { kind: 'status', text: 'Provider exited: lost child' }
}),
expect.objectContaining({ kind: 'tombstone' })
])
)
})
it('bounds prompt cancellation and exit bodies before lifecycle batching', () => {
const tap = recorder()
const batches: LifecycleBatch[] = []
tap.sink.appendLifecycleBatch = (settlementId, mutations) => {
batches.push({ settlementId, mutations: [...mutations] })
}
const translator = createCodexJournalTranslator({
sink: tap.sink,
primaryThreadId: () => THREAD_ID
})
const huge = 'x'.repeat(MAX_JOURNAL_LIFECYCLE_BATCH_BYTES + 1_024)
translator.handle(TURN_STARTED)
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { command: huge, availableDecisions: ['accept', 'decline'] },
codexItemId: 'exec-1',
promptKey: `approval-${huge}`
})
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_USER_INPUT_METHOD,
params: {
questions: [
{
id: `question-${huge}`,
question: huge,
options: [{ label: huge }, { label: `${huge}b` }]
}
]
},
codexItemId: 'exec-1',
promptKey: `question-${huge}`
})
translator.handle({
type: 'ended',
sessionId: SESSION_ID,
reason: huge,
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
})
expectLifecycleBatchBounds(batches)
expect(JSON.stringify(batches)).toContain('output truncated')
})
it('splits many large terminal items before the lifecycle row byte boundary', () => {
const batches = terminalExitBatches(120, 20_000)
const flattened = batches.flatMap((batch) => batch.mutations)
expect(batches.length).toBeGreaterThan(1)
expect(batches.map((batch) => batch.settlementId)).toEqual(
batches.map(
(_batch, index) =>
`provider-exit:${SESSION_ID}:7:generation-1:${index + 1}/${batches.length}`
)
)
expect(flattened).toHaveLength(122)
expect(flattened.at(-2)).toMatchObject({
kind: 'item',
body: { kind: 'status', text: 'Provider exited: lost child' }
})
expect(flattened.at(-1)).toMatchObject({ kind: 'tombstone' })
expectLifecycleBatchBounds(batches)
})
it('partitions large terminal settlements by both byte and mutation bounds', () => {
const batches = terminalExitBatches(240, 20_000)
const mutationOnlyChunkCount = Math.ceil(
batches.flatMap((batch) => batch.mutations).length / MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS
)
expect(batches.length).toBeGreaterThan(mutationOnlyChunkCount)
expectLifecycleBatchBounds(batches)
})
it('bounds one streamed assistant settlement before lifecycle batching', () => {
const tap = recorder()
const batches: LifecycleBatch[] = []
tap.sink.appendLifecycleBatch = (settlementId, mutations) => {
batches.push({ settlementId, mutations: [...mutations] })
}
const translator = createCodexJournalTranslator({
sink: tap.sink,
primaryThreadId: () => THREAD_ID,
maxRetainedBytes: MAX_JOURNAL_LIFECYCLE_BATCH_BYTES + 1_024
})
const oversized = 'a'.repeat(MAX_JOURNAL_LIFECYCLE_BATCH_BYTES + 1_024)
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', { item: { type: 'agentMessage', id: 'assistant-1', text: '' } })
)
translator.handle(
notification('item/agentMessage/delta', { itemId: 'assistant-1', delta: oversized })
)
translator.handle({
type: 'ended',
sessionId: SESSION_ID,
reason: 'lost child',
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
})
const checkpoint = tap.rows.find((row) => row.body.kind === 'message')?.body
const settled = batches
.flatMap((batch) => batch.mutations)
.find((mutation) => mutation.kind === 'item' && mutation.body.kind === 'message') as
| Extract<JournalLifecycleMutationInput, { kind: 'item' }>
| undefined
const checkpointText =
checkpoint?.kind === 'message' && checkpoint.blocks[0]?.type === 'text'
? checkpoint.blocks[0].text
: ''
const settledText =
settled?.body.kind === 'message' && settled.body.blocks[0]?.type === 'text'
? settled.body.blocks[0].text
: ''
expect(checkpointText).toContain('output truncated')
expect(settledText).toContain('output truncated')
expect(Buffer.byteLength(settledText, 'utf8')).toBeLessThan(20 * 1024)
expectLifecycleBatchBounds(batches)
})
it('bounds authoritative completed assistant text before the journal append', () => {
const { translator, tap } = translatorWith()
const oversized = 'b'.repeat(MAX_JOURNAL_LIFECYCLE_BATCH_BYTES + 1_024)
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'assistant-1', text: oversized }
})
)
const body = tap.rows[0]?.body
const text =
body?.kind === 'message' && body.blocks[0]?.type === 'text' ? body.blocks[0].text : ''
expect(text).toContain('output truncated')
expect(Buffer.byteLength(JSON.stringify(body), 'utf8')).toBeLessThan(20 * 1024)
})
it('terminalizes an active tool when its turn completes', () => {
const tap = recorder()
const batches: { settlementId: string; mutations: unknown[] }[] = []
tap.sink.appendLifecycleBatch = (settlementId, mutations) => {
batches.push({ settlementId, mutations: [...mutations] })
}
const translator = createCodexJournalTranslator({
sink: tap.sink,
primaryThreadId: () => THREAD_ID
})
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: { type: 'commandExecution', id: 'exec-active', command: 'run', status: 'inProgress' }
})
)
translator.handle(
notification('item/commandExecution/outputDelta', {
itemId: 'exec-active',
delta: 'partial'
})
)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
expect(batches).toEqual([
{
settlementId: `turn-completed:${SESSION_ID}:${THREAD_ID}:${TURN_ID}`,
mutations: [
expect.objectContaining({
kind: 'item',
identity: expect.objectContaining({ provider: 'orca' }),
body: expect.objectContaining({
kind: 'tool-call',
state: 'failed',
output: expect.objectContaining({ head: 'partial' })
})
}),
expect.objectContaining({
kind: 'tombstone',
identity: {
provider: 'legacy',
agent: 'codex',
sessionId: SESSION_ID,
recordId: `turn-lifecycle:${TURN_ID}`
}
})
]
}
])
})
it('journals an approval naming the command the item already announced, and binds it', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: 'item-2',
command: 'rm -rf build',
status: 'inProgress'
}
})
)
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'item-2',
promptKey: 'item-2'
})
const approval = tap.rows.at(-1)
expect(approval?.key).toBe('orca:codex-prompt%3Athread-abc%3Aitem-2')
expect(approval?.body).toMatchObject({ kind: 'approval', detail: 'rm -rf build' })
expect(tap.bound).toEqual([['orca:codex-prompt%3Athread-abc%3Aitem-2', THREAD_ID, 'item-2']])
})
it('journals one row per approval when a tool item asks twice', () => {
const { translator, tap } = translatorWith()
const ask = (promptKey: string): void => {
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'item-2',
promptKey
})
}
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: { type: 'commandExecution', id: 'item-2', command: 'ls', status: 'inProgress' }
})
)
ask('approval-a')
ask('approval-b')
// Two asks, two answerable rows — keying by the tool item would have made the
// second ask overwrite the first, leaving the turn blocked.
const approvals = tap.rows.slice(-2)
expect(approvals.map((row) => row.key)).toEqual([
'orca:codex-prompt%3Athread-abc%3Aapproval-a',
'orca:codex-prompt%3Athread-abc%3Aapproval-b'
])
// Both still name the command the shared item announced.
expect(approvals.every((row) => (row.body as { detail?: string }).detail === 'ls')).toBe(true)
expect(tap.bound.map(([, , promptKey]) => promptKey)).toEqual(['approval-a', 'approval-b'])
})
it('journals and binds one row per question in a user-input request', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_USER_INPUT_METHOD,
params: {
questions: [
{ id: 'q1', question: 'Which branch?', options: [{ label: 'main' }] },
{ id: 'q2', question: 'Proceed?', options: [{ label: 'yes' }] }
]
},
codexItemId: 'item-3',
promptKey: 'item-3'
})
expect(tap.rows.map((row) => row.key)).toEqual([
'orca:codex-prompt%3Athread-abc%3Aitem-3%3Aq1',
'orca:codex-prompt%3Athread-abc%3Aitem-3%3Aq2'
])
expect(tap.bound.map(([, , promptKey]) => promptKey)).toEqual(['item-3', 'item-3'])
})
it('starts a new turn at ordinal zero and refuses to adopt an ended turn', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', { item: { type: 'userMessage', id: 'item-0', text: 'one' } })
)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-1', text: 'orphan' }
})
)
translator.handle(notification('turn/started', { turn: { id: 'turn-2' } }))
translator.handle(
notification('item/completed', { item: { type: 'userMessage', id: 'item-2', text: 'two' } })
)
expect(tap.rows.map((row) => row.key)).toEqual([
'codex:thread-abc:turn-1:0',
'orca:codex-item%3Athread-abc%3Aitem-1',
'codex:thread-abc:turn-2:0'
])
})
it('prefers a turn id the event carries over the turn currently open', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', {
turnId: 'turn-9',
item: { type: 'userMessage', id: 'item-0', text: 'late' }
})
)
expect(tap.rows[0]?.key).toBe('codex:thread-abc:turn-9:0')
})
it('keeps interleaved thread turns, items, and deltas separate', () => {
const { translator, tap } = translatorWith()
const child = (method: string, params: unknown): CodexStructuredSessionEvent => ({
type: 'notification',
sessionId: SESSION_ID,
threadId: 'thread-child',
method,
params
})
translator.handle(TURN_STARTED)
translator.handle(child('turn/started', { threadId: 'thread-child', turnId: 'turn-child' }))
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-0', text: 'root' }
})
)
translator.handle(
child('item/completed', { item: { type: 'agentMessage', id: 'item-0', text: 'child' } })
)
translator.handle(child('turn/completed', { turnId: 'turn-child' }))
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-1', text: 'still root' }
})
)
expect(tap.rows.map((row) => row.key)).toEqual([
'codex:thread-abc:turn-1:0',
'codex:thread-child:turn-child:0',
'codex:thread-abc:turn-1:1'
])
})
it('checkpoints long streams geometrically and flushes the final snapshot', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', { item: { type: 'agentMessage', id: 'item-1', text: '' } })
)
for (let index = 0; index < 512; index += 1) {
translator.handle(notification('item/agentMessage/delta', { itemId: 'item-1', delta: 'x' }))
window.fire()
}
translator.flush()
expect(tap.rows.length).toBeLessThan(40)
expect(tap.rows.at(-1)?.body).toMatchObject({
blocks: [{ type: 'text', text: 'x'.repeat(512) }]
})
})
it('folds long-running command output into one exec item and zero generic rows', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: { type: 'commandExecution', id: 'exec-1', command: 'long-task', status: 'inProgress' }
})
)
for (let index = 0; index < 512; index += 1) {
translator.handle(
notification('item/commandExecution/outputDelta', { itemId: 'exec-1', delta: 'x' })
)
window.fire()
}
translator.flush()
expect(new Set(tap.rows.map((row) => row.key))).toEqual(
new Set(['orca:codex-item%3Athread-abc%3Aexec-1'])
)
expect(tap.rows.every((row) => row.body.kind === 'tool-call')).toBe(true)
expect(tap.rows.length).toBeLessThan(40)
expect(tap.rows.at(-1)?.body).toMatchObject({
kind: 'tool-call',
output: { head: 'x'.repeat(512) }
})
})
it('folds reasoning and patch streams into their parent rows', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(notification('item/started', { item: { type: 'reasoning', id: 'r-1' } }))
translator.handle(
notification('item/reasoning/summaryTextDelta', { itemId: 'r-1', delta: 'thinking' })
)
translator.handle(
notification('item/started', {
item: { type: 'fileChange', id: 'patch-1', changes: [], status: 'inProgress' }
})
)
translator.handle(
notification('item/fileChange/patchUpdated', {
itemId: 'patch-1',
changes: [{ path: 'src/app.ts', kind: { type: 'update' }, diff: '@@ -1 +1 @@' }]
})
)
window.fire()
const reduced = new Map(tap.rows.map((row) => [row.key, row.body]))
expect(reduced.get('orca:codex-item%3Athread-abc%3Ar-1')).toEqual({
kind: 'status',
text: 'thinking'
})
expect(reduced.get('orca:codex-item%3Athread-abc%3Apatch-1')).toMatchObject({
kind: 'diff',
path: 'src/app.ts',
patch: { head: '@@ -1 +1 @@' }
})
})
it('retains a rejected patch update for a later admission retry', () => {
const { translator, tap } = translatorWith()
let rejectPatch = true
tap.sink.tryAppendItem = (identity, body, blobs) => {
if (body.kind === 'diff' && rejectPatch) {
return { accepted: false as const, reason: 'backpressure' as const }
}
tap.sink.appendItem(identity, body, blobs)
return { accepted: true as const }
}
translator.handle(
notification('item/started', {
item: { type: 'fileChange', id: 'patch-retry', changes: [], status: 'inProgress' }
})
)
const rejected = translator.handle(
notification('item/fileChange/patchUpdated', {
itemId: 'patch-retry',
changes: [{ path: 'src/app.ts', kind: { type: 'update' }, diff: '@@ -1 +1 @@' }]
})
)
expect(rejected).toEqual({ accepted: false, reason: 'backpressure' })
expect(tap.rows.some((row) => row.body.kind === 'diff')).toBe(false)
rejectPatch = false
expect(translator.flush()).toBeUndefined()
expect(tap.rows.some((row) => row.body.kind === 'diff')).toBe(true)
})
})
@@ -0,0 +1,575 @@
import { describe, expect, it, vi } from 'vitest'
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../shared/agent-session-journal-types'
import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key'
import { projectStructuredItemsToNativeChat } from '../../shared/structured-agent-session-projection'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import { CodexTurnOrdinals } from './codex-structured-item-translation'
import {
createCodexJournalTranslator,
MAX_CODEX_GENERIC_BOOKKEEPING_ENTRIES,
MAX_CODEX_GENERIC_ROWS_PER_TURN,
MAX_CODEX_GENERIC_TURN_BUCKETS
} from './codex-structured-journal-translation'
import { CODEX_COMMAND_APPROVAL_METHOD } from './codex-structured-prompt-replies'
import type { CodexStructuredSessionEvent } from './codex-structured-session-adapter'
const SESSION_ID = 'session-1'
const THREAD_ID = 'thread-abc'
const TURN_ID = 'turn-1'
type Row = { key: string; body: AgentJournalItemBody }
function recorder() {
const rows: Row[] = []
const tombstones: string[] = []
const bound: [string, string, string][] = []
let publishes = 0
const sink: StructuredAgentSessionEventSink = {
appendItem: (identity: AgentJournalItemIdentity, body) =>
rows.push({ key: agentJournalItemKey(identity), body }),
appendTombstone: (identity) => tombstones.push(agentJournalItemKey(identity)),
publish: () => {
publishes += 1
}
}
return {
sink,
rows,
tombstones,
bound,
publishes: () => publishes,
bindPromptItemId: (journalItemId: string, threadId: string, promptKey: string) =>
bound.push([journalItemId, threadId, promptKey])
}
}
/** Fires the coalescing window on demand instead of on wall time. */
function manualWindow() {
const pending: (() => void)[] = []
return {
schedule: (run: () => void) => {
pending.push(run)
return () => {
const index = pending.indexOf(run)
if (index !== -1) {
pending.splice(index, 1)
}
}
},
fire: () => {
const due = pending.splice(0)
for (const run of due) {
run()
}
},
idle: () => pending.length === 0
}
}
function notification(method: string, params: unknown): CodexStructuredSessionEvent {
return { type: 'notification', sessionId: SESSION_ID, threadId: THREAD_ID, method, params }
}
const TURN_STARTED = notification('turn/started', { turn: { id: TURN_ID } })
function translatorWith(tap = recorder(), window = manualWindow()) {
const translator = createCodexJournalTranslator({
sink: tap.sink,
bindPromptItemId: tap.bindPromptItemId,
schedule: window.schedule
})
return { translator, tap, window }
}
describe('codex journal translation', () => {
it('retains active state when eviction settlement is backpressured', () => {
const { translator, tap } = translatorWith()
let rejectTerminal = true
const appendItem = tap.sink.appendItem
tap.sink.tryAppendItem = (identity, body, blobs, options) => {
if (rejectTerminal && body.kind === 'tool-call' && body.state === 'failed') {
return { accepted: false as const, reason: 'backpressure' as const }
}
appendItem(identity, body, blobs, options)
return { accepted: true as const }
}
for (let index = 0; index <= 256; index += 1) {
const result = translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: `evict-${index}`,
command: 'run',
status: 'inProgress'
}
})
)
if (index === 256) {
expect(result).toEqual({ accepted: false, reason: 'backpressure' })
}
}
rejectTerminal = false
expect(
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: 'evict-retry',
command: 'run',
status: 'inProgress'
}
})
)
).toEqual({ accepted: true })
expect(
tap.rows.filter((row) => row.body.kind === 'tool-call' && row.body.state === 'failed').length
).toBeGreaterThan(0)
})
it('terminalizes evicted pending prompts instead of silently forgetting them', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
for (let index = 0; index <= 128; index += 1) {
expect(
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: {},
codexItemId: `prompt-item-${index}`,
promptKey: `prompt-${index}`
})
).toEqual({ accepted: true })
}
expect(
tap.rows.some(
(row) => row.body.kind === 'approval' && row.body.resolution.state === 'cancelled'
)
).toBe(true)
})
it('publishes after every write so a subscriber never trails the journal', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', { item: { type: 'userMessage', id: 'item-0', text: 'hi' } })
)
expect(tap.publishes()).toBe(1)
})
it('releases a turn ordinal map when the turn completes', () => {
const spy = vi.spyOn(CodexTurnOrdinals.prototype, 'forgetTurn')
try {
const { translator } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
expect(spy).toHaveBeenCalledWith(THREAD_ID, TURN_ID)
} finally {
spy.mockRestore()
}
})
it('journals malformed item events but never malformed deltas', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(notification('item/completed', {}))
translator.handle(notification('item/agentMessage/delta', { delta: 'orphan' }))
window.fire()
expect(tap.rows.map((row) => row.body)).toEqual([
expect.objectContaining({
kind: 'status',
providerFrame: expect.objectContaining({ kind: 'notification:item/completed' })
})
])
})
it('journals unknown notifications, server requests, and decoded provider frames', () => {
const { translator, tap } = translatorWith()
translator.handle(notification('future/notification', { value: 1 }))
translator.handle({
type: 'server-request',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: 'future/request',
params: { value: 2 }
})
translator.handle({
type: 'provider-frame',
sessionId: SESSION_ID,
threadId: THREAD_ID,
kind: 'frame:unclassified',
payload: { value: 3 }
})
expect(
tap.rows.map((row) => (row.body.kind === 'status' ? row.body.providerFrame?.kind : undefined))
).toEqual(['notification:future/notification', 'request:future/request', 'frame:unclassified'])
})
it('terminalizes the active streamed item when an oversized notification is rejected', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: 'exec-oversized',
command: 'run',
status: 'inProgress'
}
})
)
const admission = translator.handle({
type: 'provider-frame',
sessionId: SESSION_ID,
threadId: THREAD_ID,
kind: 'frame:oversized-notification',
payload: {
reason: 'record-too-large',
observedBytes: 20 * 1024 * 1024,
maxBytes: 16 * 1024 * 1024,
classification: 'notification',
method: 'item/commandExecution/outputDelta'
}
})
expect(admission).toEqual({ accepted: true })
expect(tap.rows).toEqual([
expect.objectContaining({
body: expect.objectContaining({ kind: 'tool-call', state: 'running' })
}),
expect.objectContaining({
body: expect.objectContaining({ kind: 'tool-call', state: 'failed' })
}),
expect.objectContaining({
body: expect.objectContaining({
kind: 'status',
providerFrame: expect.objectContaining({ kind: 'frame:oversized-notification' })
})
})
])
const diagnostic = tap.rows[2]?.body
expect(
diagnostic?.kind === 'status' ? diagnostic.providerFrame?.payload.byteLength : 0
).toBeGreaterThan(0)
expect(JSON.stringify(diagnostic)).toContain('record-too-large')
})
it('admits suppressed diagnostics before settling a completed turn', () => {
const tap = recorder()
let rejectSuppression = true
const appendItem = tap.sink.appendItem
tap.sink.tryAppendItem = (...args) => {
const body = args[1]
if (body.kind === 'status' && body.text.includes('more provider notification')) {
return rejectSuppression
? { accepted: false as const, reason: 'backpressure' as const }
: (appendItem(...args), { accepted: true as const })
}
appendItem(...args)
return { accepted: true as const }
}
const { translator } = translatorWith(tap)
translator.handle(TURN_STARTED)
for (let index = 0; index < MAX_CODEX_GENERIC_ROWS_PER_TURN + 1; index += 1) {
translator.handle(notification('future/notification', { value: index }))
}
translator.handle(
notification('item/started', {
item: { type: 'commandExecution', id: 'exec-order', command: 'run', status: 'inProgress' }
})
)
expect(translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))).toEqual({
accepted: false,
reason: 'backpressure'
})
expect(tap.tombstones).toEqual([])
rejectSuppression = false
expect(translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))).toEqual({
accepted: true
})
})
it('bounds generic rows per turn while keeping the suppression visible and countable', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
for (let index = 0; index < MAX_CODEX_GENERIC_ROWS_PER_TURN + 20; index += 1) {
translator.handle(notification('future/notification', { value: index }))
}
translator.handle(notification('item/future/outputDelta', { itemId: 'future', delta: 'x' }))
window.fire()
const generic = tap.rows.filter(
(row) => row.body.kind === 'status' && row.body.providerFrame !== undefined
)
expect(generic).toHaveLength(MAX_CODEX_GENERIC_ROWS_PER_TURN)
expect(generic[0]?.body).toMatchObject({
kind: 'status',
providerFrame: { kind: 'notification:future/notification' }
})
// The 20 capped frames reduce to ONE summary row whose count is exact, so
// suppressed provider activity is never invisible.
const summaries = new Map(
tap.rows
.filter((row) => row.key.includes('provider-frame-suppressed'))
.map((row) => [row.key, row.body])
)
expect(summaries.size).toBe(1)
expect([...summaries.values()][0]).toEqual({
kind: 'status',
text: '20 more provider notifications not shown for this turn'
})
expect(
tap.rows.some(
(row) =>
row.body.kind === 'status' &&
row.body.providerFrame?.kind === 'notification:item/future/outputDelta'
)
).toBe(false)
})
it('coalesces a suppressed provider-frame flood into one append and publish', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
for (let index = 0; index < MAX_CODEX_GENERIC_ROWS_PER_TURN; index += 1) {
translator.handle(notification('future/notification', { value: index }))
}
const publishesBeforeSuppression = tap.publishes()
for (let index = 0; index < 500; index += 1) {
translator.handle(notification('future/notification', { value: `suppressed-${index}` }))
}
expect(tap.rows.filter((row) => row.key.includes('provider-frame-suppressed'))).toHaveLength(0)
expect(tap.publishes()).toBe(publishesBeforeSuppression)
window.fire()
const summaries = tap.rows.filter((row) => row.key.includes('provider-frame-suppressed'))
expect(summaries).toHaveLength(1)
expect(summaries[0]?.body).toEqual({
kind: 'status',
text: '500 more provider notifications not shown for this turn'
})
expect(tap.publishes()).toBe(publishesBeforeSuppression + 1)
})
it('does not advance generic or suppression state when the sink rejects', () => {
const { tap, window } = translatorWith()
let reject = true
const appendItem = tap.sink.appendItem
tap.sink.tryAppendItem = (...args) => {
if (reject) {
return { accepted: false as const, reason: 'backpressure' as const }
}
appendItem(...args)
return { accepted: true as const }
}
const translator = createCodexJournalTranslator({
sink: tap.sink,
primaryThreadId: () => THREAD_ID,
schedule: window.schedule
})
translator.handle(TURN_STARTED)
translator.handle(notification('future/notification', { value: 1 }))
reject = false
translator.handle(notification('future/notification', { value: 2 }))
expect(
tap.rows.filter((row) => row.body.kind === 'status' && row.body.providerFrame)
).toHaveLength(1)
for (let index = 1; index < MAX_CODEX_GENERIC_ROWS_PER_TURN; index += 1) {
translator.handle(notification('future/notification', { value: index + 2 }))
}
reject = true
translator.handle(notification('future/notification', { value: 'suppressed' }))
window.fire()
expect(tap.rows.filter((row) => row.key.includes('provider-frame-suppressed'))).toHaveLength(0)
reject = false
window.fire()
expect(tap.rows.filter((row) => row.key.includes('provider-frame-suppressed'))).toHaveLength(1)
})
it('bounds error-surface provider frames under the generic-row cap', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
for (let index = 0; index < MAX_CODEX_GENERIC_ROWS_PER_TURN + 3; index += 1) {
translator.handle(notification('future/notification', { value: index }))
}
translator.handle(notification('future/failure', { error: 'provider exploded' }))
window.fire()
const generic = tap.rows.filter(
(row) => row.body.kind === 'status' && row.body.providerFrame !== undefined
)
expect(generic).toHaveLength(MAX_CODEX_GENERIC_ROWS_PER_TURN)
expect(tap.rows.filter((row) => row.key.includes('provider-frame-suppressed'))).toHaveLength(1)
expect(tap.rows.at(-1)?.body).toEqual({
kind: 'status',
text: '4 more provider notifications not shown for this turn'
})
})
it('coalesces oldest unique turn buckets while preserving counts and completion', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
const uniqueTurns = MAX_CODEX_GENERIC_TURN_BUCKETS + 12
for (let turn = 0; turn < uniqueTurns; turn += 1) {
const turnId = `adversarial-${turn}`
for (let row = 0; row < MAX_CODEX_GENERIC_ROWS_PER_TURN + 1; row += 1) {
translator.handle(
notification('future/notification', {
turn: { id: turnId },
value: `${turnId}-${row}`
})
)
}
}
window.fire()
const summaries = tap.rows.filter((row) => row.key.includes('provider-frame-suppressed'))
expect(
summaries.some(
(row) => row.body.kind === 'status' && row.body.text.includes('across evicted turns')
)
).toBe(true)
expect(
summaries.reduce((total, row) => {
if (row.body.kind !== 'status') {
return total
}
const match = row.body.text.match(/^(\d+) more provider notification/)
return total + (match ? Number(match[1]) : 0)
}, 0)
).toBe(uniqueTurns)
expect(translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))).toEqual({
accepted: true
})
expect(tap.tombstones).toContain('legacy:codex:session-1:turn-lifecycle%3Aturn-1')
// The two maps share one bounded bucket budget; this assertion documents
// the contract for future changes even though the maps are private.
expect(MAX_CODEX_GENERIC_BOOKKEEPING_ENTRIES).toBeGreaterThanOrEqual(
MAX_CODEX_GENERIC_TURN_BUCKETS
)
})
it('keeps a fresh session timeline empty through startup and status notifications', () => {
const { translator, tap } = translatorWith()
translator.handle(notification('thread/started', { thread: { id: THREAD_ID } }))
for (let index = 0; index < 8; index += 1) {
translator.handle(
notification('mcpServer/startupStatus/updated', {
server: `server-${index}`,
status: 'starting'
})
)
}
translator.handle(notification('remoteControl/status/changed', { status: 'disabled' }))
const timeline = projectStructuredItemsToNativeChat(
tap.rows.map((row, index) => ({
itemId: row.key,
revision: 1,
sequence: index + 1,
observedAt: index + 1,
body: row.body
}))
)
expect(timeline).toEqual([])
})
it('projects only user and assistant content for a complete turn with hooks', () => {
const { translator, tap } = translatorWith()
translator.handle(notification('thread/started', { thread: { id: THREAD_ID } }))
translator.handle(notification('hook/started', { run: { id: 'hook-1', status: 'running' } }))
translator.handle(notification('account/rateLimits/updated', { rateLimits: { primary: null } }))
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', {
item: { type: 'userMessage', id: 'item-0', text: 'hi' }
})
)
translator.handle(
notification('hook/completed', { run: { id: 'hook-1', status: 'completed' } })
)
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-1', text: 'hello' }
})
)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
const timeline = projectStructuredItemsToNativeChat(
tap.rows.map((row, index) => ({
itemId: row.key,
revision: 1,
sequence: index + 1,
observedAt: index + 1,
body: row.body
}))
)
expect(timeline.map(({ role, blocks }) => ({ role, blocks }))).toEqual([
{ role: 'user', blocks: [{ type: 'text', text: 'hi' }] },
{ role: 'assistant', blocks: [{ type: 'text', text: 'hello' }] }
])
})
it('renders a system error carried by a suppressed status kind', () => {
const { translator, tap } = translatorWith()
translator.handle(
notification('thread/status/changed', {
threadId: THREAD_ID,
status: { type: 'systemError' }
})
)
const timeline = projectStructuredItemsToNativeChat(
tap.rows.map((row, index) => ({
itemId: row.key,
revision: 1,
sequence: index + 1,
observedAt: index + 1,
body: row.body
}))
)
expect(timeline).toEqual([
expect.objectContaining({
role: 'system',
blocks: [
expect.objectContaining({
providerFrame: expect.objectContaining({
kind: 'notification:thread/status/changed'
})
})
]
})
])
})
it('writes nothing more after dispose', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', { item: { type: 'agentMessage', id: 'item-1', text: '' } })
)
translator.handle(notification('item/agentMessage/delta', { itemId: 'item-1', delta: 'gone' }))
translator.dispose()
window.fire()
expect(tap.rows).toEqual([])
})
})
@@ -0,0 +1,43 @@
import { describe, expect, it } from 'vitest'
import {
CodexJournalActiveTurns,
MAX_CODEX_ACTIVE_TURN_BYTES,
MAX_CODEX_ACTIVE_TURNS
} from './codex-structured-journal-translation-turn-state'
describe('CodexJournalActiveTurns', () => {
it('refuses new turns at the bounded active capacity without evicting live state', () => {
const active = new CodexJournalActiveTurns()
for (let index = 0; index < MAX_CODEX_ACTIVE_TURNS; index += 1) {
expect(active.remember(`thread-${index}`, `turn-${index}`)).toBe(true)
}
expect(active.remember('thread-overflow', 'turn-overflow')).toBe(false)
expect(active.size).toBe(MAX_CODEX_ACTIVE_TURNS)
expect(active.byThread.size).toBe(MAX_CODEX_ACTIVE_TURNS)
expect(active.current('thread-0')).toBe('turn-0')
expect(active.current(`thread-${MAX_CODEX_ACTIVE_TURNS - 1}`)).toBe(
`turn-${MAX_CODEX_ACTIVE_TURNS - 1}`
)
})
it('admits a new turn after an earlier turn settles', () => {
const active = new CodexJournalActiveTurns()
for (let index = 0; index < MAX_CODEX_ACTIVE_TURNS; index += 1) {
active.remember('thread', `turn-${index}`)
}
active.forget('thread', 'turn-0')
expect(active.remember('thread', 'turn-new')).toBe(true)
expect(active.size).toBe(MAX_CODEX_ACTIVE_TURNS)
expect(active.current('thread')).toBe('turn-new')
})
it('refuses provider identifiers that would exceed the aggregate byte bound', () => {
const active = new CodexJournalActiveTurns()
expect(active.remember('thread', 'x'.repeat(MAX_CODEX_ACTIVE_TURN_BYTES))).toBe(false)
expect(active.size).toBe(0)
expect(active.bytes).toBe(0)
})
})
@@ -0,0 +1,70 @@
export const MAX_CODEX_ACTIVE_TURNS = 256
export const MAX_CODEX_ACTIVE_TURN_BYTES = 256 * 1024
export class CodexJournalActiveTurns {
/** Bounds active turn keys retained across provider threads. */
static readonly MAX_ENTRIES = MAX_CODEX_ACTIVE_TURNS
readonly byThread = new Map<string, Set<string>>()
private activeCount = 0
private retainedBytes = 0
get size(): number {
return this.activeCount
}
get bytes(): number {
return this.retainedBytes
}
private entryBytes(threadId: string, turnId: string): number {
return Buffer.byteLength(threadId, 'utf8') + Buffer.byteLength(turnId, 'utf8')
}
canRemember(threadId: string, turnId: string): boolean {
const active = this.byThread.get(threadId)
return (
active?.has(turnId) === true ||
(this.activeCount < CodexJournalActiveTurns.MAX_ENTRIES &&
this.retainedBytes + this.entryBytes(threadId, turnId) <= MAX_CODEX_ACTIVE_TURN_BYTES)
)
}
current(threadId: string): string | null {
return [...(this.byThread.get(threadId) ?? [])].at(-1) ?? null
}
remember(threadId: string, turnId: string): boolean {
const active = this.byThread.get(threadId)
if (active?.has(turnId)) {
return true
}
if (!this.canRemember(threadId, turnId)) {
return false
}
if (active) {
active.add(turnId)
} else {
this.byThread.set(threadId, new Set([turnId]))
}
this.activeCount += 1
this.retainedBytes += this.entryBytes(threadId, turnId)
return true
}
forget(threadId: string, turnId: string): void {
const active = this.byThread.get(threadId)
if (active?.delete(turnId)) {
this.activeCount -= 1
this.retainedBytes = Math.max(0, this.retainedBytes - this.entryBytes(threadId, turnId))
}
if (!active?.size) {
this.byThread.delete(threadId)
}
}
clear(): void {
this.byThread.clear()
this.activeCount = 0
this.retainedBytes = 0
}
}
@@ -0,0 +1,66 @@
import type {
StructuredAgentSessionEventSink,
StructuredAgentSessionSinkAdmission
} from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
const ADMITTED: StructuredAgentSessionSinkAdmission = { accepted: true }
export function publishCodexTurnLifecycle(input: {
sink: StructuredAgentSessionEventSink
primaryThreadId: string | null
sessionId: string
threadId: string
turnId: string
state: 'running' | 'completed'
}): StructuredAgentSessionSinkAdmission {
if (input.primaryThreadId !== input.threadId) {
return ADMITTED
}
const identity = {
provider: 'legacy' as const,
agent: 'codex' as const,
sessionId: input.sessionId,
recordId: `turn-lifecycle:${input.turnId}`
}
if (input.state === 'completed') {
if (input.sink.tryAppendTombstone) {
const admission = input.sink.tryAppendTombstone(identity, { lifecycle: true })
if (!admission.accepted) {
return admission
}
} else {
input.sink.appendTombstone(identity, { lifecycle: true })
}
} else {
const admission = input.sink.tryAppendItem
? input.sink.tryAppendItem(
identity,
{
kind: 'status',
text: 'Codex is working…',
turnLifecycle: { turnId: input.turnId, state: input.state }
},
[],
{ lifecycle: true }
)
: (input.sink.appendItem(
identity,
{
kind: 'status',
text: 'Codex is working…',
turnLifecycle: { turnId: input.turnId, state: input.state }
},
[],
{ lifecycle: true }
),
ADMITTED)
if (!admission.accepted) {
return admission
}
}
if (input.sink.tryPublish) {
return input.sink.tryPublish({ lifecycle: true })
}
input.sink.publish({ lifecycle: true })
return ADMITTED
}
@@ -0,0 +1,11 @@
export function readCodexJournalRecord(value: unknown): Record<string, unknown> {
return typeof value === 'object' && value !== null ? (value as Record<string, unknown>) : {}
}
export function readCodexJournalString(
source: Record<string, unknown>,
key: string
): string | null {
const value = source[key]
return typeof value === 'string' && value.length > 0 ? value : null
}
@@ -4,16 +4,15 @@ import type {
AgentJournalItemIdentity
} from '../../shared/agent-session-journal-types'
import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key'
import type { JournalLifecycleMutationInput } from '../native-chat/agent-session-journal/journal-row-builders'
import { projectStructuredAgentSessionStatus } from '../../shared/structured-agent-session-projection'
import {
projectStructuredAgentSessionStatus,
projectStructuredItemsToNativeChat
} from '../../shared/structured-agent-session-projection'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import { CodexTurnOrdinals } from './codex-structured-item-translation'
import {
createCodexJournalTranslator,
MAX_CODEX_GENERIC_ROWS_PER_TURN
} from './codex-structured-journal-translation'
createDeferredStructuredAgentSessionEventSink,
type StructuredAgentSessionEventSink,
type StructuredAgentSessionEventTarget
} from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import { createCodexJournalTranslator } from './codex-structured-journal-translation'
import { MAX_CODEX_ACTIVE_TURNS } from './codex-structured-journal-translation-turn-state'
import {
CODEX_COMMAND_APPROVAL_METHOD,
CODEX_USER_INPUT_METHOD
@@ -52,20 +51,24 @@ function recorder() {
/** Fires the coalescing window on demand instead of on wall time. */
function manualWindow() {
let pending: (() => void) | null = null
const pending: (() => void)[] = []
return {
schedule: (run: () => void) => {
pending = run
pending.push(run)
return () => {
pending = null
const index = pending.indexOf(run)
if (index !== -1) {
pending.splice(index, 1)
}
}
},
fire: () => {
const run = pending
pending = null
run?.()
const due = pending.splice(0)
for (const run of due) {
run()
}
},
idle: () => pending === null
idle: () => pending.length === 0
}
}
@@ -84,7 +87,76 @@ function translatorWith(tap = recorder(), window = manualWindow()) {
return { translator, tap, window }
}
function deferredTarget(
log: AgentJournalItemBody[],
publishes: string[] = []
): StructuredAgentSessionEventTarget {
return {
fence: 7,
journal: {
appendItem: vi.fn(async (_identity: AgentJournalItemIdentity, body: AgentJournalItemBody) => {
log.push(body)
return { cursor: { epoch: 'e', sequence: log.length } }
}),
appendTombstone: vi.fn(async () => ({ epoch: 'e', sequence: log.length })),
appendLifecycleBatch: vi.fn(
async (input: { mutations: readonly JournalLifecycleMutationInput[] }) => {
for (const mutation of input.mutations) {
if (mutation.kind === 'item') {
log.push(mutation.body)
}
}
return { epoch: 'e', sequence: log.length }
}
)
} as unknown as StructuredAgentSessionEventTarget['journal'],
publish: vi.fn(() => {
publishes.push('publish')
})
}
}
function hardWatermarkDeferred() {
return createDeferredStructuredAgentSessionEventSink({
watermarks: {
pauseQueuedBytes: 1,
maxQueuedBytes: 1,
lowQueuedBytes: 0,
pauseQueuedOperations: 1,
maxQueuedOperations: 0,
lowQueuedOperations: 0
}
})
}
describe('codex journal translation', () => {
it('refuses an active-turn overflow before publishing an un-settleable lifecycle row', () => {
const tap = recorder()
const translator = createCodexJournalTranslator({
sink: tap.sink,
primaryThreadId: () => THREAD_ID
})
for (let index = 0; index < MAX_CODEX_ACTIVE_TURNS; index += 1) {
expect(
translator.handle(notification('turn/started', { turn: { id: `turn-${index}` } }))
).toEqual({ accepted: true })
}
expect(
translator.handle(notification('turn/started', { turn: { id: 'turn-overflow' } }))
).toEqual({ accepted: false, reason: 'backpressure' })
expect(tap.rows.filter((row) => row.body.kind === 'status')).toHaveLength(
MAX_CODEX_ACTIVE_TURNS
)
expect(translator.handle(notification('turn/completed', { turn: { id: 'turn-0' } }))).toEqual({
accepted: true
})
expect(
translator.handle(notification('turn/started', { turn: { id: 'turn-overflow' } }))
).toEqual({ accepted: true })
})
it('projects turns restored by thread/resume into durable conversation rows', () => {
const { translator, tap } = translatorWith()
@@ -118,6 +190,26 @@ describe('codex journal translation', () => {
])
})
it('refuses an old-provider restore above the operation bound before partial import', () => {
const { translator, tap } = translatorWith()
const result = translator.restoreThread(THREAD_ID, {
turns: [
{
id: 'turn-restored',
items: Array.from({ length: 1_025 }, (_, index) => ({
type: 'agentMessage',
id: `agent-${index}`,
text: `answer-${index}`
}))
}
]
})
expect(result).toEqual({ accepted: false, reason: 'backpressure' })
expect(tap.rows).toEqual([])
expect(tap.publishes()).toBe(0)
})
it('durably opens and closes the primary turn cancellation lifecycle', () => {
const tap = recorder()
const translator = createCodexJournalTranslator({
@@ -152,10 +244,11 @@ describe('codex journal translation', () => {
translator.handle(notification('turn/started', { turn: { id: 'turn-later' } }))
translator.handle({ type: 'ended', sessionId: SESSION_ID, reason: 'app-server exited' })
expect(tap.rows.filter((row) => row.body.kind === 'status')).toHaveLength(2)
expect(tap.rows.filter((row) => row.body.kind === 'status')).toHaveLength(3)
expect(tap.rows.map((row) => row.body)).toEqual([
expect.objectContaining({ turnLifecycle: { turnId: 'turn-stale', state: 'running' } }),
expect.objectContaining({ turnLifecycle: { turnId: 'turn-later', state: 'running' } })
expect.objectContaining({ turnLifecycle: { turnId: 'turn-later', state: 'running' } }),
expect.objectContaining({ text: 'Provider exited: app-server exited' })
])
expect(tap.tombstones).toEqual([
'legacy:codex:session-1:turn-lifecycle%3Aturn-stale',
@@ -309,80 +402,195 @@ describe('codex journal translation', () => {
translator.handle(notification('item/agentMessage/delta', { itemId: 'item-1', delta: 'half' }))
translator.handle({ type: 'ended', sessionId: SESSION_ID, reason: 'app-server exited' })
expect(tap.rows.at(-1)?.body).toMatchObject({ blocks: [{ type: 'text', text: 'half' }] })
expect(tap.rows.map((row) => row.body)).toEqual(
expect.arrayContaining([
expect.objectContaining({ blocks: [{ type: 'text', text: 'half' }] }),
{ kind: 'status', text: 'Provider exited: app-server exited' }
])
)
expect(window.idle()).toBe(true)
})
it('journals an approval naming the command the item already announced, and binds it', () => {
const { translator, tap } = translatorWith()
it('settles tools, prompts, exit status, and turn tombstone in one ordered batch', () => {
const tap = recorder()
const batches: { settlementId: string; mutations: unknown[] }[] = []
tap.sink.appendLifecycleBatch = (settlementId, mutations) => {
batches.push({ settlementId, mutations: [...mutations] })
}
const translator = createCodexJournalTranslator({
sink: tap.sink,
bindPromptItemId: tap.bindPromptItemId,
primaryThreadId: () => THREAD_ID
})
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: {
type: 'commandExecution',
id: 'item-2',
command: 'rm -rf build',
status: 'inProgress'
}
item: { type: 'commandExecution', id: 'exec-1', command: 'run', status: 'inProgress' }
})
)
translator.handle(
notification('item/commandExecution/outputDelta', { itemId: 'exec-1', delta: 'partial' })
)
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'item-2',
promptKey: 'item-2'
params: {},
codexItemId: 'exec-1',
promptKey: 'approval-1'
})
const approval = tap.rows.at(-1)
expect(approval?.key).toBe('orca:codex-prompt%3Athread-abc%3Aitem-2')
expect(approval?.body).toMatchObject({ kind: 'approval', detail: 'rm -rf build' })
expect(tap.bound).toEqual([['orca:codex-prompt%3Athread-abc%3Aitem-2', THREAD_ID, 'item-2']])
translator.handle({
type: 'ended',
sessionId: SESSION_ID,
reason: 'lost child',
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
})
expect(batches).toHaveLength(1)
expect(batches[0]?.settlementId).toBe('provider-exit:session-1:7:generation-1')
expect(batches[0]?.mutations).toEqual([
expect.objectContaining({
kind: 'item',
body: expect.objectContaining({ kind: 'tool-call', state: 'failed' })
}),
expect.objectContaining({
kind: 'item',
body: expect.objectContaining({
kind: 'approval',
resolution: expect.objectContaining({ state: 'cancelled' })
})
}),
expect.objectContaining({
kind: 'item',
body: { kind: 'status', text: 'Provider exited: lost child' }
}),
expect.objectContaining({ kind: 'tombstone' })
])
})
it('journals one row per approval when a tool item asks twice', () => {
const { translator, tap } = translatorWith()
const ask = (promptKey: string): void => {
translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'item-2',
promptKey
})
}
it('admits authoritative item completion across the deferred sink hard watermark', async () => {
const bodies: AgentJournalItemBody[] = []
const publishes: string[] = []
const readingControl = { pauseReading: vi.fn(), resumeReading: vi.fn() }
const deferred = createDeferredStructuredAgentSessionEventSink({
watermarks: {
pauseQueuedBytes: 1,
maxQueuedBytes: 1,
lowQueuedBytes: 0,
pauseQueuedOperations: 1,
maxQueuedOperations: 0,
lowQueuedOperations: 0
},
readingControl
})
const translator = createCodexJournalTranslator({ sink: deferred.sink })
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: { type: 'commandExecution', id: 'item-2', command: 'ls', status: 'inProgress' }
item: { type: 'commandExecution', id: 'exec-hard-watermark', status: 'inProgress' }
})
)
translator.handle(
notification('item/completed', {
item: {
type: 'commandExecution',
id: 'exec-hard-watermark',
command: 'run',
status: 'completed',
aggregated_output: 'done'
}
})
)
ask('approval-a')
ask('approval-b')
// Two asks, two answerable rows — keying by the tool item would have made the
// second ask overwrite the first, leaving the turn blocked.
const approvals = tap.rows.slice(-2)
expect(approvals.map((row) => row.key)).toEqual([
'orca:codex-prompt%3Athread-abc%3Aapproval-a',
'orca:codex-prompt%3Athread-abc%3Aapproval-b'
expect(deferred.state()).toMatchObject({ queuedOperations: 3, backpressured: true })
expect(readingControl.pauseReading).toHaveBeenCalled()
deferred.bind(deferredTarget(bodies, publishes))
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
expect(bodies).toEqual([
expect.objectContaining({ kind: 'tool-call', state: 'running' }),
expect.objectContaining({
kind: 'tool-call',
state: 'completed',
output: expect.objectContaining({ head: 'done' })
})
])
// Both still name the command the shared item announced.
expect(approvals.every((row) => (row.body as { detail?: string }).detail === 'ls')).toBe(true)
expect(tap.bound.map(([, , promptKey]) => promptKey)).toEqual(['approval-a', 'approval-b'])
expect(publishes).toHaveLength(1)
expect(deferred.state()).toMatchObject({ queuedOperations: 0, backpressured: false })
expect(readingControl.resumeReading).toHaveBeenCalled()
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'next-message', text: 'next turn still works' }
})
)
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
expect(bodies.at(-1)).toMatchObject({
kind: 'message',
blocks: [{ type: 'text', text: 'next turn still works' }]
})
})
it('journals and binds one row per question in a user-input request', () => {
const { translator, tap } = translatorWith()
it('admits command approval prompts and their publish across the hard watermark', async () => {
const bodies: AgentJournalItemBody[] = []
const publishes: string[] = []
const bound: [string, string, string][] = []
const deferred = hardWatermarkDeferred()
const translator = createCodexJournalTranslator({
sink: deferred.sink,
bindPromptItemId: (journalItemId, threadId, promptKey) =>
bound.push([journalItemId, threadId, promptKey])
})
translator.handle(TURN_STARTED)
translator.handle({
const admission = translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { availableDecisions: ['accept', 'decline'] },
codexItemId: 'exec-1',
promptKey: 'approval-hard-watermark'
})
expect(admission).toEqual({ accepted: true })
expect(bound).toEqual([
[
'orca:codex-prompt%3Athread-abc%3Aapproval-hard-watermark',
THREAD_ID,
'approval-hard-watermark'
]
])
expect(deferred.state()).toMatchObject({ queuedOperations: 2, backpressured: true })
deferred.bind(deferredTarget(bodies, publishes))
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
expect(bodies).toEqual([
expect.objectContaining({
kind: 'approval',
resolution: expect.objectContaining({ state: 'pending' })
})
])
expect(publishes).toHaveLength(1)
})
it('admits user-input prompt questions and their publish across the hard watermark', async () => {
const bodies: AgentJournalItemBody[] = []
const publishes: string[] = []
const bound: [string, string, string][] = []
const deferred = hardWatermarkDeferred()
const translator = createCodexJournalTranslator({
sink: deferred.sink,
bindPromptItemId: (journalItemId, threadId, promptKey) =>
bound.push([journalItemId, threadId, promptKey])
})
const admission = translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: THREAD_ID,
@@ -393,393 +601,71 @@ describe('codex journal translation', () => {
{ id: 'q2', question: 'Proceed?', options: [{ label: 'yes' }] }
]
},
codexItemId: 'item-3',
promptKey: 'item-3'
codexItemId: 'exec-1',
promptKey: 'input-hard-watermark'
})
expect(tap.rows.map((row) => row.key)).toEqual([
'orca:codex-prompt%3Athread-abc%3Aitem-3%3Aq1',
'orca:codex-prompt%3Athread-abc%3Aitem-3%3Aq2'
expect(admission).toEqual({ accepted: true })
expect(bound.map(([journalItemId]) => journalItemId)).toEqual([
'orca:codex-prompt%3Athread-abc%3Ainput-hard-watermark%3Aq1',
'orca:codex-prompt%3Athread-abc%3Ainput-hard-watermark%3Aq2'
])
expect(tap.bound.map(([, , promptKey]) => promptKey)).toEqual(['item-3', 'item-3'])
})
expect(deferred.state()).toMatchObject({ queuedOperations: 2, backpressured: true })
it('starts a new turn at ordinal zero and refuses to adopt an ended turn', () => {
const { translator, tap } = translatorWith()
deferred.bind(deferredTarget(bodies, publishes))
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', { item: { type: 'userMessage', id: 'item-0', text: 'one' } })
)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-1', text: 'orphan' }
})
)
translator.handle(notification('turn/started', { turn: { id: 'turn-2' } }))
translator.handle(
notification('item/completed', { item: { type: 'userMessage', id: 'item-2', text: 'two' } })
)
expect(tap.rows.map((row) => row.key)).toEqual([
'codex:thread-abc:turn-1:0',
'orca:codex-item%3Athread-abc%3Aitem-1',
'codex:thread-abc:turn-2:0'
expect(bodies).toEqual([
expect.objectContaining({ kind: 'question', question: 'Which branch?' }),
expect.objectContaining({ kind: 'question', question: 'Proceed?' })
])
expect(publishes).toHaveLength(1)
})
it('prefers a turn id the event carries over the turn currently open', () => {
const { translator, tap } = translatorWith()
it('refuses an untranslated user-input prompt without binding live state', () => {
const tap = recorder()
const translator = createCodexJournalTranslator({
sink: tap.sink,
bindPromptItemId: tap.bindPromptItemId
})
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', {
turnId: 'turn-9',
item: { type: 'userMessage', id: 'item-0', text: 'late' }
})
)
expect(tap.rows[0]?.key).toBe('codex:thread-abc:turn-9:0')
})
it('keeps interleaved thread turns, items, and deltas separate', () => {
const { translator, tap } = translatorWith()
const child = (method: string, params: unknown): CodexStructuredSessionEvent => ({
type: 'notification',
const admission = translator.handle({
type: 'prompt',
sessionId: SESSION_ID,
threadId: 'thread-child',
method,
params
threadId: THREAD_ID,
method: CODEX_USER_INPUT_METHOD,
params: { questions: [{ id: 'q1' }] },
codexItemId: 'exec-1',
promptKey: 'untranslated-input'
})
translator.handle(TURN_STARTED)
translator.handle(child('turn/started', { threadId: 'thread-child', turnId: 'turn-child' }))
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-0', text: 'root' }
})
)
translator.handle(
child('item/completed', { item: { type: 'agentMessage', id: 'item-0', text: 'child' } })
)
translator.handle(child('turn/completed', { turnId: 'turn-child' }))
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-1', text: 'still root' }
})
)
expect(tap.rows.map((row) => row.key)).toEqual([
'codex:thread-abc:turn-1:0',
'codex:thread-child:turn-child:0',
'codex:thread-abc:turn-1:1'
])
expect(admission).toEqual({ accepted: false, reason: 'untranslated' })
expect(tap.rows).toEqual([])
expect(tap.bound).toEqual([])
expect(tap.publishes()).toBe(0)
})
it('checkpoints long streams geometrically and flushes the final snapshot', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', { item: { type: 'agentMessage', id: 'item-1', text: '' } })
)
for (let index = 0; index < 512; index += 1) {
translator.handle(notification('item/agentMessage/delta', { itemId: 'item-1', delta: 'x' }))
window.fire()
}
translator.flush()
expect(tap.rows.length).toBeLessThan(40)
expect(tap.rows.at(-1)?.body).toMatchObject({
blocks: [{ type: 'text', text: 'x'.repeat(512) }]
it('admits turn start publication across the hard watermark', async () => {
const bodies: AgentJournalItemBody[] = []
const publishes: string[] = []
const deferred = hardWatermarkDeferred()
const translator = createCodexJournalTranslator({
sink: deferred.sink,
primaryThreadId: () => THREAD_ID
})
})
it('folds long-running command output into one exec item and zero generic rows', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', {
item: { type: 'commandExecution', id: 'exec-1', command: 'long-task', status: 'inProgress' }
})
)
expect(translator.handle(TURN_STARTED)).toEqual({ accepted: true })
expect(deferred.state()).toMatchObject({ queuedOperations: 2, backpressured: true })
for (let index = 0; index < 512; index += 1) {
translator.handle(
notification('item/commandExecution/outputDelta', { itemId: 'exec-1', delta: 'x' })
)
window.fire()
}
translator.flush()
deferred.bind(deferredTarget(bodies, publishes))
await expect(deferred.lifecycleBarrier()).resolves.toEqual({ ok: true })
expect(new Set(tap.rows.map((row) => row.key))).toEqual(
new Set(['orca:codex-item%3Athread-abc%3Aexec-1'])
)
expect(tap.rows.every((row) => row.body.kind === 'tool-call')).toBe(true)
expect(tap.rows.length).toBeLessThan(40)
expect(tap.rows.at(-1)?.body).toMatchObject({
kind: 'tool-call',
output: { head: 'x'.repeat(512) }
})
})
it('folds reasoning and patch streams into their parent rows', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(notification('item/started', { item: { type: 'reasoning', id: 'r-1' } }))
translator.handle(
notification('item/reasoning/summaryTextDelta', { itemId: 'r-1', delta: 'thinking' })
)
translator.handle(
notification('item/started', {
item: { type: 'fileChange', id: 'patch-1', changes: [], status: 'inProgress' }
})
)
translator.handle(
notification('item/fileChange/patchUpdated', {
itemId: 'patch-1',
changes: [{ path: 'src/app.ts', kind: { type: 'update' }, diff: '@@ -1 +1 @@' }]
})
)
window.fire()
const reduced = new Map(tap.rows.map((row) => [row.key, row.body]))
expect(reduced.get('orca:codex-item%3Athread-abc%3Ar-1')).toEqual({
kind: 'status',
text: 'thinking'
})
expect(reduced.get('orca:codex-item%3Athread-abc%3Apatch-1')).toMatchObject({
kind: 'diff',
path: 'src/app.ts',
patch: { head: '@@ -1 +1 @@' }
})
})
it('publishes after every write so a subscriber never trails the journal', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', { item: { type: 'userMessage', id: 'item-0', text: 'hi' } })
)
expect(tap.publishes()).toBe(1)
})
it('releases a turn ordinal map when the turn completes', () => {
const spy = vi.spyOn(CodexTurnOrdinals.prototype, 'forgetTurn')
try {
const { translator } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
expect(spy).toHaveBeenCalledWith(THREAD_ID, TURN_ID)
} finally {
spy.mockRestore()
}
})
it('journals malformed item events but never malformed deltas', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(notification('item/completed', {}))
translator.handle(notification('item/agentMessage/delta', { delta: 'orphan' }))
window.fire()
expect(tap.rows.map((row) => row.body)).toEqual([
expect(bodies).toEqual([
expect.objectContaining({
kind: 'status',
providerFrame: expect.objectContaining({ kind: 'notification:item/completed' })
turnLifecycle: { turnId: TURN_ID, state: 'running' }
})
])
})
it('journals unknown notifications, server requests, and decoded provider frames', () => {
const { translator, tap } = translatorWith()
translator.handle(notification('future/notification', { value: 1 }))
translator.handle({
type: 'server-request',
sessionId: SESSION_ID,
threadId: THREAD_ID,
method: 'future/request',
params: { value: 2 }
})
translator.handle({
type: 'provider-frame',
sessionId: SESSION_ID,
threadId: THREAD_ID,
kind: 'frame:unclassified',
payload: { value: 3 }
})
expect(
tap.rows.map((row) => (row.body.kind === 'status' ? row.body.providerFrame?.kind : undefined))
).toEqual(['notification:future/notification', 'request:future/request', 'frame:unclassified'])
})
it('bounds generic rows per turn while keeping the suppression visible and countable', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
for (let index = 0; index < MAX_CODEX_GENERIC_ROWS_PER_TURN + 20; index += 1) {
translator.handle(notification('future/notification', { value: index }))
}
translator.handle(notification('item/future/outputDelta', { itemId: 'future', delta: 'x' }))
const generic = tap.rows.filter(
(row) => row.body.kind === 'status' && row.body.providerFrame !== undefined
)
expect(generic).toHaveLength(MAX_CODEX_GENERIC_ROWS_PER_TURN)
expect(generic[0]?.body).toMatchObject({
kind: 'status',
providerFrame: { kind: 'notification:future/notification' }
})
// The 20 capped frames reduce to ONE summary row whose count is exact, so
// suppressed provider activity is never invisible.
const summaries = new Map(
tap.rows
.filter((row) => row.key.includes('provider-frame-suppressed'))
.map((row) => [row.key, row.body])
)
expect(summaries.size).toBe(1)
expect([...summaries.values()][0]).toEqual({
kind: 'status',
text: '20 more provider notifications not shown for this turn'
})
expect(
tap.rows.some(
(row) =>
row.body.kind === 'status' &&
row.body.providerFrame?.kind === 'notification:item/future/outputDelta'
)
).toBe(false)
})
it('never lets the generic-row cap hide an error frame', () => {
const { translator, tap } = translatorWith()
translator.handle(TURN_STARTED)
for (let index = 0; index < MAX_CODEX_GENERIC_ROWS_PER_TURN + 3; index += 1) {
translator.handle(notification('future/notification', { value: index }))
}
translator.handle(notification('future/failure', { error: 'provider exploded' }))
expect(
tap.rows.some(
(row) =>
row.body.kind === 'status' &&
row.body.providerFrame?.kind === 'notification:future/failure'
)
).toBe(true)
})
it('keeps a fresh session timeline empty through startup and status notifications', () => {
const { translator, tap } = translatorWith()
translator.handle(notification('thread/started', { thread: { id: THREAD_ID } }))
for (let index = 0; index < 8; index += 1) {
translator.handle(
notification('mcpServer/startupStatus/updated', {
server: `server-${index}`,
status: 'starting'
})
)
}
translator.handle(notification('remoteControl/status/changed', { status: 'disabled' }))
const timeline = projectStructuredItemsToNativeChat(
tap.rows.map((row, index) => ({
itemId: row.key,
revision: 1,
sequence: index + 1,
observedAt: index + 1,
body: row.body
}))
)
expect(timeline).toEqual([])
})
it('projects only user and assistant content for a complete turn with hooks', () => {
const { translator, tap } = translatorWith()
translator.handle(notification('thread/started', { thread: { id: THREAD_ID } }))
translator.handle(notification('hook/started', { run: { id: 'hook-1', status: 'running' } }))
translator.handle(notification('account/rateLimits/updated', { rateLimits: { primary: null } }))
translator.handle(TURN_STARTED)
translator.handle(
notification('item/completed', {
item: { type: 'userMessage', id: 'item-0', text: 'hi' }
})
)
translator.handle(
notification('hook/completed', { run: { id: 'hook-1', status: 'completed' } })
)
translator.handle(
notification('item/completed', {
item: { type: 'agentMessage', id: 'item-1', text: 'hello' }
})
)
translator.handle(notification('turn/completed', { turn: { id: TURN_ID } }))
const timeline = projectStructuredItemsToNativeChat(
tap.rows.map((row, index) => ({
itemId: row.key,
revision: 1,
sequence: index + 1,
observedAt: index + 1,
body: row.body
}))
)
expect(timeline.map(({ role, blocks }) => ({ role, blocks }))).toEqual([
{ role: 'user', blocks: [{ type: 'text', text: 'hi' }] },
{ role: 'assistant', blocks: [{ type: 'text', text: 'hello' }] }
])
})
it('renders a system error carried by a suppressed status kind', () => {
const { translator, tap } = translatorWith()
translator.handle(
notification('thread/status/changed', {
threadId: THREAD_ID,
status: { type: 'systemError' }
})
)
const timeline = projectStructuredItemsToNativeChat(
tap.rows.map((row, index) => ({
itemId: row.key,
revision: 1,
sequence: index + 1,
observedAt: index + 1,
body: row.body
}))
)
expect(timeline).toEqual([
expect.objectContaining({
role: 'system',
blocks: [
expect.objectContaining({
providerFrame: expect.objectContaining({
kind: 'notification:thread/status/changed'
})
})
]
})
])
})
it('writes nothing more after dispose', () => {
const { translator, tap, window } = translatorWith()
translator.handle(TURN_STARTED)
translator.handle(
notification('item/started', { item: { type: 'agentMessage', id: 'item-1', text: '' } })
)
translator.handle(notification('item/agentMessage/delta', { itemId: 'item-1', delta: 'gone' }))
translator.dispose()
window.fire()
expect(tap.rows).toEqual([])
expect(publishes).toHaveLength(1)
})
})
@@ -1,335 +1,240 @@
import type { AgentJournalItemIdentity } from '../../shared/agent-session-journal-types'
import { agentJournalItemKey } from '../../shared/agent-session-journal-item-key'
import type { AgentSessionDeltaCoalescerDeps } from '../native-chat/agent-session-wire/agent-session-delta-coalescer'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import { unhandledProviderFrameJournalItem } from '../native-chat/agent-session-wire/unhandled-provider-frame'
import type { CodexStructuredSessionEvent } from './codex-structured-session-adapter'
import { CodexJournalGenericFrames } from './codex-structured-journal-generic-frames'
import { CodexJournalItems } from './codex-structured-journal-items'
import { CodexJournalPrompts } from './codex-structured-journal-prompts'
import {
codexItemIdentity,
codexJournalItem,
CodexTurnOrdinals,
readCodexThreadItem
} from './codex-structured-item-translation'
CODEX_JOURNAL_ADMITTED,
type CodexJournalTranslationAdmission,
type CodexJournalTranslator,
type CodexJournalTranslatorDeps
} from './codex-structured-journal-contracts'
import {
codexStructuredItemKey,
createCodexStructuredItemStreams
} from './codex-structured-item-streams'
settleCodexJournalSession,
settleCodexJournalTurn,
settleCodexOversizedNotification
} from './codex-structured-journal-settlement'
import { restoreCodexJournalThread } from './codex-structured-journal-translation-restore'
import { CodexJournalActiveTurns } from './codex-structured-journal-translation-turn-state'
import { publishCodexTurnLifecycle } from './codex-structured-journal-translation-turns'
import {
codexApprovalItem,
codexPromptIdentity,
codexQuestionItems
} from './codex-structured-prompt-items'
import { CODEX_USER_INPUT_METHOD } from './codex-structured-prompt-replies'
readCodexJournalRecord,
readCodexJournalString
} from './codex-structured-journal-translation-values'
import { readCodexTurnId } from './codex-structured-thread-facts'
// The one place Codex events become journal rows.
//
// Every durable decision lives here rather than in the adapter: the adapter
// knows the protocol, this knows what a user is owed after a reconnect. It is
// per-session and per-acquisition — a new lease gets a new translator and a new
// sink, so a superseded child cannot keep writing.
export const MAX_CODEX_GENERIC_ROWS_PER_TURN = 8
export type CodexJournalTranslatorDeps = {
sink: StructuredAgentSessionEventSink
/** Points an answered journal item back at the live Codex request. */
bindPromptItemId?: (journalItemId: string, threadId: string, promptKey: string) => void
primaryThreadId?: () => string | null
coalesceMs?: number
schedule?: AgentSessionDeltaCoalescerDeps['schedule']
}
export type CodexJournalTranslator = {
handle: (event: CodexStructuredSessionEvent) => void
restoreThread: (threadId: string, thread: Record<string, unknown>) => void
flush: () => void
dispose: () => void
}
function readRecord(value: unknown): Record<string, unknown> {
return typeof value === 'object' && value !== null ? (value as Record<string, unknown>) : {}
}
function readString(source: Record<string, unknown>, key: string): string | null {
const value = source[key]
return typeof value === 'string' && value.length > 0 ? value : null
}
export type {
CodexJournalTranslationAdmission,
CodexJournalTranslator,
CodexJournalTranslatorDeps
} from './codex-structured-journal-contracts'
export {
MAX_CODEX_ACTIVE_ITEMS,
MAX_CODEX_DETAIL_BYTES,
MAX_CODEX_DETAIL_ENTRIES,
MAX_CODEX_GENERIC_BOOKKEEPING_BYTES,
MAX_CODEX_GENERIC_BOOKKEEPING_ENTRIES,
MAX_CODEX_GENERIC_ROWS_PER_TURN,
MAX_CODEX_GENERIC_TURN_BUCKETS,
MAX_CODEX_IDENTITY_ENTRIES,
MAX_CODEX_PENDING_PROMPTS
} from './codex-structured-journal-limits'
export function createCodexJournalTranslator(
deps: CodexJournalTranslatorDeps
): CodexJournalTranslator {
const ordinals = new CodexTurnOrdinals()
/** Identity assigned when an item was announced, reused by its deltas and by
* its completion so all three upsert one row. */
const identities = new Map<string, AgentJournalItemIdentity>()
/** What each announced item is, so an approval can name what it approves. */
const details = new Map<string, string>()
/** Turns announced by the provider and not yet closed. */
const currentTurnIds = new Map<string, Set<string>>()
const genericRowsByTurn = new Map<string, number>()
const suppressedRowsByTurn = new Map<string, number>()
let fallbackSequence = 0
const currentTurnIdFor = (threadId: string): string | null =>
[...(currentTurnIds.get(threadId) ?? [])].at(-1) ?? null
const rememberTurn = (threadId: string, turnId: string): void => {
currentTurnIds.set(threadId, new Set([...(currentTurnIds.get(threadId) ?? []), turnId]))
}
const forgetTurn = (threadId: string, turnId: string): void => {
const active = currentTurnIds.get(threadId)
active?.delete(turnId)
if (!active?.size) {
currentTurnIds.delete(threadId)
}
}
const appendUnhandled = (kind: string, payload: unknown, threadId = 'session'): void => {
const translated = unhandledProviderFrameJournalItem('codex', kind, payload)
if (!translated) {
return
}
const turnId = readCodexTurnId(payload) ?? currentTurnIdFor(threadId) ?? 'outside-turn'
const bucket = `${encodeURIComponent(threadId)}:${encodeURIComponent(turnId)}`
const rowCount = genericRowsByTurn.get(bucket) ?? 0
// The cap bounds noise, never evidence: an error frame is always journaled,
// and capped frames stay countable through one summary row per turn.
const capped =
rowCount >= MAX_CODEX_GENERIC_ROWS_PER_TURN && translated.classification !== 'error-surface'
if (capped) {
const suppressed = (suppressedRowsByTurn.get(bucket) ?? 0) + 1
suppressedRowsByTurn.set(bucket, suppressed)
deps.sink.appendItem(
{ provider: 'orca', clientMessageId: `provider-frame-suppressed:codex:${bucket}` },
{
kind: 'status',
text: `${suppressed} more provider notification${suppressed === 1 ? '' : 's'} not shown for this turn`
}
)
deps.sink.publish()
return
}
genericRowsByTurn.set(bucket, rowCount + 1)
fallbackSequence += 1
deps.sink.appendItem(
{ provider: 'orca', clientMessageId: `provider-frame:codex:${fallbackSequence}` },
translated.body,
translated.blobs
)
deps.sink.publish()
}
const publishTurnLifecycle = (
sessionId: string,
threadId: string,
turnId: string,
state: 'running' | 'completed'
): void => {
if (deps.primaryThreadId?.() !== threadId) {
return
}
const identity = {
provider: 'legacy' as const,
agent: 'codex' as const,
sessionId,
recordId: `turn-lifecycle:${turnId}`
}
if (state === 'completed') {
deps.sink.appendTombstone(identity)
} else {
deps.sink.appendItem(identity, {
kind: 'status',
text: 'Codex is working…',
turnLifecycle: { turnId, state }
})
}
deps.sink.publish()
}
const identityFor = (
threadId: string,
turnId: string | null,
item: { type: string; id: string }
): AgentJournalItemIdentity => {
const key = codexStructuredItemKey(threadId, item.id)
const existing = identities.get(key)
if (existing) {
return existing
}
const identity = codexItemIdentity({ threadId, turnId, item, ordinals })
identities.set(key, identity)
return identity
}
const streams = createCodexStructuredItemStreams({
sink: deps.sink,
coalesceMs: deps.coalesceMs,
schedule: deps.schedule,
identityFor: (threadId, params, item) => {
const turnId = readCodexTurnId(params) ?? currentTurnIdFor(threadId)
return identityFor(threadId, turnId, item)
}
})
const handleItemEvent = (event: {
threadId: string
method: string
params: unknown
}): boolean => {
const params = readRecord(event.params)
const item = readCodexThreadItem(params.item)
if (!item) {
return false
}
const turnId = readCodexTurnId(event.params) ?? currentTurnIdFor(event.threadId)
const identity = identityFor(event.threadId, turnId, item)
const translated = codexJournalItem(item)
const command = readString(item, 'command')
if (command) {
details.set(codexStructuredItemKey(event.threadId, item.id), command)
}
if (event.method === 'item/completed') {
// The completed body is authoritative; the coalesced text is now stale.
streams.forget(event.threadId, item.id)
} else {
streams.track(event.threadId, item, identity)
}
if (!translated.body) {
return true
}
deps.sink.appendItem(identity, translated.body, translated.blobs)
deps.sink.publish()
return true
}
// The row is keyed by the prompt and the announced command is looked up by the
// tool item, because one item can ask more than once.
const handlePrompt = (event: {
threadId: string
method: string
params: unknown
codexItemId: string
promptKey: string
}): void => {
if (event.method === CODEX_USER_INPUT_METHOD) {
for (const question of codexQuestionItems({
threadId: event.threadId,
promptKey: event.promptKey,
params: event.params
})) {
deps.sink.appendItem(question.identity, question.body)
deps.bindPromptItemId?.(
agentJournalItemKey(question.identity),
event.threadId,
event.promptKey
)
}
deps.sink.publish()
return
}
const identity = codexPromptIdentity({
threadId: event.threadId,
promptKey: event.promptKey
})
deps.sink.appendItem(
identity,
codexApprovalItem({
method: event.method,
params: event.params,
detail: details.get(codexStructuredItemKey(event.threadId, event.codexItemId)) ?? null
})
)
deps.bindPromptItemId?.(agentJournalItemKey(identity), event.threadId, event.promptKey)
deps.sink.publish()
}
const activeTurns = new CodexJournalActiveTurns()
const genericFrames = new CodexJournalGenericFrames(deps, (threadId) =>
activeTurns.current(threadId)
)
const items = new CodexJournalItems(
deps,
(threadId) => activeTurns.current(threadId),
(threadId, turnId) => genericFrames.suppress(threadId, turnId)
)
const prompts = new CodexJournalPrompts(deps, (threadId, itemId) =>
items.detailFor(threadId, itemId)
)
const flushStreams = (): CodexJournalTranslationAdmission =>
items.streams.flush() ? CODEX_JOURNAL_ADMITTED : { accepted: false, reason: 'backpressure' }
return {
restoreThread: (threadId, thread) => {
const turns = Array.isArray(thread.turns) ? thread.turns : []
for (const rawTurn of turns) {
const turn = readRecord(rawTurn)
const turnId = readString(turn, 'id')
if (!turnId) {
continue
}
currentTurnIds.set(threadId, new Set([turnId]))
for (const item of Array.isArray(turn.items) ? turn.items : []) {
handleItemEvent({ threadId, method: 'item/completed', params: { turnId, item } })
}
currentTurnIds.delete(threadId)
ordinals.forgetTurn(threadId, turnId)
}
streams.flush()
},
restoreThread: (threadId, thread) =>
restoreCodexJournalThread({
threadId,
thread,
currentTurnIds: activeTurns.byThread,
ordinals: items.ordinals,
handleItem: (event) => {
const translated = items.handle(event)
return translated.handled
? translated.admission
: { accepted: false, reason: 'untranslated' }
},
flush: items.streams.flush
}),
handle: (event) => {
if (event.type === 'ended') {
streams.flush()
for (const [threadId, turnIds] of currentTurnIds) {
for (const turnId of turnIds) {
publishTurnLifecycle(event.sessionId, threadId, turnId, 'completed')
ordinals.forgetTurn(threadId, turnId)
}
const streamAdmission = flushStreams()
if (!streamAdmission.accepted) {
return streamAdmission
}
currentTurnIds.clear()
return
const suppressionAdmission = genericFrames.flush()
if (!suppressionAdmission.accepted) {
return suppressionAdmission
}
const admission = settleCodexJournalSession({
event,
sink: deps.sink,
streams: items.streams,
activeItems: items.activeItems,
pendingPrompts: prompts.pending,
currentTurnIds: activeTurns.byThread,
primaryThreadId: deps.primaryThreadId?.() ?? null,
ordinals: items.ordinals
})
if (!admission.accepted) {
return admission
}
items.activeItems.clear()
prompts.pending.clear()
activeTurns.clear()
return CODEX_JOURNAL_ADMITTED
}
if (
event.type === 'notification' &&
streams.handle(event.threadId, event.method, event.params)
) {
return
if (event.type === 'notification') {
const streamResult = items.streams.handle(event.threadId, event.method, event.params)
if (streamResult.handled) {
return streamResult.admission
}
}
const streamAdmission = flushStreams()
if (!streamAdmission.accepted) {
return streamAdmission
}
// Lifecycle bypass: nothing may be journaled ahead of the text it follows.
streams.flush()
if (event.type === 'prompt') {
handlePrompt(event)
return
const suppressionAdmission = genericFrames.flush()
return suppressionAdmission.accepted ? prompts.handle(event) : suppressionAdmission
}
if (event.type === 'server-request') {
appendUnhandled(`request:${event.method}`, event.params, event.threadId)
return
return genericFrames.appendUnhandled(
`request:${event.method}`,
event.params,
event.threadId
)
}
if (event.type === 'provider-frame') {
appendUnhandled(event.kind, event.payload, event.threadId)
return
const settlement = settleOversizedNotification(event)
if (settlement && !settlement.accepted) {
return settlement
}
return genericFrames.appendUnhandled(event.kind, event.payload, event.threadId)
}
if (event.method === 'turn/started') {
const turnId = readCodexTurnId(event.params)
if (turnId) {
rememberTurn(event.threadId, turnId)
publishTurnLifecycle(event.sessionId, event.threadId, turnId, 'running')
}
return
return startTurn(event)
}
if (event.method === 'turn/completed') {
const turnId = readCodexTurnId(event.params) ?? currentTurnIdFor(event.threadId)
if (turnId) {
publishTurnLifecycle(event.sessionId, event.threadId, turnId, 'completed')
ordinals.forgetTurn(event.threadId, turnId)
forgetTurn(event.threadId, turnId)
}
// A later item without its own turn id falls back to another active
// turn, if one exists; completed turns are never adopted again.
return
return completeTurn(event)
}
if (event.method === 'item/started' || event.method === 'item/completed') {
if (!handleItemEvent(event)) {
appendUnhandled(`notification:${event.method}`, event.params, event.threadId)
}
return
const translated = items.handle(event)
return translated.handled
? translated.admission
: genericFrames.appendUnhandled(
`notification:${event.method}`,
event.params,
event.threadId
)
}
appendUnhandled(`notification:${event.method}`, event.params, event.threadId)
return genericFrames.appendUnhandled(
`notification:${event.method}`,
event.params,
event.threadId
)
},
resolvePrompt: (journalItemId) => prompts.resolve(journalItemId),
flush: () => {
items.streams.flush()
genericFrames.flush()
},
flush: streams.flush,
dispose: () => {
streams.dispose()
identities.clear()
details.clear()
currentTurnIds.clear()
genericRowsByTurn.clear()
suppressedRowsByTurn.clear()
items.dispose()
prompts.dispose()
genericFrames.dispose()
activeTurns.clear()
}
}
function settleOversizedNotification(event: {
sessionId: string
threadId: string
kind: string
payload: unknown
}): CodexJournalTranslationAdmission | null {
if (event.kind !== 'frame:oversized-notification') {
return null
}
const method = readCodexJournalString(readCodexJournalRecord(event.payload), 'method')
return method
? settleCodexOversizedNotification({
sessionId: event.sessionId,
threadId: event.threadId,
method,
sink: deps.sink,
streams: items.streams,
activeItems: items.activeItems
})
: null
}
function startTurn(event: {
sessionId: string
threadId: string
params: unknown
}): CodexJournalTranslationAdmission {
const turnId = readCodexTurnId(event.params)
if (!turnId) {
return CODEX_JOURNAL_ADMITTED
}
if (!activeTurns.canRemember(event.threadId, turnId)) {
return { accepted: false, reason: 'backpressure' }
}
const admission = publishCodexTurnLifecycle({
sink: deps.sink,
primaryThreadId: deps.primaryThreadId?.() ?? null,
sessionId: event.sessionId,
threadId: event.threadId,
turnId,
state: 'running'
})
if (admission.accepted) {
activeTurns.remember(event.threadId, turnId)
}
return admission
}
function completeTurn(event: {
sessionId: string
threadId: string
params: unknown
}): CodexJournalTranslationAdmission {
const suppressionAdmission = genericFrames.flush()
if (!suppressionAdmission.accepted) {
return suppressionAdmission
}
const turnId = readCodexTurnId(event.params) ?? activeTurns.current(event.threadId)
if (!turnId) {
return CODEX_JOURNAL_ADMITTED
}
const admission = settleCodexJournalTurn({
sink: deps.sink,
sessionId: event.sessionId,
threadId: event.threadId,
turnId,
streams: items.streams,
activeItems: items.activeItems
})
if (admission.accepted) {
items.ordinals.forgetTurn(event.threadId, turnId)
activeTurns.forget(event.threadId, turnId)
}
return admission
}
}
@@ -0,0 +1,161 @@
import type { CodexAppServerConnection } from './codex-app-server-connection'
import type { CodexJournalTranslationAdmission } from './codex-structured-journal-translation'
import type { CodexSession } from './codex-structured-session-state'
const MAX_RETRY_EVENTS = 256
const MAX_RETRY_BYTES = 8 * 1024 * 1024
const RETRY_DELAY_MS = 25
type PendingNotification = { method: string; params: unknown; bytes: number }
type RetryState = {
connection: CodexAppServerConnection
events: PendingNotification[]
bytes: number
timer: ReturnType<typeof setTimeout> | null
running: boolean
failed: boolean
}
export function createCodexStructuredNotificationRetry(deps: {
sessionFor: (sessionId: string) => CodexSession | undefined
translate: (
sessionId: string,
session: CodexSession,
method: string,
params: unknown
) => CodexJournalTranslationAdmission
}) {
const states = new Map<string, RetryState>()
const retry = (sessionId: string, connection: CodexAppServerConnection): void => {
const state = states.get(sessionId)
if (!state || state.connection !== connection || state.running) {
return
}
if (state.timer) {
clearTimeout(state.timer)
state.timer = null
}
state.running = true
try {
while (state.events.length > 0) {
const pending = state.events[0]
if (!pending) {
break
}
const session = deps.sessionFor(sessionId)
if (!session || session.connection !== connection || session.ended) {
fail(sessionId, state, 'notification retry owner is no longer live')
break
}
const admission = deps.translate(sessionId, session, pending.method, pending.params)
if (!admission.accepted) {
if (admission.reason === 'backpressure') {
state.timer = setTimeout(() => {
state.timer = null
retry(sessionId, connection)
}, RETRY_DELAY_MS)
state.timer.unref?.()
} else {
fail(sessionId, state, `notification admission failed (${admission.reason})`)
}
break
}
state.events.shift()
state.bytes = Math.max(0, state.bytes - pending.bytes)
}
if (state.events.length === 0) {
states.delete(sessionId)
}
} finally {
state.running = false
}
}
const fail = (sessionId: string, state: RetryState, reason: string): void => {
if (state.failed) {
return
}
state.failed = true
if (state.timer) {
clearTimeout(state.timer)
state.timer = null
}
// The queue is no longer replayable. Drop it explicitly, release the read
// pause, and enter the adapter's generation-checked unexpected-exit seam.
state.events.length = 0
state.bytes = 0
state.connection.resumeReading?.()
states.delete(sessionId)
const session = deps.sessionFor(sessionId)
if (session?.connection === state.connection) {
void session.forceCloseUnexpected?.(new Error(reason))
}
}
const enqueue = (
sessionId: string,
connection: CodexAppServerConnection,
method: string,
params: unknown
): void => {
const bytes = Buffer.byteLength(JSON.stringify({ method, params }), 'utf8')
let state = states.get(sessionId)
if (!state || state.connection !== connection) {
state = { connection, events: [], bytes: 0, timer: null, running: false, failed: false }
states.set(sessionId, state)
}
if (state.events.length >= MAX_RETRY_EVENTS || state.bytes + bytes > MAX_RETRY_BYTES) {
// A bounded queue cannot retain more traffic. Fail it truthfully so the
// provider's generation enters host recovery instead of stranding a pause.
fail(sessionId, state, 'notification retry queue overflow')
return
}
state.events.push({ method, params, bytes })
state.bytes += bytes
connection.pauseReading?.()
}
return {
handle: (
sessionId: string,
method: string,
params: unknown
): CodexJournalTranslationAdmission => {
const session = deps.sessionFor(sessionId)
if (!session) {
return { accepted: true }
}
const state = states.get(sessionId)
if (state && state.events.length > 0) {
enqueue(sessionId, state.connection, method, params)
retry(sessionId, state.connection)
return { accepted: false, reason: 'backpressure' }
}
const admission = deps.translate(sessionId, session, method, params)
if (!admission.accepted) {
enqueue(sessionId, session.connection, method, params)
retry(sessionId, session.connection)
}
return admission
},
retry,
clear: (sessionId: string, connection: CodexAppServerConnection | null): void => {
const state = states.get(sessionId)
if (!state || (connection && state.connection !== connection)) {
return
}
if (state.timer) {
clearTimeout(state.timer)
}
state.events.length = 0
state.bytes = 0
state.connection.resumeReading?.()
states.delete(sessionId)
}
}
}
export type CodexStructuredNotificationRetry = ReturnType<
typeof createCodexStructuredNotificationRetry
>
@@ -10,6 +10,7 @@ import {
CODEX_FILE_CHANGE_APPROVAL_METHOD,
encodeCodexQuestionOptionId
} from './codex-structured-prompt-replies'
import { MAX_JOURNAL_LIFECYCLE_BATCH_BYTES } from '../native-chat/agent-session-journal/journal-row-schema'
const THREAD_ID = 'thread-abc'
const CODEX_ITEM_ID = 'item-4'
@@ -91,6 +92,17 @@ describe('codex approval items', () => {
}).detail
).toBe('"/outside"')
})
it('bounds large approval details before they can enter lifecycle settlement', () => {
const item = codexApprovalItem({
method: CODEX_COMMAND_APPROVAL_METHOD,
params: { command: 'x'.repeat(2_000_000) },
detail: null
})
expect(item.detail).toContain('output truncated')
expect(Buffer.byteLength(JSON.stringify(item), 'utf8')).toBeLessThan(32 * 1024)
})
})
describe('codex question items', () => {
@@ -171,6 +183,36 @@ describe('codex question items', () => {
})
})
it('bounds question text, options, labels, and prompt identity components', () => {
const longQuestionId = 'question-id-'.repeat(500)
const longLabel = 'option '.repeat(5_000)
const items = codexQuestionItems({
threadId: 'thread-'.repeat(500),
promptKey: 'prompt-'.repeat(500),
params: {
questions: [
{
id: longQuestionId,
question: 'question '.repeat(500_000),
options: Array.from({ length: 80 }, () => ({ label: longLabel }))
}
]
}
})
const item = items[0]
if (!item) {
throw new Error('expected a bounded question item')
}
expect(item.body.question).toContain('output truncated')
expect(item.body.options).toHaveLength(64)
expect(item.body.options[0]?.label).toContain('output truncated')
expect(Buffer.byteLength(item.body.options[0]?.id ?? '', 'utf8')).toBeLessThan(1024)
expect(
Buffer.byteLength(JSON.stringify({ identity: item.identity, body: item.body }), 'utf8')
).toBeLessThan(MAX_JOURNAL_LIFECYCLE_BATCH_BYTES)
})
it('keys an approval without a question id', () => {
expect(codexPromptIdentity({ threadId: THREAD_ID, promptKey: CODEX_ITEM_ID })).toEqual({
provider: 'orca',
@@ -4,11 +4,16 @@ import type {
AgentJournalPromptOption,
AgentJournalQuestionItem
} from '../../shared/agent-session-journal-types'
import {
boundInlineText,
DEFAULT_JOURNAL_PAYLOAD_LIMITS
} from '../native-chat/agent-session-journal/journal-payload-bounds'
import {
CODEX_APPROVAL_DECISIONS,
CODEX_COMMAND_APPROVAL_METHOD,
CODEX_FILE_CHANGE_APPROVAL_METHOD,
encodeCodexQuestionOptionId,
codexJournalPromptIdPart,
encodeCodexJournalQuestionOptionId,
type CodexApprovalDecision
} from './codex-structured-prompt-replies'
@@ -33,6 +38,10 @@ const PENDING = {
resolvedAt: null
} as const
const MAX_CODEX_PROMPT_QUESTIONS = 64
const MAX_CODEX_PROMPT_OPTIONS = 64
const PROMPT_OPTION_LIMITS = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, inlineHeadBytes: 1024 }
function readParams(params: unknown): Record<string, unknown> {
return typeof params === 'object' && params !== null ? (params as Record<string, unknown>) : {}
}
@@ -42,6 +51,14 @@ function readString(source: Record<string, unknown>, key: string): string | null
return typeof value === 'string' && value.length > 0 ? value : null
}
function boundPromptText(value: string): string {
return boundInlineText(value, DEFAULT_JOURNAL_PAYLOAD_LIMITS).text
}
function boundPromptOptionLabel(value: string): string {
return boundInlineText(value, PROMPT_OPTION_LIMITS).text
}
/**
* Codex offers a per-request decision set, so the options come off the request
* when it names them. Falling back to the full set is deliberate: a build that
@@ -75,12 +92,16 @@ export function codexApprovalItem(input: {
: input.method === CODEX_COMMAND_APPROVAL_METHOD
? 'Run a command?'
: 'Approve this action?',
detail: approvalDetail(params) ?? input.detail,
detail: boundNullablePromptText(approvalDetail(params) ?? input.detail),
options: codexApprovalOptions(input.params),
resolution: { ...PENDING }
}
}
function boundNullablePromptText(value: string | null): string | null {
return value === null ? null : boundPromptText(value)
}
function approvalDetail(params: Record<string, unknown>): string | null {
const command = params.command
if (typeof command === 'string' && command.length > 0) {
@@ -119,7 +140,7 @@ export function codexQuestionItems(input: {
return []
}
const items: CodexQuestionItem[] = []
for (const entry of questions) {
for (const entry of questions.slice(0, MAX_CODEX_PROMPT_QUESTIONS)) {
const question = readParams(entry)
const questionId = readString(question, 'id')
const prompt = readString(question, 'question') ?? readString(question, 'header')
@@ -131,9 +152,11 @@ export function codexQuestionItems(input: {
identity: codexPromptIdentity({ ...input, questionId }),
body: {
kind: 'question',
question: prompt,
question: boundPromptText(prompt),
options: questionOptions(question, questionId),
...(questionAllowsFreeText(question) ? { freeTextQuestionId: questionId } : {}),
...(questionAllowsFreeText(question)
? { freeTextQuestionId: codexJournalPromptIdPart(questionId) }
: {}),
resolution: { ...PENDING }
}
})
@@ -160,12 +183,18 @@ function questionOptions(
}
const mapped: AgentJournalPromptOption[] = []
for (const entry of options) {
if (mapped.length >= MAX_CODEX_PROMPT_OPTIONS) {
break
}
const option = readParams(entry)
const label = readString(option, 'label')
if (label !== null && option.isOther !== true) {
// The option id has to name its question: Codex's reply is a map keyed by
// question id, and the client only ever hands back an option id.
mapped.push({ id: encodeCodexQuestionOptionId(questionId, label), label })
mapped.push({
id: encodeCodexJournalQuestionOptionId(questionId, label),
label: boundPromptOptionLabel(label)
})
}
}
return mapped
@@ -180,9 +209,11 @@ export function codexPromptIdentity(input: {
promptKey: string
questionId?: string
}): AgentJournalItemIdentity {
const suffix = input.questionId ? `:${input.questionId}` : ''
const suffix = input.questionId ? `:${codexJournalPromptIdPart(input.questionId)}` : ''
const threadId = codexJournalPromptIdPart(input.threadId)
const promptKey = codexJournalPromptIdPart(input.promptKey)
return {
provider: 'orca',
clientMessageId: `codex-prompt:${input.threadId}:${input.promptKey}${suffix}`
clientMessageId: `codex-prompt:${threadId}:${promptKey}${suffix}`
}
}
@@ -2,7 +2,10 @@ import { describe, expect, it } from 'vitest'
import {
applyCodexPromptAnswer,
CodexPromptRegistry,
MAX_CODEX_PROMPT_REGISTRY_ENTRIES,
codexJournalPromptIdPart,
decodeCodexQuestionOptionId,
encodeCodexJournalQuestionOptionId,
encodeCodexQuestionOptionId
} from './codex-structured-prompt-replies'
@@ -36,6 +39,28 @@ describe('codex question option ids', () => {
it('reads nothing from an id with no separator', () => {
expect(decodeCodexQuestionOptionId('accept')).toBeNull()
})
it('bounds journal option ids while preserving the exact Codex answer', () => {
const longQuestionId = 'q'.repeat(5_000)
const longAnswer = 'answer '.repeat(5_000)
const optionId = encodeCodexJournalQuestionOptionId(longQuestionId, longAnswer)
const registry = new CodexPromptRegistry()
const prompt = registry.register({
id: 9,
method: 'item/tool/requestUserInput',
params: {
itemId: 'codex-item-1',
threadId: 'thread-1',
questions: [{ id: longQuestionId, options: [{ label: longAnswer }] }]
}
})
expect(Buffer.byteLength(optionId, 'utf8')).toBeLessThan(1024)
expect(codexJournalPromptIdPart(longQuestionId)).not.toBe(longQuestionId)
expect(applyCodexPromptAnswer(prompt as NonNullable<typeof prompt>, optionId)).toEqual({
answers: { [longQuestionId]: { answers: [longAnswer] } }
})
})
})
describe('CodexPromptRegistry', () => {
@@ -100,6 +125,22 @@ describe('CodexPromptRegistry', () => {
expect(registry.find('journal-child')?.requestId).toBe(2)
expect(registry.find('item-2')).toBeNull()
})
it('keeps a journal-bound pending prompt answerable after the lookup window evicts it', () => {
const registry = new CodexPromptRegistry()
const first = registry.register(userInputRequest(['q1']))
registry.bindJournalItemId('journal-first', 'thread-1', 'codex-item-1')
for (let index = 0; index <= MAX_CODEX_PROMPT_REGISTRY_ENTRIES; index += 1) {
registry.register({
id: index + 10,
method: 'item/commandExecution/requestApproval',
params: { itemId: `item-${index}`, threadId: 'thread-1' }
})
}
expect(registry.find('journal-first')).toBe(first)
})
})
describe('applyCodexPromptAnswer', () => {
@@ -138,4 +179,28 @@ describe('applyCodexPromptAnswer', () => {
answers: { q1: { answers: ['second'] } }
})
})
it('refuses question and option collections that exceed bounded live state', () => {
const registry = new CodexPromptRegistry()
const tooManyQuestions = registry.register(
userInputRequest(Array.from({ length: 65 }, (_, index) => `q${index}`))
)
expect(tooManyQuestions).toBeNull()
const hugeOptionRequest = {
id: 10,
method: 'item/tool/requestUserInput',
params: {
itemId: 'item-huge-options',
threadId: 'thread-1',
questions: [
{ id: 'q1', options: Array.from({ length: 257 }, (_, i) => ({ label: `option-${i}` })) }
]
}
}
expect(registry.register(hugeOptionRequest)).toBeNull()
const hugeQuestionId = 'x'.repeat(32 * 1024 + 1)
expect(registry.register(userInputRequest([hugeQuestionId]))).toBeNull()
})
})
+143 -20
View File
@@ -1,4 +1,20 @@
import type { CodexAppServerConnection } from './codex-app-server-connection'
import {
CODEX_PROMPT_MAX_ANSWER_BYTES,
MAX_CODEX_PROMPT_JOURNAL_BINDINGS,
MAX_CODEX_PROMPT_REGISTRY_BYTES,
MAX_CODEX_PROMPT_REGISTRY_ENTRIES,
codexJournalPromptIdPart,
readQuestionIds,
readQuestionOptionAnswers
} from './codex-prompt-registry-bounds'
export {
codexJournalPromptIdPart,
MAX_CODEX_PROMPT_REGISTRY_ENTRIES,
MAX_CODEX_PROMPT_JOURNAL_BINDINGS,
MAX_CODEX_PROMPT_REGISTRY_BYTES,
encodeCodexJournalQuestionOptionId
} from './codex-prompt-registry-bounds'
// Codex asks for approvals and tool input by sending JSON-RPC REQUESTS back to
// Orca, and the turn blocks until each one is answered. The journal answers them
@@ -26,6 +42,9 @@ export type CodexPendingPrompt = {
promptKey: string
/** One entry per question for a user-input request; empty for an approval. */
questionIds: readonly string[]
/** Journal-facing ids can be bounded; replies still need Codex's exact ids. */
questionIdAliases: ReadonlyMap<string, string>
optionAnswers: ReadonlyMap<string, { questionId: string; answer: string }>
answers: Map<string, string>
}
@@ -60,16 +79,6 @@ function readString(params: unknown, key: string): string | null {
return typeof value === 'string' && value.length > 0 ? value : null
}
function readQuestionIds(params: unknown): string[] {
const questions = (params as { questions?: unknown } | null)?.questions
if (!Array.isArray(questions)) {
return []
}
return questions
.map((question) => (question as { id?: unknown })?.id)
.filter((id): id is string => typeof id === 'string' && id.length > 0)
}
export function isCodexPromptMethod(method: string): boolean {
return (
method === CODEX_COMMAND_APPROVAL_METHOD ||
@@ -88,6 +97,62 @@ export class CodexPromptRegistry {
private readonly byAddress = new Map<string, CodexPendingPrompt>()
/** Journal item id to thread-scoped prompt address. */
private readonly journalItemIds = new Map<string, string>()
/** Bound prompts survive LRU eviction of the lookup window until answered. */
private readonly boundPrompts = new Map<string, CodexPendingPrompt>()
get sizes(): { prompts: number; journalBindings: number } {
return { prompts: this.byAddress.size, journalBindings: this.journalItemIds.size }
}
get bytes(): number {
return this.retainedPromptBytes()
}
private promptBytes(prompt: CodexPendingPrompt): number {
let bytes = 0
for (const value of [
prompt.threadId,
prompt.turnId ?? '',
prompt.codexItemId,
prompt.promptKey
]) {
bytes += Buffer.byteLength(value, 'utf8')
}
for (const id of prompt.questionIds) {
bytes += Buffer.byteLength(id, 'utf8')
}
for (const entry of prompt.optionAnswers.values()) {
bytes += Buffer.byteLength(entry.questionId, 'utf8') + Buffer.byteLength(entry.answer, 'utf8')
}
for (const value of prompt.answers.values()) {
bytes += Buffer.byteLength(value, 'utf8')
}
return bytes
}
private retainedPromptBytes(): number {
const prompts = new Set([...this.byAddress.values(), ...this.boundPrompts.values()])
return [...prompts].reduce((total, prompt) => total + this.promptBytes(prompt), 0)
}
private trim(): void {
while (this.byAddress.size > MAX_CODEX_PROMPT_REGISTRY_ENTRIES) {
const oldest = this.byAddress.values().next().value as CodexPendingPrompt | undefined
if (!oldest) {
break
}
const address = this.address(oldest.threadId, oldest.promptKey)
this.byAddress.delete(address)
}
while (this.journalItemIds.size > MAX_CODEX_PROMPT_JOURNAL_BINDINGS) {
const oldest = this.journalItemIds.keys().next().value as string | undefined
if (!oldest) {
break
}
this.journalItemIds.delete(oldest)
this.boundPrompts.delete(oldest)
}
}
private address(threadId: string, promptKey: string): string {
return `${encodeURIComponent(threadId)}:${encodeURIComponent(promptKey)}`
@@ -105,6 +170,18 @@ export class CodexPromptRegistry {
if (!isCodexPromptMethod(request.method) || !codexItemId || !threadId) {
return null
}
const questionIds =
request.method === CODEX_USER_INPUT_METHOD ? readQuestionIds(request.params) : []
if (questionIds === null) {
return null
}
const optionAnswers =
request.method === CODEX_USER_INPUT_METHOD
? readQuestionOptionAnswers(request.params)
: new Map<string, { questionId: string; answer: string }>()
if (optionAnswers === null) {
return null
}
const prompt: CodexPendingPrompt = {
requestId: request.id,
method: request.method,
@@ -112,17 +189,53 @@ export class CodexPromptRegistry {
turnId: readString(request.params, 'turnId'),
codexItemId,
promptKey: readString(request.params, 'approvalId') ?? codexItemId,
questionIds:
request.method === CODEX_USER_INPUT_METHOD ? readQuestionIds(request.params) : [],
questionIds,
questionIdAliases:
request.method === CODEX_USER_INPUT_METHOD
? new Map(questionIds.map((id) => [codexJournalPromptIdPart(id), id]))
: new Map(),
optionAnswers,
answers: new Map()
}
this.byAddress.set(this.address(prompt.threadId, prompt.promptKey), prompt)
const promptBytes = this.promptBytes(prompt)
if (promptBytes > MAX_CODEX_PROMPT_REGISTRY_BYTES) {
return null
}
while (
this.retainedPromptBytes() + promptBytes > MAX_CODEX_PROMPT_REGISTRY_BYTES &&
this.byAddress.size > 0
) {
const oldest = this.byAddress.values().next().value as CodexPendingPrompt | undefined
if (!oldest) {
break
}
this.byAddress.delete(this.address(oldest.threadId, oldest.promptKey))
}
if (this.retainedPromptBytes() + promptBytes > MAX_CODEX_PROMPT_REGISTRY_BYTES) {
return null
}
const address = this.address(prompt.threadId, prompt.promptKey)
this.byAddress.delete(address)
this.byAddress.set(address, prompt)
this.trim()
return prompt
}
/** Called by the translation module once the prompt has a journal id. */
bindJournalItemId(journalItemId: string, threadId: string, promptKey: string): void {
this.journalItemIds.set(journalItemId, this.address(threadId, promptKey))
const existing = this.journalItemIds.get(journalItemId)
if (existing) {
this.boundPrompts.delete(journalItemId)
}
this.journalItemIds.delete(journalItemId)
const address = this.address(threadId, promptKey)
const prompt = this.byAddress.get(address)
if (!prompt) {
return
}
this.journalItemIds.set(journalItemId, address)
this.boundPrompts.set(journalItemId, prompt)
this.trim()
}
/** Falls back to treating the id as a prompt key, which is what it is before
@@ -130,7 +243,7 @@ export class CodexPromptRegistry {
find(journalItemId: string): CodexPendingPrompt | null {
const address = this.journalItemIds.get(journalItemId)
if (address) {
return this.byAddress.get(address) ?? null
return this.boundPrompts.get(journalItemId) ?? this.byAddress.get(address) ?? null
}
const matches = [...this.byAddress.values()].filter(
(prompt) => prompt.promptKey === journalItemId
@@ -140,10 +253,13 @@ export class CodexPromptRegistry {
forget(prompt: CodexPendingPrompt): void {
const address = this.address(prompt.threadId, prompt.promptKey)
this.byAddress.delete(address)
for (const [journalItemId, boundAddress] of this.journalItemIds) {
if (boundAddress === address) {
if (this.byAddress.get(address) === prompt) {
this.byAddress.delete(address)
}
for (const [journalItemId, boundPrompt] of this.boundPrompts) {
if (boundPrompt === prompt) {
this.journalItemIds.delete(journalItemId)
this.boundPrompts.delete(journalItemId)
}
}
}
@@ -151,6 +267,7 @@ export class CodexPromptRegistry {
clear(): void {
this.byAddress.clear()
this.journalItemIds.clear()
this.boundPrompts.clear()
}
}
@@ -169,13 +286,19 @@ export function applyCodexPromptAnswer(
}
return { decision: optionId }
}
const decoded = decodeCodexQuestionOptionId(optionId)
const mapped = prompt.optionAnswers.get(optionId)
const decoded = mapped ?? decodeCodexQuestionOptionId(optionId)
const questionId =
decoded?.questionId ?? (prompt.questionIds.length === 1 ? prompt.questionIds[0] : null)
(decoded?.questionId
? (prompt.questionIdAliases.get(decoded.questionId) ?? decoded.questionId)
: null) ?? (prompt.questionIds.length === 1 ? prompt.questionIds[0] : null)
const answer = decoded?.answer ?? optionId
if (!questionId || !prompt.questionIds.includes(questionId)) {
throw new Error(`${optionId} does not name a question on Codex item ${prompt.codexItemId}`)
}
if (Buffer.byteLength(answer, 'utf8') > CODEX_PROMPT_MAX_ANSWER_BYTES) {
throw new Error('codex prompt answer exceeds bounded registry state')
}
prompt.answers.set(questionId, answer)
if (prompt.questionIds.some((id) => !prompt.answers.has(id))) {
return null
@@ -1,9 +1,13 @@
import type { CodexAppServerServerRequest } from './codex-app-server-connection'
import { disposeCodexServerRequest } from './codex-server-request-disposition'
import type { CodexJournalTranslationAdmission } from './codex-structured-journal-translation'
import type { CodexSession, CodexStructuredSessionEvent } from './codex-structured-session-state'
import { readCodexThreadId, readCodexTurnId } from './codex-structured-thread-facts'
type EmitCodexEvent = (session: CodexSession, event: CodexStructuredSessionEvent) => void
type EmitCodexEvent = (
session: CodexSession,
event: CodexStructuredSessionEvent
) => CodexJournalTranslationAdmission
export function deliverCodexNotification(
sessionId: string,
@@ -11,17 +15,22 @@ export function deliverCodexNotification(
method: string,
params: unknown,
emit: EmitCodexEvent
): void {
): CodexJournalTranslationAdmission {
if (!session) {
return
return { accepted: true }
}
const threadId = readCodexThreadId(params) ?? session.threadId
const turnId =
method === 'turn/started' && threadId === session.threadId ? readCodexTurnId(params) : null
const turnWaiter = turnId ? session.turnIdWaiters[0] : undefined
const admission = emit(session, { type: 'notification', sessionId, threadId, method, params })
if (method === 'turn/started' && threadId === session.threadId) {
const turnId = readCodexTurnId(params)
const waiter = turnId ? session.turnIdWaiters.shift() : undefined
waiter?.(turnId as string)
if (admission.accepted && turnId && session.turnIdWaiters[0] === turnWaiter) {
session.turnIdWaiters.shift()
turnWaiter?.(turnId)
}
}
emit(session, { type: 'notification', sessionId, threadId, method, params })
return admission
}
export function deliverCodexServerRequest(
@@ -29,24 +38,31 @@ export function deliverCodexServerRequest(
session: CodexSession | undefined,
request: CodexAppServerServerRequest,
emit: EmitCodexEvent
): void {
): CodexJournalTranslationAdmission {
if (!session) {
return
return { accepted: true }
}
const disposition = disposeCodexServerRequest(session.prompts, session.connection, request)
const threadId = readCodexThreadId(request.params) ?? session.threadId
if (disposition.kind === 'responded') {
emit(session, {
const admission = emit(session, {
type: 'server-request',
sessionId,
threadId,
method: request.method,
params: request.params
})
return
if (!admission.accepted) {
void session.forceCloseUnexpected?.(
new Error(
`Codex server request ${request.method} could not be durably recorded (${admission.reason})`
)
)
}
return admission
}
const prompt = disposition.prompt
emit(session, {
const admission = emit(session, {
type: 'prompt',
sessionId,
threadId: prompt.threadId,
@@ -55,6 +71,15 @@ export function deliverCodexServerRequest(
codexItemId: prompt.codexItemId,
promptKey: prompt.promptKey
})
if (!admission.accepted) {
session.prompts.forget(prompt)
session.connection.respondWithError(
request.id,
-32001,
`Orca could not durably record ${request.method} prompt (${admission.reason})`
)
}
return admission
}
export function deliverCodexUnhandledFrame(
@@ -63,15 +88,24 @@ export function deliverCodexUnhandledFrame(
kind: string,
payload: unknown,
emit: EmitCodexEvent
): void {
): CodexJournalTranslationAdmission {
if (!session) {
return
return { accepted: true }
}
emit(session, {
const admission = emit(session, {
type: 'provider-frame',
sessionId,
threadId: readCodexThreadId(payload) ?? session.threadId,
kind,
payload
})
if (!admission.accepted) {
// There is no safe replay cursor for malformed/unhandled frames. Close the
// provider so host recovery records a truthful terminal failure instead of
// silently dropping the diagnostic under sink backpressure.
void session.forceCloseUnexpected?.(
new Error(`Codex provider frame ${kind} could not be durably recorded (${admission.reason})`)
)
}
return admission
}
@@ -0,0 +1,233 @@
import {
AgentSessionAcquisitionRefusal,
AgentSessionPreSpawnError,
type AgentSessionAcquisition,
type StructuredAgentSessionAcquireInput
} from '../native-chat/agent-session-wire/structured-agent-session-adapter'
import {
closeFailedCodexAcquisition,
stopSupersededCodexAcquisition
} from './codex-structured-acquisition-lifecycle'
import { createCodexJournalTranslator } from './codex-structured-journal-translation'
import { openCodexAppServerConnection } from './codex-app-server-connection'
import { codexProcessIdentity, codexProviderHandleLink } from './codex-structured-owner-identity'
import { buildCodexStructuredChildEnvironment } from './codex-structured-child-environment'
import { openCodexThread } from './codex-structured-thread-open'
import {
closeCodexPublishedSession,
handleCodexSessionExit
} from './codex-structured-session-close'
import {
reportedCodexThreadOptions,
restoredCodexSessionOptions
} from './codex-structured-session-options'
import {
codexSessionLifecycle,
mintCodexAcquisitionGeneration,
type CodexAcquisitionRegistry,
type CodexAcquisitionAttempt,
type CodexSession,
type CodexStructuredSessionAdapterDeps
} from './codex-structured-session-state'
import type { CodexStructuredTurnCancellation } from './codex-structured-turn-cancellation'
import type { CodexStructuredNotificationRetry } from './codex-structured-notification-retry'
import type { deliverCodexServerRequest } from './codex-structured-provider-events'
export async function acquireCodexStructuredSession(input: {
input: StructuredAgentSessionAcquireInput
deps: CodexStructuredSessionAdapterDeps
sessions: Map<string, CodexSession>
acquisitions: CodexAcquisitionRegistry
turnCancellation: CodexStructuredTurnCancellation
notificationRetries: CodexStructuredNotificationRetry
deliver: (
acquisition: CodexAcquisitionAttempt['window'],
sessionId: string,
event: () => unknown,
retainedBytes?: number
) => void
handleServerRequest: (
sessionId: string,
request: Parameters<typeof deliverCodexServerRequest>[2]
) => void
handleUnhandledFrame: (sessionId: string, kind: string, payload: unknown) => void
forceCloseUnexpected: (
sessionId: string,
fence: number,
acquisitionGeneration: string,
reason: Error
) => Promise<boolean>
}): Promise<AgentSessionAcquisition> {
const {
input: acquireInput,
deps,
sessions,
acquisitions,
turnCancellation,
notificationRetries
} = input
const sessionId = acquireInput.identity.sessionId
const { previousAttempt, attempt } = acquisitions.start(sessionId)
const acquisition = attempt.window
let unbindReadingControl: (() => void) | undefined
let primaryThreadId =
acquireInput.identity.providerHandle.kind === 'codex'
? acquireInput.identity.providerHandle.threadId
: null
const translator = acquireInput.events
? createCodexJournalTranslator({
sink: acquireInput.events,
primaryThreadId: () => primaryThreadId,
bindPromptItemId: (journalItemId, threadId, promptKey) =>
acquisition.prompts.bindJournalItemId(journalItemId, threadId, promptKey)
})
: null
const open = deps.openConnection ?? openCodexAppServerConnection
try {
await stopSupersededCodexAcquisition({
sessionId,
registry: acquisitions,
replacement: attempt,
previous: previousAttempt
})
acquisitions.assertCurrent(sessionId, attempt)
if (!(await closeCodexPublishedSession(sessions, sessionId, deps.onEvent))) {
throw new Error(`codex app-server for session ${sessionId} could not be stopped`)
}
acquisitions.assertCurrent(sessionId, attempt)
const launch = await deps
.resolveLaunch({ identity: acquireInput.identity })
.catch((error: unknown) => {
throw new AgentSessionPreSpawnError(error)
})
acquisitions.assertCurrent(sessionId, attempt)
const connection = await open(
{
command: launch.command,
args: launch.args,
cwd: launch.cwd,
env: buildCodexStructuredChildEnvironment(launch, acquireInput.spawnToken)
},
{
onNotification: (method, params) =>
input.deliver(
acquisition,
sessionId,
() => notificationRetries.handle(sessionId, method, params),
Buffer.byteLength(JSON.stringify(params ?? null), 'utf8')
),
onServerRequest: (request) =>
input.deliver(
acquisition,
sessionId,
() => input.handleServerRequest(sessionId, request),
Buffer.byteLength(JSON.stringify(request), 'utf8')
),
onUnhandledFrame: (kind, payload) =>
input.deliver(
acquisition,
sessionId,
() => input.handleUnhandledFrame(sessionId, kind, payload),
Buffer.byteLength(JSON.stringify(payload ?? null), 'utf8')
),
onExit: (error) => {
try {
handleCodexSessionExit({
sessions,
sessionId,
connection: acquisition.connection,
error,
prompts: acquisition.prompts,
...(deps.onEvent ? { onEvent: deps.onEvent } : {})
})
} finally {
notificationRetries.clear(sessionId, acquisition.connection)
}
}
}
)
acquisition.connection = connection
if (connection.pauseReading && connection.resumeReading) {
unbindReadingControl = acquireInput.events?.bindReadingControl?.({
pauseReading: connection.pauseReading,
resumeReading: () => {
connection.resumeReading?.()
notificationRetries.retry(sessionId, connection)
}
})
}
acquisitions.assertCurrent(sessionId, attempt)
const opened = await openCodexThread(connection, launch, deps.requestTimeoutMs)
acquisitions.assertCurrent(sessionId, attempt)
primaryThreadId = opened.threadId
const restoreAdmission = translator?.restoreThread(opened.threadId, opened.thread ?? {})
if (restoreAdmission && !restoreAdmission.accepted) {
throw new AgentSessionAcquisitionRefusal(
'Codex thread history exceeds the bounded restore queue; history was not partially imported.'
)
}
const process = await codexProcessIdentity(
{ ...acquireInput, pid: connection.pid },
deps.readProcessStartTime
)
acquisitions.assertCurrent(sessionId, attempt)
const acquired: AgentSessionAcquisition = {
process,
link: codexProviderHandleLink({
threadId: opened.threadId,
resumed: launch.resumeThreadId !== null,
fence: acquireInput.fence,
linkId: deps.mintLinkId?.(),
observedAt: deps.now?.() ?? Date.now()
}),
acquisitionGeneration: mintCodexAcquisitionGeneration(deps)
}
if (connection.closed) {
throw new Error(`codex app-server for session ${sessionId} exited while being acquired`)
}
acquisitions.assertCurrent(sessionId, attempt)
acquisitions.deleteIfCurrent(sessionId, attempt)
const session: CodexSession = {
connection,
...codexSessionLifecycle(acquireInput.fence, acquired.acquisitionGeneration as string),
threadId: opened.threadId,
historyPath: opened.historyPath,
prompts: acquisition.prompts,
options: restoredCodexSessionOptions(acquireInput.options),
reportedOptions: reportedCodexThreadOptions(opened),
turnIdWaiters: [],
translator,
forceCloseUnexpected: (reason) =>
input.forceCloseUnexpected(
sessionId,
acquireInput.fence,
acquired.acquisitionGeneration as string,
reason
),
...(unbindReadingControl ? { unbindReadingControl } : {})
}
turnCancellation.register(session)
sessions.set(sessionId, session)
for (const event of acquisition.drain()) {
event()
}
return acquired
} catch (error) {
if (sessions.get(sessionId)?.connection !== acquisition.connection) {
return closeFailedCodexAcquisition({
sessionId,
registry: acquisitions,
attempt,
cause: error,
dispose: () => {
unbindReadingControl?.()
translator?.dispose()
}
})
}
acquisitions.deleteIfCurrent(sessionId, attempt)
throw error
} finally {
attempt.finish()
}
}
@@ -0,0 +1,276 @@
import { describe, expect, it } from 'vitest'
import type {
AgentJournalMessageItem,
AgentSessionJournalIdentity
} from '../../shared/agent-session-journal-types'
import type {
CodexAppServerConnection,
CodexAppServerConnectionHandlers,
CodexAppServerLaunch,
openCodexAppServerConnection
} from './codex-app-server-connection'
import type { StructuredAgentSessionEventSink } from '../native-chat/agent-session-wire/structured-agent-session-event-sink'
import {
CodexStructuredSessionAdapter,
type CodexStructuredLaunch,
type CodexStructuredSessionAdapterDeps,
type CodexStructuredSessionEvent
} from './codex-structured-session-adapter'
const THREAD_ID = 'thread-abc'
function identityFor(sessionId: string): AgentSessionJournalIdentity {
return {
sessionId,
workspaceId: 'ws-1',
hostId: 'host-1',
agent: 'codex',
providerHandle: { kind: 'codex', threadId: THREAD_ID }
}
}
const USER_MESSAGE: AgentJournalMessageItem = {
kind: 'message',
role: 'user',
blocks: [{ type: 'text', text: 'ship it' }]
}
type Route = (params: Record<string, unknown> | undefined) => unknown
// `closed` is readonly on the real connection; the fake flips it so a test can
// kill the child at a chosen moment.
type FakeConnection = Omit<CodexAppServerConnection, 'closed'> & {
closed: boolean
launch: CodexAppServerLaunch
handlers: CodexAppServerConnectionHandlers
calls: { method: string; params?: Record<string, unknown> }[]
replies: { id: number | string; result?: unknown; code?: number; message?: string }[]
closeCount: number
}
/** Stands in for a live `codex app-server`: every RPC is answered from `routes`,
* and the test drives Codex's own traffic through `handlers`. */
function fakeCodex(routes: Record<string, Route> = {}): {
connections: FakeConnection[]
openConnection: typeof openCodexAppServerConnection
routes: Record<string, Route>
} {
const connections: FakeConnection[] = []
const openConnection = (async (launch, handlers = {}) => {
const connection: FakeConnection = {
launch,
handlers,
calls: [],
replies: [],
closeCount: 0,
pid: 4321,
closed: false,
request: async (method, params) => {
connection.calls.push({ method, params })
const route = routes[method]
return route ? route(params) : {}
},
notify: () => {},
respond: (id, result) => connection.replies.push({ id, result }),
respondWithError: (id, code, message) => connection.replies.push({ id, code, message }),
close: async () => {
connection.closeCount += 1
connection.closed = true
return true
}
}
connections.push(connection)
return connection
}) as typeof openCodexAppServerConnection
routes['thread/start'] ??= () => ({
thread: { id: THREAD_ID, path: '/rollouts/abc.jsonl' },
model: 'gpt-live',
reasoningEffort: 'medium'
})
routes['thread/resume'] ??= (params) => ({
thread: { id: (params as { threadId: string }).threadId },
model: 'gpt-live',
reasoningEffort: 'medium'
})
return { connections, openConnection, routes }
}
function adapterFor(
codex: ReturnType<typeof fakeCodex>,
launch: Partial<CodexStructuredLaunch> = {},
events: CodexStructuredSessionEvent[] = [],
processControl: Partial<
Pick<CodexStructuredSessionAdapterDeps, 'captureTurnProcesses' | 'terminateTurnProcesses'>
> = {}
): CodexStructuredSessionAdapter {
let acquisitionGeneration = 0
return new CodexStructuredSessionAdapter({
resolveLaunch: async () => ({
command: 'codex',
args: ['app-server'],
cwd: '/work/repo',
codexHome: null,
resumeThreadId: null,
...launch
}),
onEvent: (event) => events.push(event),
openConnection: codex.openConnection,
readProcessStartTime: async () => 1_700_000_000_000,
captureTurnProcesses: async () => ({ platform: 'win32', identities: new Map() }),
terminateTurnProcesses: async () => true,
now: () => 1_700_000_000_500,
mintAcquisitionGeneration: () => `generation-${++acquisitionGeneration}`,
...processControl
})
}
async function acquired(
codex: ReturnType<typeof fakeCodex>,
launch: Partial<CodexStructuredLaunch> = {},
events: CodexStructuredSessionEvent[] = []
): Promise<CodexStructuredSessionAdapter> {
const adapter = adapterFor(codex, launch, events)
await adapter.acquire({ identity: identityFor('session-1'), fence: 7, spawnToken: 'spawn-9' })
return adapter
}
describe('CodexStructuredSessionAdapter lifecycle', () => {
it('keeps sessions isolated and closes each child once', async () => {
const codex = fakeCodex()
const adapter = adapterFor(codex)
await adapter.acquire({ identity: identityFor('session-1'), fence: 1, spawnToken: 'spawn-a' })
await adapter.acquire({ identity: identityFor('session-2'), fence: 1, spawnToken: 'spawn-b' })
codex.connections[0].handlers.onServerRequest?.({
id: 21,
method: 'item/fileChange/requestApproval',
params: { itemId: 'codex-item-1', threadId: THREAD_ID, turnId: 'turn-1' }
})
await expect(
adapter.answerPrompt({
sessionId: 'session-2',
itemId: 'codex-item-1',
kind: 'approval',
optionId: 'accept',
fence: 1
})
).rejects.toThrow('no longer waiting on')
await adapter.closeAll()
expect(codex.connections.map((connection) => connection.closeCount)).toEqual([1, 1])
await expect(
adapter.cancelTurn({ sessionId: 'session-1', turnId: 'turn-1', fence: 1 })
).rejects.toThrow('no live codex app-server for session session-1')
})
it('retains ownership until a child exit is proven and reports it once', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const adapter = await acquired(codex, {}, events)
const connection = codex.connections[0]
connection.close = async () => {
connection.closeCount += 1
return false
}
connection.handlers.onExit?.(new Error('codex app-server connection ended'))
expect(events.at(-1)).toEqual({
type: 'ended',
sessionId: 'session-1',
reason: 'codex app-server connection ended',
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
})
await expect(
adapter.dispatch({
sessionId: 'session-1',
clientMessageId: 'client-1',
body: USER_MESSAGE,
fence: 7
})
).rejects.toThrow('no live codex app-server')
expect(await adapter.historyFilePath({ identity: identityFor('session-1') })).toBe(
'/rollouts/abc.jsonl'
)
await expect(adapter.closeSession('session-1')).resolves.toBe(false)
expect(events.filter((event) => event.type === 'ended')).toHaveLength(1)
})
it('keeps the live session when a child it already replaced dies', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const adapter = await acquired(codex, {}, events)
await adapter.acquire({ identity: identityFor('session-1'), fence: 8, spawnToken: 'spawn-10' })
const endedBeforeStaleExit = events.filter((event) => event.type === 'ended').length
codex.connections[0].handlers.onExit?.(new Error('the superseded child died'))
expect(events.filter((event) => event.type === 'ended')).toHaveLength(endedBeforeStaleExit)
expect(await adapter.historyFilePath({ identity: identityFor('session-1') })).toBe(
'/rollouts/abc.jsonl'
)
})
it('ignores Codex traffic that arrives after the session is gone', async () => {
const codex = fakeCodex()
const adapter = await acquired(codex)
const connection = codex.connections[0]
await adapter.closeSession('session-1')
connection.handlers.onNotification?.('item/agentMessage/delta', { delta: 'x' })
connection.handlers.onServerRequest?.({
id: 31,
method: 'item/fileChange/requestApproval',
params: { itemId: 'codex-item-9', threadId: THREAD_ID }
})
expect(connection.replies).toEqual([])
})
it('flushes the final coalesced text before a graceful close', async () => {
const codex = fakeCodex()
const bodies: AgentJournalMessageItem[] = []
const tombstones: unknown[] = []
const sink: StructuredAgentSessionEventSink = {
appendItem: (_identity, body) => {
if (body.kind === 'message') {
bodies.push(body)
}
},
appendTombstone: (identity) => {
tombstones.push(identity)
},
publish: () => {}
}
const adapter = adapterFor(codex)
await adapter.acquire({
identity: identityFor('session-1'),
fence: 7,
spawnToken: 'spawn-9',
events: sink
})
const notify = codex.connections[0]!.handlers.onNotification
notify?.('turn/started', { threadId: THREAD_ID, turn: { id: 'turn-1' } })
notify?.('item/started', {
threadId: THREAD_ID,
item: { type: 'agentMessage', id: 'item-1', text: '' }
})
notify?.('item/agentMessage/delta', {
threadId: THREAD_ID,
itemId: 'item-1',
delta: 'last words'
})
await adapter.closeSession('session-1')
expect(bodies.at(-1)?.blocks).toEqual([{ type: 'text', text: 'last words' }])
expect(tombstones).toContainEqual({
provider: 'legacy',
agent: 'codex',
sessionId: 'session-1',
recordId: 'turn-lifecycle:turn-1'
})
})
})
@@ -106,6 +106,7 @@ function adapterFor(
Pick<CodexStructuredSessionAdapterDeps, 'captureTurnProcesses' | 'terminateTurnProcesses'>
> = {}
): CodexStructuredSessionAdapter {
let acquisitionGeneration = 0
return new CodexStructuredSessionAdapter({
resolveLaunch: async () => ({
command: 'codex',
@@ -121,6 +122,7 @@ function adapterFor(
captureTurnProcesses: async () => ({ platform: 'win32', identities: new Map() }),
terminateTurnProcesses: async () => true,
now: () => 1_700_000_000_500,
mintAcquisitionGeneration: () => `generation-${++acquisitionGeneration}`,
...processControl
})
}
@@ -168,6 +170,7 @@ describe('CodexStructuredSessionAdapter.acquire', () => {
mintedAtFence: 7,
observedAt: 1_700_000_000_500
})
expect(acquisition.acquisitionGeneration).toBe('generation-1')
})
it('resumes the thread the durable handle chain names, not the client one', async () => {
@@ -185,11 +188,11 @@ describe('CodexStructuredSessionAdapter.acquire', () => {
expect(codex.connections[0].calls[0]).toEqual({
method: 'thread/resume',
params: {
params: expect.objectContaining({
threadId: 'thread-proven',
cwd: '/work/repo',
path: '/rollouts/thread-proven.jsonl'
}
})
})
expect(acquisition.link.origin).toBe('resumed')
expect(acquisition.link.handle).toEqual({ provider: 'codex', threadId: 'thread-proven' })
@@ -256,6 +259,42 @@ describe('CodexStructuredSessionAdapter.acquire', () => {
expect(codex.connections[0].replies).toEqual([{ id: 5, result: { decision: 'accept' } }])
})
it('retries a notification rejected by journal admission instead of dropping it', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
let attempts = 0
const sink: StructuredAgentSessionEventSink = {
appendItem: vi.fn(),
appendTombstone: vi.fn(),
publish: vi.fn(),
tryAppendItem: vi.fn((identity, body, blobs) => {
attempts += 1
if (attempts === 1) {
return { accepted: false as const, reason: 'backpressure' as const }
}
sink.appendItem(identity, body, blobs)
return { accepted: true as const }
})
}
const adapter = adapterFor(codex, {}, events)
await adapter.acquire({
identity: identityFor('session-1'),
fence: 7,
spawnToken: 'spawn-9',
events: sink
})
codex.connections[0].handlers.onNotification?.('item/completed', {
item: { type: 'userMessage', id: 'message-1', text: 'hello' }
})
await vi.waitFor(() => {
expect(events).toHaveLength(1)
expect(events[0]).toMatchObject({ type: 'notification', method: 'item/completed' })
})
expect(attempts).toBe(2)
})
it('refuses to publish a session whose child died while it was being acquired', async () => {
const codex = fakeCodex()
const adapter = new CodexStructuredSessionAdapter({
@@ -618,6 +657,99 @@ describe('CodexStructuredSessionAdapter prompts', () => {
expect(codex.connections[0].replies).toHaveLength(1)
})
it('responds with an error when a prompt cannot be admitted to the journal sink', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const sink: StructuredAgentSessionEventSink = {
appendItem: vi.fn(),
appendTombstone: vi.fn(),
publish: vi.fn(),
tryAppendItem: vi.fn(() => ({ accepted: false as const, reason: 'closed' as const }))
}
const adapter = adapterFor(codex, {}, events)
await adapter.acquire({
identity: identityFor('session-1'),
fence: 7,
spawnToken: 'spawn-9',
events: sink
})
askApproval(codex)
expect(events.filter((event) => event.type === 'prompt')).toEqual([])
expect(codex.connections[0].replies).toEqual([
{
id: 11,
code: -32001,
message:
'Orca could not durably record item/commandExecution/requestApproval prompt (closed)'
}
])
await expect(
adapter.answerPrompt({
sessionId: 'session-1',
itemId: 'codex-item-1',
kind: 'approval',
optionId: 'accept',
fence: 7
})
).rejects.toThrow('no longer waiting on')
})
it('force-closes when an unhandled provider frame cannot be admitted', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const sink: StructuredAgentSessionEventSink = {
appendItem: vi.fn(),
appendTombstone: vi.fn(),
publish: vi.fn(),
tryAppendItem: vi.fn(() => ({ accepted: false as const, reason: 'backpressure' as const }))
}
const adapter = adapterFor(codex, {}, events)
await adapter.acquire({
identity: identityFor('session-1'),
fence: 7,
spawnToken: 'spawn-9',
events: sink
})
codex.connections[0].handlers.onUnhandledFrame?.('frame:invalid-json', '{')
await vi.waitFor(() => expect(codex.connections[0].closeCount).toBe(1))
expect(events.filter((event) => event.type === 'ended')).toMatchObject([
{ cause: 'unexpected-exit', fence: 7, acquisitionGeneration: 'generation-1' }
])
})
it('force-closes after a responded server request is not durably admitted', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const sink: StructuredAgentSessionEventSink = {
appendItem: vi.fn(),
appendTombstone: vi.fn(),
publish: vi.fn(),
tryAppendItem: vi.fn(() => ({ accepted: false as const, reason: 'failed' as const }))
}
const adapter = adapterFor(codex, {}, events)
await adapter.acquire({
identity: identityFor('session-1'),
fence: 7,
spawnToken: 'spawn-9',
events: sink
})
codex.connections[0].handlers.onServerRequest?.({
id: 17,
method: 'item/permissions/requestApproval',
params: { threadId: THREAD_ID }
})
await vi.waitFor(() => expect(codex.connections[0].closeCount).toBe(1))
expect(events.filter((event) => event.type === 'ended')).toMatchObject([
{ cause: 'unexpected-exit', fence: 7, acquisitionGeneration: 'generation-1' }
])
})
it('answers each approval a tool item asks for separately', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
@@ -687,7 +819,10 @@ describe('CodexStructuredSessionAdapter prompts', () => {
itemId: 'codex-item-2',
threadId: THREAD_ID,
turnId: 'turn-1',
questions: [{ id: 'q1' }, { id: 'q2' }]
questions: [
{ id: 'q1', question: 'Use this answer?' },
{ id: 'q2', question: 'Use that answer?' }
]
}
})
@@ -745,139 +880,3 @@ describe('CodexStructuredSessionAdapter prompts', () => {
).rejects.toThrow('no longer waiting on codex-item-gone')
})
})
describe('CodexStructuredSessionAdapter lifecycle', () => {
it('keeps sessions isolated and closes each child once', async () => {
const codex = fakeCodex()
const adapter = adapterFor(codex)
await adapter.acquire({ identity: identityFor('session-1'), fence: 1, spawnToken: 'spawn-a' })
await adapter.acquire({ identity: identityFor('session-2'), fence: 1, spawnToken: 'spawn-b' })
codex.connections[0].handlers.onServerRequest?.({
id: 21,
method: 'item/fileChange/requestApproval',
params: { itemId: 'codex-item-1', threadId: THREAD_ID, turnId: 'turn-1' }
})
await expect(
adapter.answerPrompt({
sessionId: 'session-2',
itemId: 'codex-item-1',
kind: 'approval',
optionId: 'accept',
fence: 1
})
).rejects.toThrow('no longer waiting on')
await adapter.closeAll()
expect(codex.connections.map((connection) => connection.closeCount)).toEqual([1, 1])
await expect(
adapter.cancelTurn({ sessionId: 'session-1', turnId: 'turn-1', fence: 1 })
).rejects.toThrow('no live codex app-server for session session-1')
})
it('retains ownership until a child exit is proven and reports it once', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const adapter = await acquired(codex, {}, events)
const connection = codex.connections[0]
connection.close = async () => {
connection.closeCount += 1
return false
}
connection.handlers.onExit?.(new Error('codex app-server connection ended'))
expect(events.at(-1)).toEqual({
type: 'ended',
sessionId: 'session-1',
reason: 'codex app-server connection ended'
})
await expect(
adapter.dispatch({
sessionId: 'session-1',
clientMessageId: 'client-1',
body: USER_MESSAGE,
fence: 7
})
).rejects.toThrow('no live codex app-server')
expect(await adapter.historyFilePath({ identity: identityFor('session-1') })).toBe(
'/rollouts/abc.jsonl'
)
await expect(adapter.closeSession('session-1')).resolves.toBe(false)
expect(events.filter((event) => event.type === 'ended')).toHaveLength(1)
})
it('keeps the live session when a child it already replaced dies', async () => {
const codex = fakeCodex()
const events: CodexStructuredSessionEvent[] = []
const adapter = await acquired(codex, {}, events)
await adapter.acquire({ identity: identityFor('session-1'), fence: 8, spawnToken: 'spawn-10' })
const endedBeforeStaleExit = events.filter((event) => event.type === 'ended').length
codex.connections[0].handlers.onExit?.(new Error('the superseded child died'))
expect(events.filter((event) => event.type === 'ended')).toHaveLength(endedBeforeStaleExit)
expect(await adapter.historyFilePath({ identity: identityFor('session-1') })).toBe(
'/rollouts/abc.jsonl'
)
})
it('ignores Codex traffic that arrives after the session is gone', async () => {
const codex = fakeCodex()
const adapter = await acquired(codex)
const connection = codex.connections[0]
await adapter.closeSession('session-1')
connection.handlers.onNotification?.('item/agentMessage/delta', { delta: 'x' })
connection.handlers.onServerRequest?.({
id: 31,
method: 'item/fileChange/requestApproval',
params: { itemId: 'codex-item-9', threadId: THREAD_ID }
})
expect(connection.replies).toEqual([])
})
it('flushes the final coalesced text before a graceful close', async () => {
const codex = fakeCodex()
const bodies: AgentJournalMessageItem[] = []
const tombstones: unknown[] = []
const sink: StructuredAgentSessionEventSink = {
appendItem: (_identity, body) => {
if (body.kind === 'message') {
bodies.push(body)
}
},
appendTombstone: (identity) => tombstones.push(identity),
publish: () => {}
}
const adapter = adapterFor(codex)
await adapter.acquire({
identity: identityFor('session-1'),
fence: 7,
spawnToken: 'spawn-9',
events: sink
})
const notify = codex.connections[0]!.handlers.onNotification
notify?.('turn/started', { threadId: THREAD_ID, turn: { id: 'turn-1' } })
notify?.('item/started', {
threadId: THREAD_ID,
item: { type: 'agentMessage', id: 'item-1', text: '' }
})
notify?.('item/agentMessage/delta', {
threadId: THREAD_ID,
itemId: 'item-1',
delta: 'last words'
})
await adapter.closeSession('session-1')
expect(bodies.at(-1)?.blocks).toEqual([{ type: 'text', text: 'last words' }])
expect(tombstones).toContainEqual({
provider: 'legacy',
agent: 'codex',
sessionId: 'session-1',
recordId: 'turn-lifecycle:turn-1'
})
})
})
+115 -167
View File
@@ -2,40 +2,29 @@ import type {
AgentJournalMessageItem,
AgentSessionJournalIdentity
} from '../../shared/agent-session-journal-types'
import {
AgentSessionPreSpawnError,
type AgentSessionAcquisition,
type AgentSessionDispatchOutcome,
type StructuredAgentSessionAcquireInput,
type StructuredAgentSessionAdapter,
type StructuredAgentSessionSetOptionInput
import type {
AgentSessionAcquisition,
AgentSessionDispatchOutcome,
StructuredAgentSessionAcquireInput,
StructuredAgentSessionAdapter,
StructuredAgentSessionSetOptionInput
} from '../native-chat/agent-session-wire/structured-agent-session-adapter'
import {
closeFailedCodexAcquisition,
stopSupersededCodexAcquisition
} from './codex-structured-acquisition-lifecycle'
import { createCodexJournalTranslator } from './codex-structured-journal-translation'
import { openCodexAppServerConnection } from './codex-app-server-connection'
import { codexProcessIdentity, codexProviderHandleLink } from './codex-structured-owner-identity'
import { buildCodexStructuredChildEnvironment } from './codex-structured-child-environment'
import type { CodexJournalTranslationAdmission } from './codex-structured-journal-translation'
import { answerCodexPrompt } from './codex-structured-prompt-replies'
import { openCodexThread } from './codex-structured-thread-open'
import { dispatchCodexTurn, isCodexTurnOptionKey } from './codex-structured-turn-start'
import { supportsCodexStructuredLocation } from './codex-structured-location-support'
import {
closeAllCodexSessions,
closeCodexPublishedSession,
closeCodexSession,
handleCodexSessionExit
closeCodexSession
} from './codex-structured-session-close'
import {
applyCodexStructuredSessionOption,
readLiveCodexSessionOptions,
reportedCodexThreadOptions,
restoredCodexSessionOptions
readLiveCodexSessionOptions
} from './codex-structured-session-options'
import {
CodexAcquisitionRegistry,
requireLiveCodexSession,
type CodexAcquisitionAttempt,
type CodexSession,
type CodexStructuredSessionAdapterDeps,
@@ -47,6 +36,8 @@ import {
deliverCodexUnhandledFrame
} from './codex-structured-provider-events'
import { CodexStructuredTurnCancellation } from './codex-structured-turn-cancellation'
import { createCodexStructuredNotificationRetry } from './codex-structured-notification-retry'
import { acquireCodexStructuredSession } from './codex-structured-session-acquire'
export type {
CodexStructuredLaunch,
@@ -58,175 +49,91 @@ export class CodexStructuredSessionAdapter implements StructuredAgentSessionAdap
private readonly sessions = new Map<string, CodexSession>()
private readonly acquisitions = new CodexAcquisitionRegistry()
private readonly turnCancellation: CodexStructuredTurnCancellation
private readonly notificationRetries: ReturnType<typeof createCodexStructuredNotificationRetry>
constructor(private readonly deps: CodexStructuredSessionAdapterDeps) {
this.notificationRetries = createCodexStructuredNotificationRetry({
sessionFor: (sessionId) => this.sessions.get(sessionId),
translate: (sessionId, session, method, params) =>
this.translateNotification(sessionId, session, method, params)
})
this.turnCancellation = new CodexStructuredTurnCancellation({
captureTurnProcesses: deps.captureTurnProcesses,
terminateTurnProcesses: deps.terminateTurnProcesses,
requestTimeoutMs: deps.requestTimeoutMs,
emit: (session, event) => this.emit(session, event)
emit: (session, event) => {
const admission = this.emit(session, event)
if (!admission.accepted && event.type === 'notification') {
this.notificationRetries.handle(event.sessionId, event.method, event.params)
}
return admission
}
})
}
supportsLocation = supportsCodexStructuredLocation
async acquire(input: StructuredAgentSessionAcquireInput): Promise<AgentSessionAcquisition> {
const sessionId = input.identity.sessionId
const { previousAttempt, attempt } = this.acquisitions.start(sessionId)
const acquisition = attempt.window
let primaryThreadId =
input.identity.providerHandle.kind === 'codex' ? input.identity.providerHandle.threadId : null
const translator = input.events
? createCodexJournalTranslator({
sink: input.events,
primaryThreadId: () => primaryThreadId,
bindPromptItemId: (journalItemId, threadId, promptKey) =>
acquisition.prompts.bindJournalItemId(journalItemId, threadId, promptKey)
})
: null
const open = this.deps.openConnection ?? openCodexAppServerConnection
try {
await stopSupersededCodexAcquisition({
sessionId,
registry: this.acquisitions,
replacement: attempt,
previous: previousAttempt
})
this.acquisitions.assertCurrent(sessionId, attempt)
if (!(await closeCodexPublishedSession(this.sessions, sessionId, this.deps.onEvent))) {
throw new Error(`codex app-server for session ${sessionId} could not be stopped`)
}
this.acquisitions.assertCurrent(sessionId, attempt)
const launch = await this.deps
.resolveLaunch({ identity: input.identity })
.catch((error: unknown) => {
throw new AgentSessionPreSpawnError(error)
})
this.acquisitions.assertCurrent(sessionId, attempt)
const connection = await open(
{
command: launch.command,
args: launch.args,
cwd: launch.cwd,
env: buildCodexStructuredChildEnvironment(launch, input.spawnToken)
},
{
onNotification: (method, params) =>
this.deliver(acquisition, sessionId, () =>
this.handleNotification(sessionId, method, params)
),
onServerRequest: (request) =>
this.deliver(acquisition, sessionId, () =>
this.handleServerRequest(sessionId, request)
),
onUnhandledFrame: (kind, payload) =>
this.deliver(acquisition, sessionId, () =>
this.handleUnhandledFrame(sessionId, kind, payload)
),
onExit: (error) => {
acquisition.prompts.clear()
handleCodexSessionExit({
sessions: this.sessions,
sessionId,
connection: acquisition.connection,
error,
...(this.deps.onEvent ? { onEvent: this.deps.onEvent } : {})
})
}
}
)
acquisition.connection = connection
this.acquisitions.assertCurrent(sessionId, attempt)
const opened = await openCodexThread(connection, launch, this.deps.requestTimeoutMs)
this.acquisitions.assertCurrent(sessionId, attempt)
primaryThreadId = opened.threadId
translator?.restoreThread(opened.threadId, opened.thread ?? {})
const process = await codexProcessIdentity(
{ ...input, pid: connection.pid },
this.deps.readProcessStartTime
)
this.acquisitions.assertCurrent(sessionId, attempt)
const acquired: AgentSessionAcquisition = {
process,
link: codexProviderHandleLink({
threadId: opened.threadId,
resumed: launch.resumeThreadId !== null,
fence: input.fence,
linkId: this.deps.mintLinkId?.(),
observedAt: this.deps.now?.() ?? Date.now()
})
}
// Publish only after every promised identity is proven and this attempt still owns the child.
if (connection.closed) {
throw new Error(`codex app-server for session ${sessionId} exited while being acquired`)
}
this.acquisitions.assertCurrent(sessionId, attempt)
this.acquisitions.deleteIfCurrent(sessionId, attempt)
const session: CodexSession = {
connection,
ended: false,
threadId: opened.threadId,
historyPath: opened.historyPath,
prompts: acquisition.prompts,
options: restoredCodexSessionOptions(input.options),
reportedOptions: reportedCodexThreadOptions(opened),
turnIdWaiters: [],
translator
}
this.turnCancellation.register(session)
this.sessions.set(sessionId, session)
for (const event of acquisition.drain()) {
event()
}
return acquired
} catch (error) {
// Reap this attempt's child only. A replacement already published for the
// same session keeps running.
if (this.sessions.get(sessionId)?.connection !== acquisition.connection) {
return closeFailedCodexAcquisition({
sessionId,
registry: this.acquisitions,
attempt,
cause: error,
dispose: () => translator?.dispose()
})
}
this.acquisitions.deleteIfCurrent(sessionId, attempt)
throw error
} finally {
attempt.finish()
}
}
acquire = (input: StructuredAgentSessionAcquireInput): Promise<AgentSessionAcquisition> =>
acquireCodexStructuredSession({
input,
deps: this.deps,
sessions: this.sessions,
acquisitions: this.acquisitions,
turnCancellation: this.turnCancellation,
notificationRetries: this.notificationRetries,
deliver: (acquisition, sessionId, event, retainedBytes) =>
this.deliver(acquisition, sessionId, event, retainedBytes),
handleServerRequest: (sessionId, request) => this.handleServerRequest(sessionId, request),
handleUnhandledFrame: (sessionId, kind, payload) =>
this.handleUnhandledFrame(sessionId, kind, payload),
forceCloseUnexpected: (sessionId, fence, acquisitionGeneration, reason) =>
this.forceCloseUnexpected(sessionId, fence, acquisitionGeneration, reason)
})
/** Buffers pre-publication events and drops events from superseded children. */
private deliver(
acquisition: CodexAcquisitionAttempt['window'],
sessionId: string,
event: () => void
event: () => unknown,
retainedBytes?: number
): void {
if (acquisition.buffer(event)) {
if (acquisition.buffer(event, retainedBytes)) {
return
}
if (this.sessions.get(sessionId)?.connection === acquisition.connection) {
event()
} else if (acquisition.isOverflowed) {
// Pre-publication overflow is an acquisition failure, not a dropped
// notification; tear down the child so callers retry explicitly.
void acquisition.connection?.close()
}
}
private handleNotification(sessionId: string, method: string, params: unknown): void {
const session = this.sessions.get(sessionId)
if (session && this.turnCancellation.handleNotification(sessionId, session, method, params)) {
return
private translateNotification(
sessionId: string,
session: CodexSession,
method: string,
params: unknown
): CodexJournalTranslationAdmission {
if (this.turnCancellation.handleNotification(sessionId, session, method, params)) {
return { accepted: true }
}
deliverCodexNotification(sessionId, session, method, params, (session, event) =>
this.emit(session, event)
return deliverCodexNotification(sessionId, session, method, params, (current, event) =>
this.emit(current, event)
)
}
/** Journal first so observers never see an event ahead of its durable row. */
private emit(session: CodexSession, event: CodexStructuredSessionEvent): void {
session.translator?.handle(event)
private emit(
session: CodexSession,
event: CodexStructuredSessionEvent
): CodexJournalTranslationAdmission {
const admission = session.translator?.handle(event) ?? { accepted: true }
if (!admission.accepted) {
return admission
}
this.deps.onEvent?.(event)
return admission
}
private handleServerRequest(
@@ -282,6 +189,7 @@ export class CodexStructuredSessionAdapter implements StructuredAgentSessionAdap
}): Promise<void> {
const session = this.session(input.sessionId)
answerCodexPrompt(session.prompts, session.connection, input.itemId, input.optionId)
session.translator?.resolvePrompt(input.itemId)
}
async setOption(
@@ -305,8 +213,52 @@ export class CodexStructuredSessionAdapter implements StructuredAgentSessionAdap
identity: AgentSessionJournalIdentity
}): Promise<string | null> => this.sessions.get(input.identity.sessionId)?.historyPath ?? null
closeSession = (sessionId: string): Promise<boolean> =>
closeCodexSession(sessionId, this.sessions, this.acquisitions, this.deps.onEvent)
closeSession = async (sessionId: string): Promise<boolean> => {
const closed = await closeCodexSession(
sessionId,
this.sessions,
this.acquisitions,
this.deps.onEvent
)
if (closed) {
this.notificationRetries.clear(sessionId, null)
}
return closed
}
forceCloseSession = async (sessionId: string): Promise<boolean> => {
const closed = await closeCodexPublishedSession(this.sessions, sessionId, this.deps.onEvent, {
allowFailedSettlement: true,
requestedClose: false
})
if (closed) {
this.notificationRetries.clear(sessionId, null)
}
return closed
}
private forceCloseUnexpected(
sessionId: string,
fence: number,
acquisitionGeneration: string,
reason: Error
): Promise<boolean> {
const session = this.sessions.get(sessionId)
if (
!session ||
session.ended ||
session.fence !== fence ||
session.acquisitionGeneration !== acquisitionGeneration
) {
return Promise.resolve(false)
}
return closeCodexPublishedSession(this.sessions, sessionId, this.deps.onEvent, {
allowFailedSettlement: true,
requestedClose: false,
expectedFence: fence,
expectedAcquisitionGeneration: acquisitionGeneration,
unexpectedReason: reason
})
}
disposeSession = (sessionId: string): Promise<boolean> => this.closeSession(sessionId)
closeAll = (): Promise<void> =>
closeAllCodexSessions(this.sessions, this.acquisitions, (sessionId) =>
@@ -316,10 +268,6 @@ export class CodexStructuredSessionAdapter implements StructuredAgentSessionAdap
this.closeSession(input.sessionId)
private session(sessionId: string): CodexSession {
const session = this.sessions.get(sessionId)
if (!session || session.ended) {
throw new Error(`no live codex app-server for session ${sessionId}`)
}
return session
return requireLiveCodexSession(this.sessions, sessionId)
}
}
@@ -0,0 +1,167 @@
import { describe, expect, it, vi } from 'vitest'
import type { AgentSessionJournalIdentity } from '../../shared/agent-session-journal-types'
import type {
CodexAppServerConnection,
CodexAppServerConnectionHandlers,
openCodexAppServerConnection
} from './codex-app-server-connection'
import {
CodexStructuredSessionAdapter,
type CodexStructuredSessionEvent
} from './codex-structured-session-adapter'
import { handleCodexSessionExit } from './codex-structured-session-close'
import type { CodexSession } from './codex-structured-session-state'
const THREAD = 'thread-1'
function identity(sessionId: string): AgentSessionJournalIdentity {
return {
sessionId,
workspaceId: 'workspace-1',
hostId: 'host-1',
agent: 'codex',
providerHandle: { kind: 'codex', threadId: THREAD }
}
}
function adapterFixture() {
const connections: {
connection: CodexAppServerConnection
handlers: CodexAppServerConnectionHandlers
}[] = []
const events: CodexStructuredSessionEvent[] = []
let generation = 0
const openConnection = (async (_launch, handlers = {}) => {
const connection: CodexAppServerConnection = {
pid: 4321,
closed: false,
request: async (method) => (method === 'thread/start' ? { thread: { id: THREAD } } : {}),
notify: () => {},
respond: () => {},
respondWithError: () => {},
close: async () => true
}
connections.push({ connection, handlers })
return connection
}) as typeof openCodexAppServerConnection
const adapter = new CodexStructuredSessionAdapter({
resolveLaunch: async () => ({
command: 'codex',
args: ['app-server'],
cwd: '/workspace',
codexHome: null,
resumeThreadId: null
}),
openConnection,
readProcessStartTime: async () => 1_700_000_000_000,
mintAcquisitionGeneration: () => `generation-${++generation}`,
onEvent: (event) => events.push(event)
})
return { adapter, connections, events }
}
describe('Codex structured session close lifecycle', () => {
it('forwards a one-shot exit when lifecycle admission is rejected', () => {
const connection: CodexAppServerConnection = {
pid: 4321,
closed: true,
request: async () => ({}),
notify: () => {},
respond: () => {},
respondWithError: () => {},
close: async () => true
}
const prompts = { clear: vi.fn() } as unknown as CodexSession['prompts']
const translator = {
handle: vi.fn().mockReturnValueOnce({ accepted: false, reason: 'backpressure' as const }),
dispose: vi.fn()
} as unknown as NonNullable<CodexSession['translator']>
const session = {
connection,
ended: false,
requestedClose: false,
fence: 7,
acquisitionGeneration: 'generation-1',
threadId: THREAD,
historyPath: null,
prompts,
options: new Map(),
reportedOptions: {},
turnIdWaiters: [],
translator
} as CodexSession
const sessions = new Map([['session-1', session]])
const onEvent = vi.fn()
expect(
handleCodexSessionExit({
sessions,
sessionId: 'session-1',
connection,
error: new Error('provider exited'),
prompts,
onEvent
})
).toBe(true)
expect(session.ended).toBe(true)
expect(prompts.clear).toHaveBeenCalledOnce()
expect(onEvent).toHaveBeenCalledOnce()
expect(translator.dispose).toHaveBeenCalledOnce()
expect(onEvent.mock.calls[0]?.[0]).toMatchObject({
cause: 'unexpected-exit',
settlementRetryRequired: true
})
expect(translator.handle).toHaveBeenCalledOnce()
})
it('mints a distinct child generation even when acquisitions share one fence', async () => {
const { adapter } = adapterFixture()
const input = { identity: identity('session-1'), fence: 7, spawnToken: 'spawn-1' }
const first = await adapter.acquire(input)
const second = await adapter.acquire(input)
expect(first.acquisitionGeneration).toBe('generation-1')
expect(second.acquisitionGeneration).toBe('generation-2')
})
it('distinguishes an observed provider death from a requested close', async () => {
const { adapter, connections, events } = adapterFixture()
await adapter.acquire({ identity: identity('session-1'), fence: 7, spawnToken: 'spawn-1' })
connections[0]?.handlers.onExit?.(new Error('provider exited'))
await adapter.acquire({ identity: identity('session-2'), fence: 9, spawnToken: 'spawn-2' })
await adapter.closeSession('session-2')
expect(events.filter((event) => event.type === 'ended')).toMatchObject([
{
cause: 'unexpected-exit',
fence: 7,
acquisitionGeneration: 'generation-1'
},
{
cause: 'requested-close',
fence: 9,
acquisitionGeneration: 'generation-2'
}
])
})
it('force-close preserves unexpected-exit evidence when the adapter reports exit during close', async () => {
const { adapter, connections, events } = adapterFixture()
await adapter.acquire({ identity: identity('session-1'), fence: 7, spawnToken: 'spawn-1' })
const current = connections[0]
if (!current) {
throw new Error('missing connection')
}
current.connection.close = async () => {
current.handlers.onExit?.(new Error('sink failed'))
return true
}
await expect(adapter.forceCloseSession?.('session-1')).resolves.toBe(true)
expect(events.filter((event) => event.type === 'ended')).toMatchObject([
{ cause: 'unexpected-exit', reason: 'sink failed', fence: 7 }
])
})
})
@@ -6,54 +6,99 @@ import {
type CodexSession,
type CodexStructuredSessionEvent
} from './codex-structured-session-state'
import type { StructuredAgentSessionLifecycleEvent } from '../native-chat/agent-session-wire/structured-agent-session-adapter'
export function handleCodexSessionExit(input: {
sessions: Map<string, CodexSession>
sessionId: string
connection: CodexAppServerConnection | null
error: Error
prompts?: CodexSession['prompts']
allowFailedSettlement?: boolean
onEvent?: (event: CodexStructuredSessionEvent) => void
}): void {
}): boolean {
const session = input.sessions.get(input.sessionId)
if (!session || session.connection !== input.connection || session.ended) {
return
input.prompts?.clear()
return false
}
const event: StructuredAgentSessionLifecycleEvent = {
type: 'ended',
sessionId: input.sessionId,
reason: input.error.message,
cause: session.requestedClose ? 'requested-close' : 'unexpected-exit',
fence: session.fence,
acquisitionGeneration: session.acquisitionGeneration
} as const
// A synchronous sink rejection (usually backpressure) is handed to host
// recovery, which appends the bounded fallback before reacquisition.
const admission = session.translator?.handle(event) ?? { accepted: true }
if (!admission.accepted) {
// The connection invokes onExit exactly once. Forward a flagged event so
// host recovery can append its no-new-blob fallback even when admission is
// backpressured; waiting for a second callback would strand the lease.
if (event.cause !== 'unexpected-exit' && !input.allowFailedSettlement) {
return false
}
event.settlementRetryRequired = true
}
session.ended = true
const event = { type: 'ended', sessionId: input.sessionId, reason: input.error.message } as const
session.translator?.handle(event)
session.unbindReadingControl?.()
input.onEvent?.(event)
session.prompts.clear()
session.translator?.dispose()
return true
}
export async function closeCodexPublishedSession(
sessions: Map<string, CodexSession>,
sessionId: string,
onEvent?: (event: CodexStructuredSessionEvent) => void
onEvent?: (event: CodexStructuredSessionEvent) => void,
options?: {
allowFailedSettlement?: boolean
requestedClose?: boolean
expectedFence?: number
expectedAcquisitionGeneration?: string
unexpectedReason?: Error
}
): Promise<boolean> {
const session = sessions.get(sessionId)
if (!session) {
return true
}
session.prompts.clear()
if (
(options?.expectedFence !== undefined && session.fence !== options.expectedFence) ||
(options?.expectedAcquisitionGeneration !== undefined &&
session.acquisitionGeneration !== options.expectedAcquisitionGeneration)
) {
return false
}
// Sink-failure recovery force-closes the child but must preserve the
// observed-exit cause so host lease settlement runs as an unexpected death.
session.requestedClose = options?.requestedClose ?? true
// Keep the session indexed until the child exit is observed. A timeout or
// failed kill must leave the live connection available for a safe retry.
const exited = await session.connection.close()
if (exited !== true) {
return false
}
sessions.delete(sessionId)
if (!session.ended) {
session.ended = true
const event: CodexStructuredSessionEvent = {
type: 'ended',
const handled = handleCodexSessionExit({
sessions,
sessionId,
reason: 'codex session closed'
connection: session.connection,
error: options?.unexpectedReason ?? new Error('codex session closed'),
prompts: session.prompts,
...(options?.allowFailedSettlement ? { allowFailedSettlement: true } : {}),
...(onEvent ? { onEvent } : {})
})
// Keep the closed session indexed when terminal settlement admission was
// rejected; a later close attempt retries the same stable lifecycle event.
if (!handled) {
return false
}
session.translator?.handle(event)
onEvent?.(event)
session.translator?.flush()
session.translator?.dispose()
}
sessions.delete(sessionId)
return true
}
@@ -21,6 +21,9 @@ function optionSession(request: CodexAppServerConnection['request']): CodexSessi
close: async () => true
},
ended: false,
requestedClose: false,
fence: 1,
acquisitionGeneration: 'generation-1',
threadId: 'thread-1',
historyPath: null,
prompts: new CodexAcquisitionWindow().prompts,
@@ -1,4 +1,5 @@
import type { AgentSessionJournalIdentity } from '../../shared/agent-session-journal-types'
import { randomUUID } from 'node:crypto'
import { cancelProcessAcquisition } from '../../shared/child-process/cancel-process-acquisition'
import type {
CodexAppServerConnection,
@@ -7,6 +8,7 @@ import type {
import { CodexAcquisitionWindow } from './codex-structured-acquisition-window'
import type { CodexJournalTranslator } from './codex-structured-journal-translation'
import type { CodexTurnProcessSnapshot } from './codex-structured-turn-processes'
import type { StructuredAgentSessionLifecycleEvent } from '../native-chat/agent-session-wire/structured-agent-session-adapter'
export type CodexStructuredLaunch = {
command: string
@@ -31,6 +33,8 @@ export type CodexStructuredSessionEvent =
codexItemId: string
promptKey: string
}
| StructuredAgentSessionLifecycleEvent
/** Translator-only compatibility for callers that do not participate in host recovery. */
| { type: 'ended'; sessionId: string; reason: string }
export type CodexStructuredSessionAdapterDeps = {
@@ -41,6 +45,7 @@ export type CodexStructuredSessionAdapterDeps = {
openConnection?: typeof openCodexAppServerConnection
readProcessStartTime?: (pid: number) => Promise<number | null>
mintLinkId?: () => string
mintAcquisitionGeneration?: () => string
now?: () => number
requestTimeoutMs?: number
captureTurnProcesses?: (rootPid: number) => Promise<CodexTurnProcessSnapshot | null>
@@ -53,6 +58,9 @@ export type CodexStructuredSessionAdapterDeps = {
export type CodexSession = {
connection: CodexAppServerConnection
ended: boolean
requestedClose: boolean
fence: number
acquisitionGeneration: string
threadId: string
historyPath: string | null
prompts: CodexAcquisitionWindow['prompts']
@@ -60,6 +68,31 @@ export type CodexSession = {
reportedOptions: { model?: string; effort?: string }
turnIdWaiters: ((turnId: string) => void)[]
translator: CodexJournalTranslator | null
unbindReadingControl?: () => void
/** Terminates this exact child as an unexpected death and enters host recovery. */
forceCloseUnexpected?: (reason: Error) => Promise<boolean>
}
export function mintCodexAcquisitionGeneration(deps: CodexStructuredSessionAdapterDeps): string {
return deps.mintAcquisitionGeneration?.() ?? randomUUID()
}
export function codexSessionLifecycle(
fence: number,
acquisitionGeneration: string
): Pick<CodexSession, 'ended' | 'requestedClose' | 'fence' | 'acquisitionGeneration'> {
return { ended: false, requestedClose: false, fence, acquisitionGeneration }
}
export function requireLiveCodexSession(
sessions: Map<string, CodexSession>,
sessionId: string
): CodexSession {
const session = sessions.get(sessionId)
if (!session || session.ended) {
throw new Error(`no live codex app-server for session ${sessionId}`)
}
return session
}
export type CodexAcquisitionAttempt = {
@@ -0,0 +1,136 @@
import { describe, expect, it, vi } from 'vitest'
import {
CODEX_APP_SERVER_MAX_RECORD_BYTES,
CodexAppServerFrameSizeError,
CodexAppServerRequestError,
type CodexAppServerConnection
} from './codex-app-server-connection'
import { openCodexThread } from './codex-structured-thread-open'
function connectionFor(
request: CodexAppServerConnection['request']
): Pick<CodexAppServerConnection, 'request'> {
return { request }
}
describe('openCodexThread', () => {
it('requests metadata-only state when resuming an existing thread', async () => {
const request = vi.fn(async () => ({
thread: { id: 'thread-1', path: '/history/thread-1.jsonl' },
model: 'gpt-live'
}))
await expect(
openCodexThread(
connectionFor(request),
{ cwd: '/workspace', resumeThreadId: 'thread-1', resumePath: '/history/thread-1.jsonl' },
2_000
)
).resolves.toMatchObject({ threadId: 'thread-1', model: 'gpt-live' })
expect(request).toHaveBeenCalledWith(
'thread/resume',
{
threadId: 'thread-1',
cwd: '/workspace',
path: '/history/thread-1.jsonl',
excludeTurns: true
},
{ timeoutMs: 2_000 }
)
})
it('caches a narrowly proven excludeTurns refusal and uses one bounded fallback', async () => {
const request = vi.fn(async (_method: string, params?: Record<string, unknown>) => {
if (params?.excludeTurns) {
throw new CodexAppServerRequestError(
'thread/resume',
-32602,
'codex app-server thread/resume failed: unknown field `excludeTurns`'
)
}
return { thread: { id: 'thread-1', turns: [{ id: 'turn-1', items: [] }] } }
})
const connection = connectionFor(request)
const first = await openCodexThread(
connection,
{ cwd: '/workspace', resumeThreadId: 'thread-1' },
2_000
)
const second = await openCodexThread(
connection,
{ cwd: '/workspace', resumeThreadId: 'thread-1' },
2_000
)
expect(first.thread?.turns).toHaveLength(1)
expect(second.thread?.turns).toHaveLength(1)
expect(request.mock.calls.map(([, params]) => params)).toEqual([
expect.objectContaining({ excludeTurns: true }),
{ threadId: 'thread-1', cwd: '/workspace' },
{ threadId: 'thread-1', cwd: '/workspace' }
])
})
it('does not retry ambiguous invalid params or oversized history responses', async () => {
const invalid = new CodexAppServerRequestError(
'thread/resume',
-32602,
'codex app-server thread/resume failed: invalid params'
)
const invalidRequest = vi.fn(async () => {
throw invalid
})
await expect(
openCodexThread(
connectionFor(invalidRequest),
{ cwd: '/workspace', resumeThreadId: 'thread-1' },
2_000
)
).rejects.toBe(invalid)
expect(invalidRequest).toHaveBeenCalledOnce()
const oversized = new CodexAppServerFrameSizeError('thread/resume', 16_777_217, 16_777_216)
const oversizedRequest = vi.fn(async () => {
throw oversized
})
await expect(
openCodexThread(
connectionFor(oversizedRequest),
{ cwd: '/workspace', resumeThreadId: 'thread-1' },
2_000
)
).rejects.toBe(oversized)
expect(oversizedRequest).toHaveBeenCalledOnce()
})
it('refuses an oversized fallback result returned by a connection double', async () => {
const request = vi.fn(async (_method: string, params?: Record<string, unknown>) => {
if (params?.excludeTurns) {
throw new CodexAppServerRequestError(
'thread/resume',
-32602,
'codex app-server thread/resume failed: unsupported excludeTurns parameter'
)
}
return {
thread: {
id: 'thread-1',
turns: [
{ id: 'turn-1', items: [{ output: 'x'.repeat(CODEX_APP_SERVER_MAX_RECORD_BYTES) }] }
]
}
}
})
await expect(
openCodexThread(
connectionFor(request),
{ cwd: '/workspace', resumeThreadId: 'thread-1' },
2_000
)
).rejects.toBeInstanceOf(CodexAppServerFrameSizeError)
expect(request).toHaveBeenCalledTimes(2)
})
})
+64 -13
View File
@@ -5,7 +5,12 @@
// recording it would make the durable handle chain lie about what this session
// actually proved.
import type { CodexAppServerConnection } from './codex-app-server-connection'
import {
CODEX_APP_SERVER_MAX_RECORD_BYTES,
CodexAppServerFrameSizeError,
isCodexAppServerRequestError,
type CodexAppServerConnection
} from './codex-app-server-connection'
import { readCodexThreadId, readCodexThreadPath } from './codex-structured-thread-facts'
export type CodexOpenedThread = {
@@ -21,22 +26,68 @@ function nonEmptyString(value: unknown): string | null {
return typeof value === 'string' && value.trim() ? value : null
}
const resumeMetadataUnsupported = new WeakSet<object>()
function isExcludeTurnsUnsupported(error: unknown): boolean {
return (
isCodexAppServerRequestError(error) &&
error.code === -32602 &&
/(?:unknown|unexpected|unsupported|unrecognized).{0,80}excludeTurns|excludeTurns.{0,80}(?:unknown|unexpected|unsupported|unrecognized)/i.test(
error.message
)
)
}
async function resumeCodexThread(
connection: Pick<CodexAppServerConnection, 'request'>,
params: Record<string, unknown>,
timeoutMs: number | undefined
): Promise<unknown> {
if (resumeMetadataUnsupported.has(connection)) {
return connection.request('thread/resume', params, { timeoutMs })
}
try {
return await connection.request(
'thread/resume',
{ ...params, excludeTurns: true },
{ timeoutMs }
)
} catch (error) {
if (!isExcludeTurnsUnsupported(error)) {
throw error
}
resumeMetadataUnsupported.add(connection)
return connection.request('thread/resume', params, { timeoutMs })
}
}
function assertBoundedAcquisitionResult(method: string, opened: unknown): void {
const encoded = JSON.stringify(opened)
if (encoded === undefined) {
return
}
const encodedBytes = Buffer.byteLength(encoded, 'utf8')
if (encodedBytes > CODEX_APP_SERVER_MAX_RECORD_BYTES) {
throw new CodexAppServerFrameSizeError(method, encodedBytes, CODEX_APP_SERVER_MAX_RECORD_BYTES)
}
}
export async function openCodexThread(
connection: CodexAppServerConnection,
connection: Pick<CodexAppServerConnection, 'request'>,
launch: { cwd: string; resumeThreadId: string | null; resumePath?: string | null },
timeoutMs: number | undefined
): Promise<CodexOpenedThread> {
const opened = await connection.request(
launch.resumeThreadId ? 'thread/resume' : 'thread/start',
launch.resumeThreadId
? {
threadId: launch.resumeThreadId,
cwd: launch.cwd,
...(launch.resumePath ? { path: launch.resumePath } : {})
}
: { cwd: launch.cwd },
{ timeoutMs }
)
const resumeParams = launch.resumeThreadId
? {
threadId: launch.resumeThreadId,
cwd: launch.cwd,
...(launch.resumePath ? { path: launch.resumePath } : {})
}
: null
const opened = resumeParams
? await resumeCodexThread(connection, resumeParams, timeoutMs)
: await connection.request('thread/start', { cwd: launch.cwd }, { timeoutMs })
assertBoundedAcquisitionResult(resumeParams ? 'thread/resume' : 'thread/start', opened)
const threadId = readCodexThreadId(opened)
if (!threadId) {
throw new Error('codex app-server did not name the thread it opened')
+148
View File
@@ -0,0 +1,148 @@
import {
boundPayload,
digestPayload
} from '../native-chat/agent-session-journal/journal-payload-bounds'
/** Maximum forgotten turn keys retained for late-frame reconciliation. */
export const MAX_CODEX_TURN_ORDINAL_ENTRIES = 256
export const MAX_CODEX_TURN_ORDINAL_BYTES = 512 * 1024
/** Assigns stable message ordinals while retaining a bounded late-frame window. */
export class CodexTurnOrdinals {
private readonly turns = new Map<
string,
{ assigned: Map<string, number>; next: number; active: boolean }
>()
private retainedBytes = 0
get forgottenTurnCount(): number {
let count = 0
for (const turn of this.turns.values()) {
if (!turn.active) {
count += 1
}
}
return count
}
get bytes(): number {
return this.retainedBytes
}
private keyPart(value: string): string {
const encoded = encodeURIComponent(value)
if (Buffer.byteLength(encoded, 'utf8') <= 256) {
return encoded
}
const suffix = `#${digestPayload(value).slice(0, 24)}`
return `${
boundPayload(encoded, {
inlineHeadBytes: 256 - Buffer.byteLength(suffix, 'utf8'),
maxSessionBytes: Number.MAX_SAFE_INTEGER,
maxAppendsPerWindow: Number.MAX_SAFE_INTEGER,
appendWindowMs: Number.MAX_SAFE_INTEGER
}).head
}${suffix}`
}
private turnKey(threadId: string, turnId: string): string {
return `${this.keyPart(threadId)}:${this.keyPart(turnId)}`
}
private trimForgotten(): void {
while (this.forgottenTurnCount > MAX_CODEX_TURN_ORDINAL_ENTRIES) {
const oldest = [...this.turns.entries()].find(([, turn]) => !turn.active)?.[0]
if (!oldest) {
break
}
const removed = this.turns.get(oldest)
this.turns.delete(oldest)
if (removed) {
this.retainedBytes = Math.max(
0,
this.retainedBytes -
Buffer.byteLength(oldest, 'utf8') -
[...removed.assigned.keys()].reduce((n, key) => n + Buffer.byteLength(key, 'utf8'), 0)
)
}
}
}
private trimBytes(currentTurnKey: string): void {
this.trimForgotten()
while (this.retainedBytes > MAX_CODEX_TURN_ORDINAL_BYTES) {
const forgotten = [...this.turns.entries()].find(([, turn]) => !turn.active)?.[0]
const oldest = forgotten ?? this.turns.keys().next().value
if (typeof oldest !== 'string') {
break
}
const turn = this.turns.get(oldest)
if (oldest === currentTurnKey && this.turns.size === 1 && turn) {
const itemKey = turn.assigned.keys().next().value
if (typeof itemKey !== 'string') {
break
}
turn.assigned.delete(itemKey)
this.retainedBytes = Math.max(
Buffer.byteLength(currentTurnKey, 'utf8'),
this.retainedBytes - Buffer.byteLength(itemKey, 'utf8')
)
continue
}
if (!turn) {
break
}
this.turns.delete(oldest)
this.retainedBytes = Math.max(
0,
this.retainedBytes -
Buffer.byteLength(oldest, 'utf8') -
[...turn.assigned.keys()].reduce((n, key) => n + Buffer.byteLength(key, 'utf8'), 0)
)
}
}
ordinalFor(threadId: string, turnId: string, codexItemId: string): number {
const turnKey = this.turnKey(threadId, turnId)
let turn = this.turns.get(turnKey)
if (!turn) {
turn = { assigned: new Map(), next: 0, active: true }
this.turns.set(turnKey, turn)
this.retainedBytes += Buffer.byteLength(turnKey, 'utf8')
} else {
if (!turn.active) {
this.turns.delete(turnKey)
this.turns.set(turnKey, turn)
}
turn.active = true
}
const itemKey = this.keyPart(codexItemId)
const existing = turn.assigned.get(itemKey)
if (existing !== undefined) {
return existing
}
const ordinal = turn.next
turn.assigned.set(itemKey, ordinal)
this.retainedBytes += Buffer.byteLength(itemKey, 'utf8')
turn.next += 1
this.trimBytes(turnKey)
return ordinal
}
forgetTurn(threadId: string, turnId: string): void {
const turnKey = this.turnKey(threadId, turnId)
const turn = this.turns.get(turnKey)
if (turn) {
const assignedBytes = [...turn.assigned.keys()].reduce(
(n, key) => n + Buffer.byteLength(key, 'utf8'),
0
)
turn.assigned = new Map()
turn.active = false
this.retainedBytes = Math.max(0, this.retainedBytes - assignedBytes)
this.turns.delete(turnKey)
this.turns.set(turnKey, turn)
this.trimForgotten()
}
}
}
+114
View File
@@ -5,6 +5,7 @@ import {
NDJSON_MAX_LINE_BYTES,
NdjsonLineTooLongError
} from './ndjson'
import { createIncrementalNdjsonFramer } from '../../shared/main-process-ndjson-framer'
describe('encodeNdjson', () => {
it('encodes an object as a JSON line ending with newline', () => {
@@ -171,4 +172,117 @@ describe('createNdjsonParser', () => {
expect(onMessage).toHaveBeenCalledOnce()
expect(onMessage).toHaveBeenCalledWith({ fresh: true })
})
it('retains a valid suffix when paused input overflows after an oversized partial line', () => {
const records: unknown[] = []
const rejected: unknown[] = []
let paused = true
const framer = createIncrementalNdjsonFramer(
(record) => records.push(record),
(error) => rejected.push(error),
{ maxLineBytes: 32, shouldPause: () => paused }
)
// The first record leaves a partial line in the paused remainder.
framer.feed('{}\nx')
framer.feed(`${'y'.repeat(70_000)}\n{"good":true}\n`)
paused = false
framer.resume()
expect(rejected).toHaveLength(1)
expect(records).toEqual([{}, { good: true }])
})
it('queues many complete records while paused without treating them as one oversized suffix', () => {
const records: unknown[] = []
let paused = true
const framer = createIncrementalNdjsonFramer(
(record) => records.push(record),
() => {
throw new Error('complete records should not be rejected')
},
{ shouldPause: () => paused }
)
const count = 100_000
framer.feed(`${JSON.stringify({ index: 0 })}\n`)
framer.feed(
Array.from({ length: count }, (_, index) => `${JSON.stringify({ index: index + 1 })}\n`).join(
''
)
)
paused = false
framer.resume()
expect(records).toHaveLength(count + 1)
expect(records.at(-1)).toEqual({ index: count })
})
it('does not drop data fed after queued records when the consumer resumes', () => {
const records: unknown[] = []
let paused = true
const framer = createIncrementalNdjsonFramer(
(record) => records.push(record),
(error) => {
throw error
},
{ shouldPause: () => paused }
)
framer.feed('{"queued":true}\n')
paused = false
framer.feed('{"after":true}\n')
expect(records).toEqual([{ queued: true }, { after: true }])
})
it('does not dispatch a pending suffix ahead of queued records after re-pause', () => {
const records: unknown[] = []
let paused = true
const framer = createIncrementalNdjsonFramer(
(record) => {
records.push(record)
if ((record as { index?: number }).index === 1) {
paused = true
}
},
(error) => {
throw new Error(`unexpected rejection: ${JSON.stringify(error)}`)
},
{ shouldPause: () => paused }
)
framer.feed('{"index":0}\n{"index":1}\n{"index":2}\n{"index":')
paused = false
framer.resume()
expect(records).toEqual([{ index: 0 }, { index: 1 }])
paused = false
framer.resume()
expect(records).toEqual([{ index: 0 }, { index: 1 }, { index: 2 }])
framer.feed('3}\n')
expect(records).toEqual([{ index: 0 }, { index: 1 }, { index: 2 }, { index: 3 }])
})
it('caps an actually incomplete paused suffix and recovers at the next delimiter', () => {
const records: unknown[] = []
const rejected: unknown[] = []
let paused = true
const framer = createIncrementalNdjsonFramer(
(record) => records.push(record),
(error) => rejected.push(error),
{ maxLineBytes: 32, shouldPause: () => paused }
)
framer.feed('{}\nx')
framer.feed('y'.repeat(70_000))
paused = false
framer.resume()
framer.feed('\n{"recovered":true}\n')
expect(rejected).toHaveLength(1)
expect(records).toEqual([{}, { recovered: true }])
})
})
+7 -111
View File
@@ -1,111 +1,7 @@
export const NDJSON_MAX_LINE_BYTES = 16 * 1024 * 1024
export class NdjsonLineTooLongError extends Error {
constructor(
readonly lineBytes: number,
readonly maxLineBytes: number
) {
super(`NDJSON line exceeds max ${maxLineBytes} bytes (${lineBytes} bytes encoded)`)
this.name = 'NdjsonLineTooLongError'
}
}
export function encodeNdjson(msg: unknown, maxLineBytes = NDJSON_MAX_LINE_BYTES): string {
const line = JSON.stringify(msg)
const lineBytes = Buffer.byteLength(line, 'utf8')
if (lineBytes > maxLineBytes) {
throw new NdjsonLineTooLongError(lineBytes, maxLineBytes)
}
return `${line}\n`
}
export type NdjsonParser = {
feed(chunk: string): void
reset(): void
}
export type NdjsonParserOptions = {
maxLineBytes?: number
}
export function createNdjsonParser(
onMessage: (msg: unknown) => void,
onError?: (err: Error) => void,
options: NdjsonParserOptions = {}
): NdjsonParser {
let buffer = ''
let bufferBytes = 0
let discardingOversizedLine = false
const maxLineBytes = Math.max(1, options.maxLineBytes ?? NDJSON_MAX_LINE_BYTES)
const clearBuffer = (): void => {
buffer = ''
bufferBytes = 0
}
const reportOversizedLine = (observedBytes: number): void => {
onError?.(
new Error(`NDJSON line exceeds max ${maxLineBytes} bytes (${observedBytes} bytes received)`)
)
}
return {
feed(chunk: string): void {
let remaining = chunk
while (remaining.length > 0) {
const newlineIndex = remaining.indexOf('\n')
const hasNewline = newlineIndex !== -1
const segment = hasNewline ? remaining.slice(0, newlineIndex) : remaining
remaining = hasNewline ? remaining.slice(newlineIndex + 1) : ''
if (discardingOversizedLine) {
if (hasNewline) {
discardingOversizedLine = false
clearBuffer()
continue
}
return
}
const segmentBytes = Buffer.byteLength(segment, 'utf8')
const nextLineBytes = bufferBytes + segmentBytes
// Why: daemon sockets are local but persistent; a peer that never sends
// a newline must not grow the parser buffer without bound.
if (nextLineBytes > maxLineBytes) {
reportOversizedLine(nextLineBytes)
clearBuffer()
if (!hasNewline) {
discardingOversizedLine = true
return
}
continue
}
buffer += segment
bufferBytes = nextLineBytes
if (!hasNewline) {
return
}
const line = buffer
clearBuffer()
if (line.length === 0) {
continue
}
try {
onMessage(JSON.parse(line))
} catch (err) {
onError?.(err instanceof Error ? err : new Error(String(err)))
}
}
},
reset(): void {
clearBuffer()
discardingOversizedLine = false
}
}
}
export {
createNdjsonParser,
encodeNdjson,
NDJSON_MAX_LINE_BYTES,
NdjsonLineTooLongError
} from '../../shared/main-process-ndjson-framer'
export type { NdjsonParser, NdjsonParserOptions } from '../../shared/main-process-ndjson-framer'
@@ -9,13 +9,13 @@ import { mkdir, readFile, readdir, rm, stat } from 'node:fs/promises'
import { join } from 'node:path'
import { durableWriteTempPath, writeFileDurable } from '../../durable-file-write'
const BLOB_DIR = 'blobs'
export const JOURNAL_BLOB_DIR = 'blobs'
const DIGEST_PATTERN = /^[0-9a-f]{64}$/
/** A digest arrives back from a row on disk, so it is untrusted by the time it
* reaches the filesystem: anything but a bare sha256 could escape the store. */
function blobPath(journalDir: string, digest: string): string | null {
return DIGEST_PATTERN.test(digest) ? join(journalDir, BLOB_DIR, digest) : null
return DIGEST_PATTERN.test(digest) ? join(journalDir, JOURNAL_BLOB_DIR, digest) : null
}
/** Persist `payload` under its digest. Returns the digest so the caller can
@@ -33,7 +33,7 @@ export async function putJournalBlob(
if (await pathExists(target)) {
return digest
}
await mkdir(join(journalDir, BLOB_DIR), { recursive: true })
await mkdir(join(journalDir, JOURNAL_BLOB_DIR), { recursive: true })
await writeFileDurable(durableWriteTempPath(target), target, payload)
return digest
}
@@ -68,7 +68,7 @@ export async function pruneJournalBlobs(
let removed = 0
let names: string[]
try {
names = await readdir(join(journalDir, BLOB_DIR))
names = await readdir(join(journalDir, JOURNAL_BLOB_DIR))
} catch {
return 0
}
@@ -76,7 +76,7 @@ export async function pruneJournalBlobs(
if (retained.has(name)) {
continue
}
await rm(join(journalDir, BLOB_DIR, name), { force: true }).catch(() => {})
await rm(join(journalDir, JOURNAL_BLOB_DIR, name), { force: true }).catch(() => {})
removed += 1
}
return removed
@@ -90,3 +90,21 @@ async function pathExists(path: string): Promise<boolean> {
return false
}
}
export async function journalBlobFileSize(
journalDir: string,
digest: string
): Promise<number | null> {
const target = blobPath(journalDir, digest)
if (!target) {
return null
}
try {
return (await stat(target)).size
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return null
}
throw error
}
}
@@ -8,12 +8,7 @@
// The retained tail must cover the longest reconnect window Orca supports, or a
// client that was merely asleep gets a full snapshot reload instead of a resume.
import {
blobDigestsInBody,
referencedBlobDigests,
renderJournalState,
type JournalReducerState
} from './journal-reducer'
import { blobDigestsInBody, renderJournalState, type JournalReducerState } from './journal-reducer'
import { pruneJournalBlobs } from './journal-blob-store'
import {
rewriteJournalLog,
@@ -23,6 +18,7 @@ import {
import { AGENT_SESSION_JOURNAL_SCHEMA_VERSION } from '../../../shared/agent-session-journal-types'
import type { JournalRow } from './journal-row-schema'
import { AgentSessionJournalError } from './journal-write-guards'
import { assertJournalPhysicalCapacity, journalDirectoryBytes } from './journal-physical-quota'
export type JournalCompactionPolicy = {
/** Always keep at least this many rows, however old they are. */
@@ -55,11 +51,14 @@ export type JournalCompactionResult = {
export async function compactJournal(input: {
journalDir: string
/** Parent quota root when compacting an in-directory staging journal. */
physicalQuotaRoot?: string
state: JournalReducerState
tailRows: readonly JournalRow[]
policy?: JournalCompactionPolicy
now: number
maxSessionBytes: number
sessionId?: string
}): Promise<JournalCompactionResult> {
const policy = input.policy ?? DEFAULT_JOURNAL_COMPACTION_POLICY
const retained = retainTail(input.tailRows, policy, input.now)
@@ -88,6 +87,7 @@ export async function compactJournal(input: {
itemId,
revision
})),
appliedSettlementIds: [...input.state.appliedSettlementIds],
tail: retained
}
@@ -99,18 +99,52 @@ export async function compactJournal(input: {
)
}
const sessionId = input.sessionId ?? input.state.sessionId
const quotaRoot = input.physicalQuotaRoot ?? input.journalDir
const retainedLogBytes = retained.reduce(
(total, row) => total + Buffer.byteLength(JSON.stringify(row), 'utf8') + 1,
0
)
// Durable writes keep the old final alongside the new temp until rename.
// Reserve the complete compaction peak up front so a later copy cannot leave
// a half-published snapshot/log pair when the quota is tight.
await assertJournalPhysicalCapacity({
journalDir: quotaRoot,
sessionId,
maxBytes: input.maxSessionBytes,
peakAdditionalBytes: snapshotBytes + retainedLogBytes
})
await writeJournalSnapshotFile(input.journalDir, snapshot)
await rewriteJournalLog(input.journalDir, retained)
// Blobs are pruned last: a crash before this leaks bytes, whereas pruning
// first would strand a snapshot pointing at a payload that no longer exists.
const retainedDigests = referencedBlobDigests(input.state)
// Recompute from exactly what the durable snapshot and retained log carry;
// this preserves reused/pre-existing blobs while allowing stale payloads to
// be pruned safely after both files are published.
const retainedDigests = new Set<string>()
for (const item of snapshot.items) {
blobDigestsInBody(item.body, retainedDigests)
}
for (const row of retained) {
if (row.kind === 'item') {
blobDigestsInBody(row.body, retainedDigests)
} else if (row.kind === 'lifecycle-batch') {
for (const mutation of row.mutations) {
if (mutation.kind === 'item') {
blobDigestsInBody(mutation.body, retainedDigests)
}
}
}
}
await pruneJournalBlobs(input.journalDir, retainedDigests)
if ((await journalDirectoryBytes(quotaRoot)) > input.maxSessionBytes) {
throw new AgentSessionJournalError(
'journal_bound_exceeded',
`agent-session journal for ${sessionId} exceeds its physical bound after compaction`
)
}
return {
tailRows: retained,
compactedThrough,
@@ -11,16 +11,36 @@ import {
rewriteJournalLog
} from './journal-log-file'
import type { JournalRow } from './journal-row-schema'
import { assertJournalPhysicalCapacity } from './journal-physical-quota'
/** Keep the readable prefix and set the unreadable suffix aside. */
export async function quarantineCorruptSuffix(
journalDir: string,
retainedRows: readonly JournalRow[],
remainder: string | undefined
remainder: string | undefined,
quota?: { sessionId: string; maxBytes: number }
): Promise<void> {
if (remainder) {
if (quota) {
await assertJournalPhysicalCapacity({
journalDir,
...quota,
peakAdditionalBytes: Buffer.byteLength(remainder, 'utf8')
})
}
await quarantineJournalRemainder(journalDir, remainder)
}
if (quota) {
const retainedBytes = retainedRows.reduce(
(total, row) => total + Buffer.byteLength(JSON.stringify(row), 'utf8') + 1,
0
)
await assertJournalPhysicalCapacity({
journalDir,
...quota,
peakAdditionalBytes: retainedBytes
})
}
await rewriteJournalLog(journalDir, retainedRows)
}
@@ -28,7 +48,10 @@ export async function quarantineCorruptSuffix(
* schema: those rows are unreadable to THIS build, not worthless. The
* snapshot is preserved as raw bytes — a future-version snapshot does not
* parse under this build's schema, and its bytes must survive verbatim. */
export async function quarantineUnreadableSchema(journalDir: string): Promise<void> {
export async function quarantineUnreadableSchema(
journalDir: string,
quota?: { sessionId: string; maxBytes: number }
): Promise<void> {
const snapshot = await readSnapshotBytes(journalDir)
const log = await readJournalLog(journalDir)
const preserved = [
@@ -39,6 +62,13 @@ export async function quarantineUnreadableSchema(journalDir: string): Promise<vo
.filter(Boolean)
.join('\n')
if (preserved) {
if (quota) {
await assertJournalPhysicalCapacity({
journalDir,
...quota,
peakAdditionalBytes: Buffer.byteLength(preserved, 'utf8')
})
}
await quarantineJournalRemainder(journalDir, preserved)
}
}
@@ -0,0 +1,87 @@
import type {
AgentJournalCursor,
AgentSessionJournalIdentity
} from '../../../shared/agent-session-journal-types'
import type { JournalCompactionPolicy } from './journal-compaction'
import { quarantineUnreadableSchema } from './journal-corruption-quarantine'
import { replaceJournalEpoch, type JournalReplacementItem } from './journal-epoch-replacement'
import { publishNewEpoch } from './journal-epoch-rollover'
import type { JournalLoad } from './journal-open'
import type { AgentJournalEpochReason } from './journal-row-schema'
import {
assertJournalFence,
assertJournalWritable,
type JournalAppendBudget
} from './journal-write-guards'
export class JournalEpochController {
constructor(
private readonly deps: {
identity: AgentSessionJournalIdentity
journalDir: string
budget: JournalAppendBudget
compaction: JournalCompactionPolicy
now: () => number
mintEpoch: () => string
serialize: <T>(run: () => Promise<T>) => Promise<T>
readOnly: () => boolean
setReadOnly: (readOnly: boolean) => void
highestFence: () => number
cursor: () => AgentJournalCursor
adopt: (loaded: JournalLoad) => void
}
) {}
async start(reason: AgentJournalEpochReason, fence: number): Promise<void> {
this.deps.adopt(
await publishNewEpoch({
journalDir: this.deps.journalDir,
sessionId: this.deps.identity.sessionId,
providerHandle: this.deps.identity.providerHandle,
epoch: this.deps.mintEpoch(),
reason,
fence,
now: this.deps.now(),
maxSessionBytes: this.deps.budget.maxSessionBytes
})
)
}
async roll(reason: AgentJournalEpochReason, fence: number): Promise<AgentJournalCursor> {
if (reason !== 'schema_unreadable') {
assertJournalWritable(this.deps.readOnly(), this.deps.identity.sessionId)
} else if (this.deps.readOnly()) {
await quarantineUnreadableSchema(this.deps.journalDir, {
sessionId: this.deps.identity.sessionId,
maxBytes: this.deps.budget.maxSessionBytes
})
}
await this.start(reason, fence)
this.deps.setReadOnly(false)
return this.deps.cursor()
}
replace(
reason: AgentJournalEpochReason,
fence: number,
items: readonly JournalReplacementItem[]
): Promise<AgentJournalCursor> {
return this.deps.serialize(async () => {
assertJournalWritable(this.deps.readOnly(), this.deps.identity.sessionId)
assertJournalFence(fence, this.deps.highestFence())
await replaceJournalEpoch({
journalDir: this.deps.journalDir,
identity: this.deps.identity,
reason,
fence,
items,
budget: this.deps.budget.fork(),
compaction: this.deps.compaction,
now: this.deps.now,
mintEpoch: this.deps.mintEpoch,
onSnapshotPublished: this.deps.adopt
})
return this.deps.cursor()
})
}
}
@@ -0,0 +1,173 @@
import { mkdtemp, readdir, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import type {
AgentJournalItemBody,
AgentSessionJournalIdentity
} from '../../../shared/agent-session-journal-types'
import { DEFAULT_JOURNAL_COMPACTION_POLICY } from './journal-compaction'
import { replaceJournalEpoch } from './journal-epoch-replacement'
import { putJournalBlob, readJournalBlob } from './journal-blob-store'
import { boundPayload, DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
import { journalDirectoryBytes } from './journal-physical-quota'
import { JournalAppendBudget } from './journal-write-guards'
import { openAgentSessionJournal } from './journal-store'
const IDENTITY: AgentSessionJournalIdentity = {
sessionId: 'session-1',
workspaceId: 'ws-1',
hostId: 'host-1',
agent: 'codex',
providerHandle: { kind: 'codex', threadId: 'thread-1' }
}
let root: string
let clock = 1_000
beforeEach(async () => {
root = await mkdtemp(join(tmpdir(), 'orca-journal-replace-'))
clock = 1_000
})
afterEach(async () => {
await rm(root, { recursive: true, force: true })
})
function now(): number {
clock += 1
return clock
}
function toolBody(output: ReturnType<typeof boundPayload>): AgentJournalItemBody {
return {
kind: 'tool-call',
name: 'shell',
input: {},
state: 'completed',
output
}
}
describe('journal epoch replacement', () => {
it('publishes one observable replacement and prunes stale root blobs afterward', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, inlineHeadBytes: 8 }
const stalePayload = 'stale'.repeat(1_000)
const retainedPayload = 'retained'.repeat(1_000)
const stale = boundPayload(stalePayload, limits)
const retained = boundPayload(retainedPayload, limits)
const published: unknown[] = []
await putJournalBlob(root, stale.digest, stalePayload)
await replaceJournalEpoch({
journalDir: root,
identity: IDENTITY,
reason: 'handle_forked',
fence: 2,
items: [
{
identity: { provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal: 0 },
body: toolBody(retained),
blobs: [{ digest: retained.digest, payload: retainedPayload }]
}
],
budget: new JournalAppendBudget(IDENTITY.sessionId, {
...limits,
maxSessionBytes: 512 * 1024
}),
compaction: DEFAULT_JOURNAL_COMPACTION_POLICY,
now,
mintEpoch: () => 'epoch-new',
onSnapshotPublished: (loaded) => published.push(loaded)
})
expect(published).toHaveLength(1)
expect(await readJournalBlob(root, stale.digest)).toBeNull()
expect(await readJournalBlob(root, retained.digest)).toBe(retainedPayload)
expect((published[0] as { sizeBytes: number }).sizeBytes).toBe(
await journalDirectoryBytes(root)
)
})
it('keeps root blobs and reports no publication when replacement never becomes authoritative', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, inlineHeadBytes: 8, maxSessionBytes: 6_000 }
const stalePayload = 'stale'.repeat(500)
const stale = boundPayload(stalePayload, limits)
const published: unknown[] = []
await putJournalBlob(root, stale.digest, stalePayload)
await expect(
replaceJournalEpoch({
journalDir: root,
identity: IDENTITY,
reason: 'handle_forked',
fence: 2,
items: [
{
identity: { provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal: 0 },
body: {
kind: 'message',
role: 'assistant',
blocks: [{ type: 'text', text: 'x'.repeat(10_000) }]
}
}
],
budget: new JournalAppendBudget(IDENTITY.sessionId, limits),
compaction: DEFAULT_JOURNAL_COMPACTION_POLICY,
now,
mintEpoch: () => 'epoch-new',
onSnapshotPublished: (loaded) => published.push(loaded)
})
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(published).toHaveLength(0)
expect(await readJournalBlob(root, stale.digest)).toBe(stalePayload)
expect((await readdir(root)).some((name) => name.startsWith('.epoch-replacement-'))).toBe(false)
})
it('charges replacement blobs cumulatively and rolls back staging on quota refusal', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, inlineHeadBytes: 8, maxSessionBytes: 7_000 }
const journal = await openAgentSessionJournal({
identity: IDENTITY,
journalDir: root,
limits,
autoCompact: false,
now,
mintEpoch: () => `epoch-${clock}`
})
const existingPayload = 'existing'.repeat(250)
const existing = boundPayload(existingPayload, limits)
await journal.appendItemWithBlobs(
{ provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal: 0 },
toolBody(existing),
[{ digest: existing.digest, payload: existingPayload }],
{ fence: 1 }
)
const replacementPayload = 'replacement'.repeat(200)
const replacement = boundPayload(replacementPayload, limits)
const secondPayload = 'second'.repeat(200)
const second = boundPayload(secondPayload, limits)
await expect(
journal.replaceEpochItems('handle_forked', 2, [
{
identity: { provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal: 1 },
body: toolBody(replacement),
blobs: [{ digest: replacement.digest, payload: replacementPayload }]
},
{
identity: { provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal: 2 },
body: toolBody(second),
blobs: [{ digest: second.digest, payload: secondPayload }]
}
])
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(journal.epoch).toMatch(/^epoch-/)
expect(await readJournalBlob(root, existing.digest)).toBe(existingPayload)
expect(await readJournalBlob(root, replacement.digest)).toBeNull()
expect(await readJournalBlob(root, second.digest)).toBeNull()
expect((await readdir(root)).some((name) => name.startsWith('.epoch-replacement-'))).toBe(false)
expect(await journalDirectoryBytes(root)).toBeLessThanOrEqual(limits.maxSessionBytes)
})
})
@@ -1,4 +1,4 @@
import { mkdtemp, rm } from 'node:fs/promises'
import { mkdir, mkdtemp, rm, stat } from 'node:fs/promises'
import { join } from 'node:path'
import { copyFileDurable } from '../../durable-file-write'
import type {
@@ -8,16 +8,31 @@ import type {
} from '../../../shared/agent-session-journal-types'
import { compactJournal, type JournalCompactionPolicy } from './journal-compaction'
import { JOURNAL_LOG_FILE, JOURNAL_SNAPSHOT_FILE, appendJournalRows } from './journal-log-file'
import { applyJournalRow, createJournalReducerState } from './journal-reducer'
import {
applyJournalRow,
blobDigestsInBody,
createJournalReducerState,
referencedBlobDigests,
type JournalReducerState
} from './journal-reducer'
import { buildJournalItemRow, journalRowBase } from './journal-row-builders'
import type { AgentJournalEpochReason, JournalRow } from './journal-row-schema'
import { journalRowByteLength } from './journal-row-schema'
import { assertJournalFence, type JournalAppendBudget } from './journal-write-guards'
import type { JournalLoad } from './journal-open'
import { assertJournalPhysicalCapacity, journalDirectoryBytes } from './journal-physical-quota'
import {
JOURNAL_BLOB_DIR,
journalBlobFileSize,
putJournalBlob,
pruneJournalBlobs,
removeJournalBlob
} from './journal-blob-store'
export type JournalReplacementItem = {
identity: AgentJournalItemIdentity
body: AgentJournalItemBody
blobs?: readonly { digest: string; payload: string }[]
observedAt?: number
}
@@ -34,6 +49,11 @@ export async function replaceJournalEpoch(input: {
onSnapshotPublished: (loaded: JournalLoad) => void
}): Promise<void> {
const stagingDir = await mkdtemp(join(input.journalDir, '.epoch-replacement-'))
const stagedBlobDigests = new Set<string>()
const publishedBlobDigests: string[] = []
let snapshotPublished = false
let adoptionReported = false
let publishedLoad: JournalLoad | null = null
try {
const epoch = input.mintEpoch()
const state = createJournalReducerState(input.identity.sessionId, epoch)
@@ -46,9 +66,18 @@ export async function replaceJournalEpoch(input: {
const rows: JournalRow[] = [epochRow]
applyJournalRow(state, epochRow)
let sizeBytes = journalRowByteLength(epochRow)
await assertStagingCapacity(input, sizeBytes)
await appendJournalRows(stagingDir, [epochRow])
for (const item of input.items) {
sizeBytes += await stageReplacementBlobs({
journalDir: input.journalDir,
stagingDir,
identity: input.identity,
budget: input.budget,
stagedBlobDigests,
blobs: item.blobs ?? []
})
const appendTime = input.now()
const row = buildJournalItemRow({
state,
@@ -60,6 +89,7 @@ export async function replaceJournalEpoch(input: {
})
assertJournalFence(row.fence, state.highestFence)
input.budget.assert(row, appendTime, sizeBytes)
await assertStagingCapacity(input, journalRowByteLength(row))
await appendJournalRows(stagingDir, [row])
applyJournalRow(state, row)
rows.push(row)
@@ -68,36 +98,201 @@ export async function replaceJournalEpoch(input: {
const compacted = await compactJournal({
journalDir: stagingDir,
physicalQuotaRoot: input.journalDir,
state,
tailRows: rows,
policy: input.compaction,
now: input.now(),
maxSessionBytes: input.budget.maxSessionBytes
})
await publishPreparedFile(stagingDir, input.journalDir, JOURNAL_SNAPSHOT_FILE)
// All destination publishes use durable temp files while the staging
// source and existing finals remain present. Reserve the whole publication
// peak before touching the live epoch so a later file cannot fail halfway
// through replacement.
const stagedSnapshotBytes = (await stat(join(stagingDir, JOURNAL_SNAPSHOT_FILE))).size
const stagedLogBytes = (await stat(join(stagingDir, JOURNAL_LOG_FILE))).size
let stagedPublishBytes = stagedSnapshotBytes + stagedLogBytes
for (const digest of stagedBlobDigests) {
stagedPublishBytes += (await stat(join(stagingDir, JOURNAL_BLOB_DIR, digest))).size
}
await assertJournalPhysicalCapacity({
journalDir: input.journalDir,
sessionId: input.identity.sessionId,
maxBytes: input.budget.maxSessionBytes,
peakAdditionalBytes: stagedPublishBytes
})
for (const digest of stagedBlobDigests) {
if (
await publishPreparedBlob(
stagingDir,
input.journalDir,
digest,
input.identity.sessionId,
input.budget.maxSessionBytes
)
) {
publishedBlobDigests.push(digest)
}
}
await publishPreparedFile(
stagingDir,
input.journalDir,
JOURNAL_SNAPSHOT_FILE,
input.identity.sessionId,
input.budget.maxSessionBytes
)
snapshotPublished = true
state.oldestSequence = compacted.oldestSequence
input.onSnapshotPublished({
publishedLoad = {
state,
tailRows: compacted.tailRows,
compactedThrough: compacted.compactedThrough,
readOnly: false,
corrupt: false,
malformedRows: 0,
sizeBytes: compacted.tailRows.reduce((total, row) => total + journalRowByteLength(row), 0)
})
await publishPreparedFile(stagingDir, input.journalDir, JOURNAL_LOG_FILE)
sizeBytes: 0
}
await publishPreparedFile(
stagingDir,
input.journalDir,
JOURNAL_LOG_FILE,
input.identity.sessionId,
input.budget.maxSessionBytes
)
await pruneJournalBlobs(
input.journalDir,
replacementRetainedBlobDigests(state, compacted.tailRows)
)
} finally {
if (!snapshotPublished) {
for (const digest of publishedBlobDigests) {
await removeJournalBlob(input.journalDir, digest)
}
}
await rm(stagingDir, { recursive: true, force: true })
if (snapshotPublished && publishedLoad && !adoptionReported) {
adoptionReported = true
input.onSnapshotPublished({
...publishedLoad,
sizeBytes: await journalDirectoryBytes(input.journalDir)
})
}
}
}
function replacementRetainedBlobDigests(
state: JournalReducerState,
tailRows: readonly JournalRow[]
): Set<string> {
const retained = referencedBlobDigests(state)
for (const row of tailRows) {
if (row.kind === 'item') {
blobDigestsInBody(row.body, retained)
} else if (row.kind === 'lifecycle-batch') {
for (const mutation of row.mutations) {
if (mutation.kind === 'item') {
blobDigestsInBody(mutation.body, retained)
}
}
}
}
return retained
}
async function stageReplacementBlobs(input: {
journalDir: string
stagingDir: string
identity: AgentSessionJournalIdentity
budget: JournalAppendBudget
stagedBlobDigests: Set<string>
blobs: readonly { digest: string; payload: string }[]
}): Promise<number> {
const toStage: { digest: string; payload: string; bytes: number }[] = []
const unique = new Map(input.blobs.map((blob) => [blob.digest, blob]))
for (const blob of unique.values()) {
if (input.stagedBlobDigests.has(blob.digest)) {
continue
}
if ((await journalBlobFileSize(input.journalDir, blob.digest)) !== null) {
continue
}
const bytes = Buffer.byteLength(blob.payload, 'utf8')
toStage.push({ ...blob, bytes })
}
// Reserve all new payloads together. The staging directory lives under the
// journal root, so the capacity check includes existing session bytes and
// every other .epoch-replacement-* directory already present.
const stagedBytes = toStage.reduce((total, blob) => total + blob.bytes, 0)
await assertStagingCapacity(input, stagedBytes)
for (const blob of toStage) {
await putJournalBlob(input.stagingDir, blob.digest, blob.payload)
input.stagedBlobDigests.add(blob.digest)
}
return stagedBytes
}
function assertStagingCapacity(
input: {
journalDir: string
identity: AgentSessionJournalIdentity
budget: JournalAppendBudget
},
additionalBytes: number
): Promise<number> {
return assertJournalPhysicalCapacity({
journalDir: input.journalDir,
sessionId: input.identity.sessionId,
maxBytes: input.budget.maxSessionBytes,
peakAdditionalBytes: additionalBytes
})
}
async function publishPreparedFile(
stagingDir: string,
journalDir: string,
fileName: string
fileName: string,
sessionId: string,
maxBytes: number
): Promise<void> {
await assertJournalPhysicalCapacity({
journalDir,
sessionId,
maxBytes,
peakAdditionalBytes: (await stat(join(stagingDir, fileName))).size
})
const copied = await copyFileDurable(join(stagingDir, fileName), join(journalDir, fileName))
if (!copied) {
throw new Error(`prepared journal file disappeared before publish: ${fileName}`)
}
}
async function publishPreparedBlob(
stagingDir: string,
journalDir: string,
digest: string,
sessionId: string,
maxBytes: number
): Promise<boolean> {
if ((await journalBlobFileSize(journalDir, digest)) !== null) {
return false
}
const size = await journalBlobFileSize(stagingDir, digest)
if (size === null) {
throw new Error(`prepared journal blob disappeared before publish: ${digest}`)
}
await assertJournalPhysicalCapacity({
journalDir,
sessionId,
maxBytes,
peakAdditionalBytes: size
})
await mkdir(join(journalDir, JOURNAL_BLOB_DIR), { recursive: true })
const copied = await copyFileDurable(
join(stagingDir, JOURNAL_BLOB_DIR, digest),
join(journalDir, JOURNAL_BLOB_DIR, digest)
)
if (!copied) {
throw new Error(`prepared journal blob disappeared before publish: ${digest}`)
}
return true
}
@@ -22,6 +22,7 @@ export async function publishNewEpoch(input: {
reason: AgentJournalEpochReason
fence: number
now: number
maxSessionBytes?: number
}): Promise<JournalLoad> {
const row: JournalRow = {
kind: 'epoch',
@@ -40,7 +41,8 @@ export async function publishNewEpoch(input: {
tailRows: [row],
policy: { minTailRows: 1, retainTailMs: Number.POSITIVE_INFINITY },
now: input.now,
maxSessionBytes: DEFAULT_JOURNAL_PAYLOAD_LIMITS.maxSessionBytes
maxSessionBytes: input.maxSessionBytes ?? DEFAULT_JOURNAL_PAYLOAD_LIMITS.maxSessionBytes,
sessionId: input.sessionId
})
applyJournalRow(state, row)
state.oldestSequence = 1
@@ -0,0 +1,69 @@
import { agentJournalItemKey } from '../../../shared/agent-session-journal-item-key'
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../../shared/agent-session-journal-types'
import { journalItemRowBuilder } from './journal-row-builders'
import type { JournalReducerState } from './journal-reducer'
import type { AgentSessionJournal } from './journal-store'
import type { JournalAppendResult } from './journal-store-contracts'
import type { JournalRow } from './journal-row-schema'
import { appendToolOutputFallback } from './journal-tool-output-fallback'
type ItemAppendOptions = { fence: number; observedAt?: number; recovered?: true }
type JournalBlob = { digest: string; payload: string }
export class JournalItemAppender {
constructor(
private readonly deps: {
journal: () => AgentSessionJournal
state: () => JournalReducerState
enqueue: (
build: (seq: number, ts: number) => JournalRow,
blobs?: readonly JournalBlob[]
) => Promise<JournalRow>
}
) {}
append(
identity: AgentJournalItemIdentity,
body: AgentJournalItemBody,
options: ItemAppendOptions
): Promise<JournalAppendResult> {
const itemId = agentJournalItemKey(identity)
return this.deps
.enqueue(journalItemRowBuilder(this.deps.state, identity, body, options))
.then((row) => itemAppendResult(row, itemId))
}
appendWithBlobs(
identity: AgentJournalItemIdentity,
body: AgentJournalItemBody,
blobs: readonly JournalBlob[],
options: ItemAppendOptions
): Promise<JournalAppendResult> {
const itemId = agentJournalItemKey(identity)
return this.deps
.enqueue(journalItemRowBuilder(this.deps.state, identity, body, options), blobs)
.then((row) => itemAppendResult(row, itemId))
.catch((error: unknown) =>
appendToolOutputFallback({
journal: this.deps.journal(),
error,
identity,
body,
blobs,
itemId,
fence: options.fence
})
)
}
}
function itemAppendResult(row: JournalRow, itemId: string): JournalAppendResult {
return {
cursor: { epoch: row.epoch, sequence: row.seq },
itemId,
revision: (row as Extract<JournalRow, { kind: 'item' }>).revision
}
}
@@ -2,19 +2,19 @@
// results by identity read off the same raw lines. Fixtures are shaped like the
// files the providers actually write.
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { agentJournalItemKey } from '../../../shared/agent-session-journal-item-key'
import type { AgentSessionJournalIdentity } from '../../../shared/agent-session-journal-types'
import { readJournalBlob } from './journal-blob-store'
import { JOURNAL_BLOB_DIR, readJournalBlob } from './journal-blob-store'
import { createLegacyIdentityTracker } from './journal-legacy-identity'
import {
appendLegacyTranscriptMessages,
importLegacyTranscriptIntoJournal
} from './journal-legacy-import'
import { DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
import { boundPayload, DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
import { openAgentSessionJournal, type AgentSessionJournal } from './journal-store'
const CLAUDE_SESSION = '29eb22a4-6a5f-4f21-9b0c-1d7f3a2e5c88'
@@ -422,9 +422,185 @@ describe('payload bounds on import', () => {
expect(body.output.head).toHaveLength(1_024)
expect(await readJournalBlob(root, body.output.digest)).toBe(output)
})
it('deduplicates staged blobs while importing a replacement epoch', async () => {
const journalDir = join(root, 'dedupe-journal')
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
inlineHeadBytes: 512,
maxSessionBytes: 512 * 1024
}
const output = 'd'.repeat(32 * 1024)
const bounded = boundPayload(output, limits)
const toolResultLine = (uuid: string) => ({
parentUuid: null,
isSidechain: false,
type: 'user',
message: {
role: 'user',
content: [{ type: 'tool_result', tool_use_id: `toolu_${uuid}`, content: output }]
},
uuid,
timestamp: '2026-08-05T10:00:09.000Z',
sessionId: CLAUDE_SESSION
})
const filePath = await writeFixture('claude-duplicate-blobs.jsonl', [
toolResultLine('aa11bb22-cc33-4d44-8e55-6f7788990011'),
toolResultLine('bb22cc33-dd44-4e55-8f66-778899001122')
])
const journal = await open('claude', CLAUDE_SESSION, {
journalDir,
limits,
autoCompact: false
})
await importLegacyTranscriptIntoJournal({
journal,
agent: 'claude',
sessionId: CLAUDE_SESSION,
fence: 1,
options: { filePath, limits }
})
expect(await readJournalBlob(journalDir, bounded.digest)).toBe(output)
expect(await readdir(join(journalDir, JOURNAL_BLOB_DIR))).toEqual([bounded.digest])
expect(
journal
.snapshot()
.items.map((item) => (item.body.kind === 'tool-call' ? item.body.output?.digest : null))
).toEqual([bounded.digest, bounded.digest])
})
it('prunes root-level blobs made stale by a later legacy import', async () => {
const journalDir = join(root, 'prune-journal')
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
inlineHeadBytes: 512,
maxSessionBytes: 512 * 1024
}
const output = 's'.repeat(32 * 1024)
const bounded = boundPayload(output, limits)
const first = await writeFixture('claude-stale-blob.jsonl', [
{
parentUuid: null,
isSidechain: false,
type: 'user',
message: {
role: 'user',
content: [{ type: 'tool_result', tool_use_id: 'toolu_stale', content: output }]
},
uuid: 'aa11bb22-cc33-4d44-8e55-6f7788990011',
timestamp: '2026-08-05T10:00:09.000Z',
sessionId: CLAUDE_SESSION
}
])
const second = await writeFixture('claude-without-blob.jsonl', [
{
parentUuid: null,
isSidechain: false,
type: 'assistant',
message: { role: 'assistant', content: [{ type: 'text', text: 'replacement' }] },
uuid: 'cc33dd44-ee55-4666-8777-889900112233',
timestamp: '2026-08-05T10:00:10.000Z',
sessionId: CLAUDE_SESSION
}
])
const journal = await open('claude', CLAUDE_SESSION, {
journalDir,
limits,
autoCompact: false
})
await importLegacyTranscriptIntoJournal({
journal,
agent: 'claude',
sessionId: CLAUDE_SESSION,
fence: 1,
options: { filePath: first, limits }
})
expect(await readJournalBlob(journalDir, bounded.digest)).toBe(output)
await importLegacyTranscriptIntoJournal({
journal,
agent: 'claude',
sessionId: CLAUDE_SESSION,
fence: 2,
options: { filePath: second, limits }
})
expect(await readJournalBlob(journalDir, bounded.digest)).toBeNull()
expect(journal.snapshot().items[0]?.body).toMatchObject({
kind: 'message',
blocks: [{ type: 'text', text: 'replacement' }]
})
})
it('uses managed catch-up appends when a tool-result blob exceeds quota', async () => {
const journalDir = join(root, 'catchup-journal')
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
inlineHeadBytes: 128,
maxSessionBytes: 8_000
}
const journal = await open('codex', CODEX_SESSION, {
journalDir,
limits,
autoCompact: false
})
const output = 'z'.repeat(12_000)
const bounded = boundPayload(output, limits)
await expect(
appendLegacyTranscriptMessages({
journal,
agent: 'codex',
sessionId: CODEX_SESSION,
fence: 1,
messages: [
{
id: 'catchup-tool-output',
role: 'tool',
blocks: [{ type: 'tool-result', output }],
timestamp: 1_800_000_000_000,
source: 'transcript'
}
]
})
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(await readJournalBlob(journalDir, bounded.digest)).toBeNull()
expect(journal.snapshot().items).toEqual([])
})
})
describe('import failures', () => {
it('rejects a legacy source above the fixed 16 MiB import cap before decoding', async () => {
const journalDir = join(root, 'oversized-source-journal')
const journal = await open('claude', CLAUDE_SESSION, {
journalDir,
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 256 * 1024 * 1024 },
autoCompact: false
})
const filePath = join(root, 'oversized-source.jsonl')
await writeFile(filePath, 'x'.repeat(16 * 1024 * 1024 + 1), 'utf8')
const epoch = journal.epoch
await expect(
importLegacyTranscriptIntoJournal({
journal,
agent: 'claude',
sessionId: CLAUDE_SESSION,
fence: 1,
options: { filePath }
})
).resolves.toMatchObject({
ok: false,
error: `Legacy transcript exceeds the ${16 * 1024 * 1024}-byte import bound`
})
expect(journal.epoch).toBe(epoch)
expect(journal.snapshot().items).toEqual([])
})
it('keeps the live epoch intact when a staged rebuild runs out of budget', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 2_000 }
const journal = await open('codex', CODEX_SESSION, { limits })
@@ -479,6 +655,104 @@ describe('import failures', () => {
})
})
it('cleans staged replacement blobs when legacy import exceeds physical quota', async () => {
const journalDir = join(root, 'replacement-journal')
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
inlineHeadBytes: 128,
maxSessionBytes: 8_000
}
const journal = await open('claude', CLAUDE_SESSION, {
journalDir,
limits,
autoCompact: false
})
const output = 'q'.repeat(12_000)
const bounded = boundPayload(output, limits)
const filePath = await writeFixture('oversized-tool-result.jsonl', [
{
parentUuid: null,
isSidechain: false,
type: 'user',
message: {
role: 'user',
content: [{ type: 'tool_result', tool_use_id: 'toolu_oversized', content: output }]
},
uuid: 'ba11ad00-1111-4222-8333-444455556666',
timestamp: '2026-08-05T10:00:09.000Z',
sessionId: CLAUDE_SESSION
}
])
const epoch = journal.epoch
await expect(
importLegacyTranscriptIntoJournal({
journal,
agent: 'claude',
sessionId: CLAUDE_SESSION,
fence: 1,
options: { filePath, limits }
})
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(journal.epoch).toBe(epoch)
expect(await readJournalBlob(journalDir, bounded.digest)).toBeNull()
expect((await readdir(journalDir)).some((name) => name.startsWith('.epoch-replacement-'))).toBe(
false
)
})
it('bounds oversized legacy tool-call input before journal publication', async () => {
const journalDir = join(root, 'bounded-tool-input-journal')
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, inlineHeadBytes: 64 }
const journal = await open('claude', CLAUDE_SESSION, {
journalDir,
limits,
autoCompact: false
})
const filePath = await writeFixture('oversized-tool-input.jsonl', [
{
parentUuid: null,
isSidechain: false,
type: 'assistant',
message: {
role: 'assistant',
content: [
{
type: 'tool_use',
id: 'toolu_large_input',
name: 'Edit',
input: { file_path: 'a.ts', patch: 'x'.repeat(10_000) }
}
]
},
uuid: 'cc11ad00-1111-4222-8333-444455556666',
timestamp: '2026-08-05T10:00:09.000Z',
sessionId: CLAUDE_SESSION
}
])
const result = await importLegacyTranscriptIntoJournal({
journal,
agent: 'claude',
sessionId: CLAUDE_SESSION,
fence: 1,
options: { filePath, limits }
})
expect(result.ok).toBe(true)
const imported = journal.snapshot().items[0]
expect(imported?.body).toMatchObject({
kind: 'tool-call',
input: {
truncated: true,
byteLength: expect.any(Number),
digest: expect.stringMatching(/^[0-9a-f]{64}$/),
head: expect.any(String)
}
})
expect(JSON.stringify(imported?.body)).not.toContain('x'.repeat(1_000))
})
it('reports a missing transcript without touching the journal', async () => {
const journal = await open('claude', CLAUDE_SESSION)
const before = journal.epoch
@@ -11,6 +11,7 @@
// writing; a later structured resume rolls the epoch again and rebuilds.
import { createReadStream } from 'node:fs'
import { stat } from 'node:fs/promises'
import type { AgentType } from '../../../shared/agent-status-types'
import type {
AgentJournalCursor,
@@ -27,12 +28,12 @@ import {
decodeOmpTranscriptLine
} from '../transcript-line-decoders'
import { decodeTranscriptStream } from '../transcript-stream-lines'
import { putJournalBlob } from './journal-blob-store'
import { createLegacyIdentityTracker } from './journal-legacy-identity'
import type { JournalReplacementItem } from './journal-epoch-replacement'
import {
boundInlineText,
boundPayload,
boundToolInput,
DEFAULT_JOURNAL_PAYLOAD_LIMITS,
type JournalPayloadLimits
} from './journal-payload-bounds'
@@ -45,6 +46,8 @@ export type LegacyImportOptions = ResolveSessionFileOptions & {
decodedMessageIdentities?: true
}
const MAX_LEGACY_IMPORT_SOURCE_BYTES = 16 * 1024 * 1024
export type LegacyImportResult =
| { ok: true; epoch: string; cursor: AgentJournalCursor; imported: number }
| { ok: false; error: string }
@@ -59,10 +62,7 @@ export async function appendLegacyTranscriptMessages(input: {
let appended = 0
for (const message of input.messages) {
const mapped = legacyItemBody(message, DEFAULT_JOURNAL_PAYLOAD_LIMITS)
for (const blob of mapped.blobs) {
await putJournalBlob(input.journal.directory, blob.digest, blob.payload)
}
await input.journal.appendItem(
await input.journal.appendItemWithBlobs(
{
provider: 'legacy',
agent: input.agent,
@@ -70,6 +70,7 @@ export async function appendLegacyTranscriptMessages(input: {
recordId: message.id
},
mapped.body,
mapped.blobs,
{ fence: input.fence, observedAt: message.timestamp ?? undefined }
)
appended += 1
@@ -96,6 +97,21 @@ export async function importLegacyTranscriptIntoJournal(input: {
return { ok: false, error: `No transcript found for ${input.agent} session ${input.sessionId}` }
}
// Refuse an oversized source before decoding any prefix. Importing a prefix
// would make the restored timeline look complete while silently omitting
// later records; callers can retry after reducing the source or quota.
try {
const sourceBytes = (await stat(filePath)).size
if (sourceBytes > MAX_LEGACY_IMPORT_SOURCE_BYTES) {
return {
ok: false,
error: `Legacy transcript exceeds the ${MAX_LEGACY_IMPORT_SOURCE_BYTES}-byte import bound`
}
}
} catch (err) {
return { ok: false, error: err instanceof Error ? err.message : String(err) }
}
let decoded: { messages: NativeChatMessage[]; identities: AgentJournalItemIdentity[] }
try {
decoded = await decodeWithIdentities({
@@ -109,6 +125,9 @@ export async function importLegacyTranscriptIntoJournal(input: {
return { ok: false, error: err instanceof Error ? err.message : String(err) }
}
if (decoded.identities.length !== decoded.messages.length) {
return { ok: false, error: 'Legacy transcript identity coverage is incomplete' }
}
const replacement: JournalReplacementItem[] = []
for (const [index, message] of decoded.messages.entries()) {
const identity = decoded.identities[index]
@@ -116,12 +135,10 @@ export async function importLegacyTranscriptIntoJournal(input: {
continue
}
const mapped = legacyItemBody(message, limits)
for (const blob of mapped.blobs) {
await putJournalBlob(input.journal.directory, blob.digest, blob.payload)
}
replacement.push({
identity,
body: mapped.body,
blobs: mapped.blobs,
observedAt: message.timestamp ?? undefined
})
}
@@ -199,7 +216,15 @@ function legacyItemBody(
const only = message.blocks.length === 1 ? message.blocks[0] : undefined
if (only?.type === 'tool-call') {
return {
body: { kind: 'tool-call', name: only.name, input: only.input, state: 'completed' },
// Legacy transcripts are untrusted and can contain arbitrarily large
// tool arguments. Keep them on the same bounded path as live events
// before the replacement epoch is staged or published.
body: {
kind: 'tool-call',
name: only.name,
input: boundToolInput(only.input, limits),
state: 'completed'
},
blobs: []
}
}
@@ -0,0 +1,160 @@
import type {
AgentJournalItemBody,
AgentJournalSnapshot
} from '../../../shared/agent-session-journal-types'
import {
dispatchReservationId,
JournalLifecycleCapacity,
lifecycleReservationIdForItem,
requiresTerminalSettlement,
terminalReservationBytes,
type JournalLifecycleReservation
} from './journal-lifecycle-capacity'
import type { JournalRow } from './journal-row-schema'
import { journalRowByteLength } from './journal-row-schema'
import { AgentSessionJournalError } from './journal-write-guards'
export type JournalLifecycleRowAdmission = {
releaseAfter: string[]
protectedBytes: number
lifecycleCovered: boolean
proposedCapacity: JournalLifecycleCapacity
}
export class JournalLifecycleAdmission {
private readonly capacity = new JournalLifecycleCapacity()
constructor(
private readonly sessionId: string,
private readonly maxBytes: number,
private readonly canonicalItemId: (itemId: string) => string,
private readonly maxAppendSlots = Number.MAX_SAFE_INTEGER
) {}
get state(): { reservedBytes: number; reservedAppendSlots: number } {
return {
reservedBytes: this.capacity.reservedBytes,
reservedAppendSlots: this.capacity.reservedAppendSlots
}
}
rebuild(snapshot: AgentJournalSnapshot, currentPhysicalBytes: number): void {
if (
!this.capacity.rebuild(snapshot, this.maxBytes, currentPhysicalBytes, this.maxAppendSlots)
) {
throw this.capacityError('cannot rebuild lifecycle capacity')
}
}
reserve(token: JournalLifecycleReservation, currentPhysicalBytes: number): boolean {
return this.capacity.reserve(token, currentPhysicalBytes, this.maxBytes, this.maxAppendSlots)
}
transfer(fromId: string, toId: string): boolean {
return this.capacity.transfer(fromId, toId)
}
release(id: string): void {
this.capacity.release(id)
}
prepare(row: JournalRow, currentPhysicalBytes: number): JournalLifecycleRowAdmission {
const proposedCapacity = this.capacity.clone()
this.ensureActionable(row, currentPhysicalBytes, proposedCapacity)
const releaseAfter = this.reservationsSettledBy(row, proposedCapacity)
const releasedBytes = releaseAfter.reduce(
(total, id) => total + (proposedCapacity.token(id)?.bytes ?? 0),
0
)
return {
releaseAfter,
protectedBytes: proposedCapacity.reservedBytes - releasedBytes,
lifecycleCovered: proposedCapacity.covers(releaseAfter, journalRowByteLength(row), 1),
proposedCapacity
}
}
commit(admission: JournalLifecycleRowAdmission): void {
this.capacity.replaceFrom(admission.proposedCapacity)
for (const id of admission.releaseAfter) {
this.capacity.release(id)
}
}
private ensureActionable(
row: JournalRow,
currentPhysicalBytes: number,
capacity: JournalLifecycleCapacity
): void {
if (row.kind === 'item') {
this.ensureActionableItem(row.itemId, row.body, currentPhysicalBytes, capacity)
return
}
if (row.kind !== 'lifecycle-batch') {
return
}
for (const mutation of row.mutations) {
if (mutation.kind === 'item') {
this.ensureActionableItem(mutation.itemId, mutation.body, currentPhysicalBytes, capacity)
}
}
}
private ensureActionableItem(
itemId: string,
body: AgentJournalItemBody,
currentPhysicalBytes: number,
capacity: JournalLifecycleCapacity
): void {
if (!requiresTerminalSettlement(body)) {
return
}
const id = lifecycleReservationIdForItem(this.canonicalItemId(itemId))
if (body.kind === 'status' && body.turnLifecycle?.state === 'running' && !capacity.has(id)) {
capacity.claimFirst('tentative-turn:', id)
}
if (
!capacity.reserve(
{ id, bytes: terminalReservationBytes(body), appendSlots: 1 },
currentPhysicalBytes,
this.maxBytes,
this.maxAppendSlots
)
) {
throw this.capacityError('cannot reserve terminal capacity')
}
}
private reservationsSettledBy(row: JournalRow, capacity: JournalLifecycleCapacity): string[] {
if (row.kind === 'dispatch') {
const id = dispatchReservationId(row.clientMessageId)
return capacity.has(id) ? [id] : []
}
const itemIds =
row.kind === 'item'
? requiresTerminalSettlement(row.body)
? []
: [row.itemId]
: row.kind === 'tombstone'
? [row.itemId]
: row.kind === 'lifecycle-batch'
? row.mutations.flatMap((mutation) =>
mutation.kind === 'item' && requiresTerminalSettlement(mutation.body)
? []
: [mutation.itemId]
)
: []
return [
...new Set(
itemIds.map((itemId) => lifecycleReservationIdForItem(this.canonicalItemId(itemId)))
)
].filter((id) => capacity.has(id))
}
private capacityError(detail: string): AgentSessionJournalError {
return new AgentSessionJournalError(
'journal_bound_exceeded',
`agent-session journal for ${this.sessionId} ${detail}`
)
}
}
@@ -0,0 +1,47 @@
import type { AgentJournalCursor } from '../../../shared/agent-session-journal-types'
import type { JournalReducerState } from './journal-reducer'
import { journalLifecycleBatchRowBuilder } from './journal-row-builders'
import type { JournalLifecycleBatchInput } from './journal-store-contracts'
import type { JournalRow } from './journal-row-schema'
const SETTLEMENT_ALREADY_APPLIED = new Error('journal_settlement_already_applied')
export class JournalLifecycleBatchAppender {
constructor(
private readonly deps: {
state: () => JournalReducerState
cursor: () => AgentJournalCursor
enqueue: (build: (seq: number, ts: number) => JournalRow) => Promise<JournalRow>
}
) {}
append(input: JournalLifecycleBatchInput): Promise<AgentJournalCursor> {
if (this.wasApplied(input.settlementId)) {
return Promise.resolve(this.deps.cursor())
}
const build = journalLifecycleBatchRowBuilder(
this.deps.state,
input.settlementId,
input.mutations,
input
)
return this.deps
.enqueue((seq, ts) => {
if (this.wasApplied(input.settlementId)) {
throw SETTLEMENT_ALREADY_APPLIED
}
return build(seq, ts)
})
.then((row) => ({ epoch: row.epoch, sequence: row.seq }))
.catch((error: unknown) => {
if (error === SETTLEMENT_ALREADY_APPLIED) {
return this.deps.cursor()
}
throw error
})
}
private wasApplied(settlementId: string): boolean {
return this.deps.state().appliedSettlementIds.has(settlementId)
}
}
@@ -0,0 +1,81 @@
import { agentJournalItemKey } from '../../../shared/agent-session-journal-item-key'
import { AGENT_SESSION_JOURNAL_SCHEMA_VERSION } from '../../../shared/agent-session-journal-types'
import type { JournalLifecycleMutationInput } from './journal-row-builders'
import type { JournalLifecycleBatchRow, JournalLifecycleMutation } from './journal-row-schema'
import {
MAX_JOURNAL_LIFECYCLE_BATCH_BYTES,
MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS
} from './journal-row-schema'
export type JournalLifecycleMutationChunk = {
settlementId: string
mutations: JournalLifecycleMutationInput[]
}
export function partitionJournalLifecycleMutations(
settlementId: string,
mutations: readonly JournalLifecycleMutationInput[]
): JournalLifecycleMutationChunk[] {
if (mutations.length === 0) {
return []
}
const chunks: JournalLifecycleMutationInput[][] = []
const probeId = chunkSettlementId(settlementId, mutations.length - 1, mutations.length)
let pending: JournalLifecycleMutationInput[] = []
for (const mutation of mutations) {
const candidate = [...pending, mutation]
if (pending.length > 0 && !serializedLifecycleBatchFits(probeId, candidate)) {
chunks.push(pending)
pending = [mutation]
} else {
pending = candidate
}
if (pending.length === MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS) {
chunks.push(pending)
pending = []
}
}
if (pending.length > 0) {
chunks.push(pending)
}
return chunks.map((chunk, index) => ({
settlementId:
chunks.length === 1 ? settlementId : chunkSettlementId(settlementId, index, chunks.length),
mutations: chunk
}))
}
function chunkSettlementId(settlementId: string, index: number, total: number): string {
return `${settlementId}:${index + 1}/${total}`
}
function serializedLifecycleBatchFits(
settlementId: string,
mutations: readonly JournalLifecycleMutationInput[]
): boolean {
const row: JournalLifecycleBatchRow = {
v: AGENT_SESSION_JOURNAL_SCHEMA_VERSION,
kind: 'lifecycle-batch',
epoch: '00000000-0000-4000-8000-000000000000',
seq: Number.MAX_SAFE_INTEGER,
fence: Number.MAX_SAFE_INTEGER,
ts: Number.MAX_SAFE_INTEGER,
settlementId,
mutations: mutations.map(lifecycleMutationRowShape)
}
return Buffer.byteLength(JSON.stringify(row), 'utf8') + 1 <= MAX_JOURNAL_LIFECYCLE_BATCH_BYTES
}
function lifecycleMutationRowShape(
mutation: JournalLifecycleMutationInput
): JournalLifecycleMutation {
const itemId = agentJournalItemKey(mutation.identity)
return mutation.kind === 'item'
? {
kind: 'item',
itemId,
revision: Number.MAX_SAFE_INTEGER,
body: mutation.body
}
: { kind: 'tombstone', itemId, revision: Number.MAX_SAFE_INTEGER }
}
@@ -0,0 +1,30 @@
import { describe, expect, it } from 'vitest'
import type { AgentJournalSnapshot } from '../../../shared/agent-session-journal-types'
import { JournalLifecycleCapacity } from './journal-lifecycle-capacity'
describe('JournalLifecycleCapacity', () => {
it('enforces append-slot limits for both rebuilt submission reservations', () => {
const snapshot: AgentJournalSnapshot = {
sessionId: 'session-1',
cursor: { epoch: 'epoch-1', sequence: 1 },
items: [],
submissions: [
{
clientMessageId: 'message-1',
fence: 0,
payloadFingerprint: 'fingerprint',
dispatchState: 'pending',
providerItemId: null,
reason: null,
submittedAt: 1,
resolvedAt: null
}
]
}
const capacity = new JournalLifecycleCapacity()
expect(capacity.rebuild(snapshot, Number.MAX_SAFE_INTEGER, 0, 1)).toBe(false)
expect(capacity.reservedAppendSlots).toBe(1)
})
})
@@ -0,0 +1,193 @@
import type {
AgentJournalItemBody,
AgentJournalSnapshot
} from '../../../shared/agent-session-journal-types'
export type JournalLifecycleReservation = {
id: string
bytes: number
appendSlots: number
}
export const JOURNAL_TURN_TERMINAL_RESERVATION_BYTES = 128 * 1024
export const JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES = 64 * 1024
export const JOURNAL_DISPATCH_RESERVATION_BYTES = 32 * 1024
export class JournalLifecycleCapacity {
private readonly reservations = new Map<string, JournalLifecycleReservation>()
get reservedBytes(): number {
return [...this.reservations.values()].reduce((total, token) => total + token.bytes, 0)
}
get reservedAppendSlots(): number {
return [...this.reservations.values()].reduce((total, token) => total + token.appendSlots, 0)
}
has(id: string): boolean {
return this.reservations.has(id)
}
token(id: string): JournalLifecycleReservation | null {
return this.reservations.get(id) ?? null
}
clone(): JournalLifecycleCapacity {
const copy = new JournalLifecycleCapacity()
for (const token of this.reservations.values()) {
copy.reservations.set(token.id, { ...token })
}
return copy
}
replaceFrom(source: JournalLifecycleCapacity): void {
this.reservations.clear()
for (const token of source.reservations.values()) {
this.reservations.set(token.id, { ...token })
}
}
reserve(
token: JournalLifecycleReservation,
currentPhysicalBytes: number,
maxBytes: number,
maxAppendSlots = Number.MAX_SAFE_INTEGER
): boolean {
if (this.reservations.has(token.id)) {
return true
}
if (currentPhysicalBytes + this.reservedBytes + token.bytes > maxBytes) {
return false
}
if (this.reservedAppendSlots + token.appendSlots > maxAppendSlots) {
return false
}
this.reservations.set(token.id, token)
return true
}
transfer(fromId: string, toId: string): boolean {
const existing = this.reservations.get(fromId)
if (!existing) {
return false
}
this.reservations.delete(fromId)
this.reservations.set(toId, { ...existing, id: toId })
return true
}
claimFirst(prefix: string, toId: string): boolean {
const fromId = [...this.reservations.keys()].find((id) => id.startsWith(prefix))
return fromId ? this.transfer(fromId, toId) : false
}
release(id: string): void {
this.reservations.delete(id)
}
covers(ids: readonly string[], bytes: number, appendSlots: number): boolean {
const tokens = ids.flatMap((id) => {
const token = this.reservations.get(id)
return token ? [token] : []
})
return (
tokens.length > 0 &&
tokens.reduce((total, token) => total + token.bytes, 0) >= bytes &&
tokens.reduce((total, token) => total + token.appendSlots, 0) >= appendSlots
)
}
rebuild(
snapshot: AgentJournalSnapshot,
maxBytes: number,
currentPhysicalBytes: number,
maxAppendSlots = Number.MAX_SAFE_INTEGER
): boolean {
this.reservations.clear()
for (const item of snapshot.items) {
if (!requiresTerminalSettlement(item.body)) {
continue
}
if (
!this.reserve(
{
id: lifecycleReservationIdForItem(item.itemId),
bytes: terminalReservationBytes(item.body),
appendSlots: 1
},
currentPhysicalBytes,
maxBytes,
maxAppendSlots
)
) {
return false
}
}
for (const submission of snapshot.submissions) {
if (submission.dispatchState !== 'pending' && submission.dispatchState !== 'unknown') {
continue
}
// A write-ahead submission owns both its dispatch attempt and the
// terminal turn settlement. Rebuild both reservations after restart;
// restoring only the tentative turn token would let a new send consume
// the dispatch headroom still owed to this unresolved submission.
if (
!this.reserve(
{
id: dispatchReservationId(submission.clientMessageId),
bytes: JOURNAL_DISPATCH_RESERVATION_BYTES,
appendSlots: 1
},
currentPhysicalBytes,
maxBytes,
maxAppendSlots
)
) {
return false
}
if (
!this.reserve(
{
id: tentativeTurnReservationId(submission.clientMessageId),
bytes: JOURNAL_TURN_TERMINAL_RESERVATION_BYTES,
appendSlots: 1
},
currentPhysicalBytes,
maxBytes,
maxAppendSlots
)
) {
return false
}
}
return true
}
}
export function lifecycleReservationIdForItem(itemId: string): string {
return `item:${itemId}`
}
export function dispatchReservationId(clientMessageId: string): string {
return `dispatch:${clientMessageId}`
}
export function tentativeTurnReservationId(clientMessageId: string): string {
return `tentative-turn:${clientMessageId}`
}
export function requiresTerminalSettlement(body: AgentJournalItemBody): boolean {
if (body.kind === 'tool-call') {
return body.state === 'running'
}
if (body.kind === 'approval' || body.kind === 'question') {
return body.resolution.state === 'pending'
}
return body.kind === 'status' && body.turnLifecycle?.state === 'running'
}
export function terminalReservationBytes(body: AgentJournalItemBody): number {
return body.kind === 'status' && body.turnLifecycle?.state === 'running'
? JOURNAL_TURN_TERMINAL_RESERVATION_BYTES
: JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES
}
@@ -8,7 +8,7 @@
// between publishing the snapshot and truncating the log leaves the log a
// superset of the tail, and recovery unions the two by sequence — never a hole.
import { appendFile, mkdir, open, readFile, type FileHandle } from 'node:fs/promises'
import { appendFile, mkdir, open, readFile, stat, type FileHandle } from 'node:fs/promises'
import { randomUUID } from 'node:crypto'
import { join } from 'node:path'
import { durableWriteTempPath, renameDurable, writeFileDurable } from '../../durable-file-write'
@@ -22,6 +22,7 @@ import {
isAdmissibleAgentJournalSubmission
} from '../../../shared/agent-session-journal-schemas'
import { parseJournalRow, serializeJournalRow, type JournalRow } from './journal-row-schema'
import { assertJournalPhysicalCapacity } from './journal-physical-quota'
export const JOURNAL_LOG_FILE = 'log.jsonl'
export const JOURNAL_SNAPSHOT_FILE = 'snapshot.json'
@@ -48,6 +49,8 @@ export type JournalSnapshotFile = {
* still reconciles into the bubble it belongs to. */
aliases: { providerItemId: string; itemId: string }[]
tombstones: { itemId: string; revision: number }[]
/** Bounded by compaction retention; used to deduplicate a replayed settlement. */
appliedSettlementIds?: string[]
tail: JournalRow[]
}
@@ -107,7 +110,31 @@ export async function readJournalSnapshot(journalDir: string): Promise<JournalSn
}
}
export async function quarantineInvalidJournalSnapshot(journalDir: string): Promise<string> {
export async function quarantineInvalidJournalSnapshot(
journalDir: string,
quota?: { sessionId: string; maxBytes: number }
): Promise<string> {
// Rename is normally same-filesystem and size-neutral, but admission must
// happen before retaining evidence so a full journal never creates an
// unbounded quarantine artifact (or relies on a copy fallback).
if (quota) {
const source = join(journalDir, JOURNAL_SNAPSHOT_FILE)
// Account for the complete source bytes: rename is usually neutral, but a
// cross-device/filesystem fallback may briefly retain both inodes.
const sourceBytes = await stat(source)
.then((info) => info.size)
.catch((error) => {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return 0
}
throw error
})
await assertJournalPhysicalCapacity({
journalDir,
...quota,
peakAdditionalBytes: sourceBytes
})
}
const source = join(journalDir, JOURNAL_SNAPSHOT_FILE)
const target = join(journalDir, `quarantine-snapshot-${Date.now()}-${randomUUID()}.json`)
await renameDurable(source, target)
@@ -189,6 +216,8 @@ function isJournalSnapshotFile(value: unknown): value is JournalSnapshotFile {
// seeding iterates this collection, so a JSON-valid wrong shape must land
// in quarantine rather than throw through startup restoration.
(snapshot.tombstones === undefined || arrayOf(snapshot.tombstones, isTombstone)) &&
(snapshot.appliedSettlementIds === undefined ||
arrayOf(snapshot.appliedSettlementIds, (entry) => typeof entry === 'string')) &&
arrayOf(snapshot.tail, (row) => parseJournalRow(JSON.stringify(row)).ok)
)
}
@@ -17,6 +17,7 @@ import {
import {
applyJournalRow,
createJournalReducerState,
rememberAppliedSettlementId,
type JournalReducerState
} from './journal-reducer'
import { journalRowByteLength, type JournalRow } from './journal-row-schema'
@@ -42,7 +43,8 @@ export type JournalLoad = {
/** Returns null when no journal exists yet for this session. */
export async function loadJournal(
journalDir: string,
sessionId: string
sessionId: string,
quota?: { maxBytes: number }
): Promise<JournalLoad | null> {
const snapshotRead = await readJournalSnapshot(journalDir)
if (snapshotRead.status === 'unreadable') {
@@ -53,7 +55,10 @@ export async function loadJournal(
return emptyReadOnlyLoad(sessionId)
}
if (snapshotRead.status === 'invalid') {
await quarantineInvalidJournalSnapshot(journalDir)
await quarantineInvalidJournalSnapshot(
journalDir,
quota ? { sessionId, maxBytes: quota.maxBytes } : undefined
)
}
const snapshot = snapshotRead.status === 'valid' ? snapshotRead.snapshot : null
const log = await readJournalLog(journalDir)
@@ -160,6 +165,9 @@ function seedState(
for (const tombstone of snapshot.tombstones ?? []) {
state.tombstones.set(tombstone.itemId, tombstone.revision)
}
for (const settlementId of snapshot.appliedSettlementIds ?? []) {
rememberAppliedSettlementId(state, settlementId)
}
state.highestFence = snapshot.highestFence ?? 0
state.lastSequence = snapshot.compactedThrough
state.oldestSequence = snapshot.compactedThrough + 1
@@ -75,6 +75,30 @@ export function boundInlineText(
}
}
/** Keep arbitrary tool input JSON bounded before lifecycle admission. */
export function boundToolInput(input: unknown, limits: JournalPayloadLimits): unknown {
let encoded: string
try {
encoded = JSON.stringify(input) ?? 'null'
} catch {
return {
truncated: true,
byteLength: 0,
digest: digestPayload(''),
head: '[unserializable input]'
}
}
const bounded = boundPayload(encoded, limits)
return bounded.truncated
? {
truncated: true,
byteLength: bounded.byteLength,
digest: bounded.digest,
head: bounded.head
}
: input
}
/** Slice at a byte budget without splitting a multi-byte character. */
function clipUtf8(buffer: Buffer, maxBytes: number): string {
let end = maxBytes
@@ -0,0 +1,125 @@
import { mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import type {
AgentJournalItemBody,
AgentJournalItemIdentity,
AgentSessionJournalIdentity
} from '../../../shared/agent-session-journal-types'
import { JOURNAL_SNAPSHOT_FILE } from './journal-log-file'
import { DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
import { journalDirectoryBytes } from './journal-physical-quota'
import { openAgentSessionJournal } from './journal-store'
const IDENTITY: AgentSessionJournalIdentity = {
sessionId: 'session-1',
workspaceId: 'ws-1',
hostId: 'host-1',
agent: 'codex',
providerHandle: { kind: 'codex', threadId: 'thread-1' }
}
let root: string
function item(ordinal: number): AgentJournalItemIdentity {
return { provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal }
}
function body(value: string): AgentJournalItemBody {
return { kind: 'message', role: 'assistant', blocks: [{ type: 'text', text: value }] }
}
beforeEach(async () => {
root = await mkdtemp(join(tmpdir(), 'orca-journal-quota-'))
})
afterEach(async () => {
await rm(root, { recursive: true, force: true })
})
describe('journal physical quota peaks', () => {
it('refuses an epoch replacement whose staging peak exceeds the quota', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 8_000 }
const journal = await openAgentSessionJournal({
identity: IDENTITY,
journalDir: root,
limits,
autoCompact: false
})
await journal.appendItem(item(1), body('old'.repeat(500)), { fence: 1 })
const epoch = journal.epoch
await expect(
journal.replaceEpochItems('handle_forked', 2, [
{ identity: item(2), body: body('replacement'.repeat(250)) }
])
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(journal.epoch).toBe(epoch)
expect((await readdir(root)).some((name) => name.startsWith('.epoch-replacement-'))).toBe(false)
expect(await journalDirectoryBytes(root)).toBeLessThanOrEqual(limits.maxSessionBytes)
})
it('refuses schema quarantine when its peak copy would exceed the quota', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 7_000 }
await openAgentSessionJournal({ identity: IDENTITY, journalDir: root, limits })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
snapshot.items = [{ body: { kind: 'future', payload: 'x'.repeat(4_000) } }]
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await openAgentSessionJournal({ identity: IDENTITY, journalDir: root, limits })
await expect(reopened.rollEpoch('schema_unreadable', 2)).rejects.toMatchObject({
code: 'journal_bound_exceeded'
})
expect(reopened.isReadOnly).toBe(true)
expect((await readdir(root)).some((name) => name.startsWith('quarantine-'))).toBe(false)
expect(await journalDirectoryBytes(root)).toBeLessThanOrEqual(limits.maxSessionBytes)
})
it('does not rename an invalid snapshot when the directory is already full', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 2_000 }
await openAgentSessionJournal({ identity: IDENTITY, journalDir: root, limits })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
await writeFile(snapshotPath, '{"invalid":', 'utf8')
const current = await journalDirectoryBytes(root)
await writeFile(
join(root, 'quota-filler'),
'x'.repeat(Math.max(0, limits.maxSessionBytes - current)),
'utf8'
)
await expect(
openAgentSessionJournal({ identity: IDENTITY, journalDir: root, limits })
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(await readFile(snapshotPath, 'utf8')).toBe('{"invalid":')
expect((await readdir(root)).some((name) => name.startsWith('quarantine-snapshot-'))).toBe(
false
)
})
it('counts pre-existing durable-write temps while staging an epoch replacement', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 8_000 }
const journal = await openAgentSessionJournal({
identity: IDENTITY,
journalDir: root,
limits,
autoCompact: false
})
await journal.appendItem(item(1), body('old'), { fence: 1 })
// Simulate a temp left by a crash. Replacement must refuse before writing
// its epoch row or creating a staging blob beside this file.
await writeFile(join(root, 'snapshot.json.crashed-write.tmp'), 'x'.repeat(7_500), 'utf8')
const epoch = journal.epoch
await expect(
journal.replaceEpochItems('handle_forked', 2, [{ identity: item(2), body: body('new') }])
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
expect(journal.epoch).toBe(epoch)
expect((await readdir(root)).some((name) => name.startsWith('.epoch-replacement-'))).toBe(false)
})
})
@@ -0,0 +1,41 @@
import { lstat, readdir } from 'node:fs/promises'
import type { Dirent } from 'node:fs'
import { join } from 'node:path'
import { AgentSessionJournalError } from './journal-write-guards'
/** Counts every physical file owned by one session, including blobs, durable
* write temps, and retained quarantine evidence. Symlinks are charged as files
* but never followed outside the journal directory. */
export async function journalDirectoryBytes(directory: string): Promise<number> {
let entries: Dirent<string>[]
try {
entries = await readdir(directory, { withFileTypes: true, encoding: 'utf8' })
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return 0
}
throw error
}
let total = 0
for (const entry of entries) {
const path = join(directory, entry.name)
total += entry.isDirectory() ? await journalDirectoryBytes(path) : (await lstat(path)).size
}
return total
}
export async function assertJournalPhysicalCapacity(input: {
journalDir: string
sessionId: string
maxBytes: number
peakAdditionalBytes?: number
}): Promise<number> {
const current = await journalDirectoryBytes(input.journalDir)
if (current + (input.peakAdditionalBytes ?? 0) > input.maxBytes) {
throw new AgentSessionJournalError(
'journal_bound_exceeded',
`agent-session journal for ${input.sessionId} reached its ${input.maxBytes}-byte physical bound`
)
}
return current
}
@@ -0,0 +1,88 @@
import type {
AgentJournalApprovalItem,
AgentJournalItemBody,
AgentJournalPromptOption,
AgentJournalQuestionItem
} from '../../../shared/agent-session-journal-types'
import {
boundInlineText,
boundPayload,
DEFAULT_JOURNAL_PAYLOAD_LIMITS
} from './journal-payload-bounds'
export const MAX_JOURNAL_PROMPT_OPTIONS = 64
const JOURNAL_PROMPT_OPTION_LIMITS = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
inlineHeadBytes: 1024
}
const JOURNAL_PROMPT_ID_MAX_BYTES = 1024
export function cancelledJournalPromptBody(
body: AgentJournalItemBody
): AgentJournalApprovalItem | AgentJournalQuestionItem | null {
if (body.kind !== 'approval' && body.kind !== 'question') {
return null
}
const bounded = boundJournalPromptBody(body)
return {
...bounded,
resolution: {
state: 'cancelled',
selectedOptionId: null,
resolvedBy: null,
resolvedAt: null
}
}
}
export function boundJournalStatusText(text: string): string {
return boundInlineText(text, DEFAULT_JOURNAL_PAYLOAD_LIMITS).text
}
function boundJournalPromptBody(
body: AgentJournalApprovalItem | AgentJournalQuestionItem
): AgentJournalApprovalItem | AgentJournalQuestionItem {
if (body.kind === 'approval') {
return {
...body,
title: boundPromptText(body.title),
detail: body.detail === null ? null : boundPromptText(body.detail),
options: boundPromptOptions(body.options)
}
}
return {
...body,
question: boundPromptText(body.question),
options: boundPromptOptions(body.options),
...(body.freeTextQuestionId
? { freeTextQuestionId: boundPromptIdentifier(body.freeTextQuestionId) }
: {})
}
}
function boundPromptOptions(
options: readonly AgentJournalPromptOption[]
): AgentJournalPromptOption[] {
return options.slice(0, MAX_JOURNAL_PROMPT_OPTIONS).map((option) => ({
id: boundPromptIdentifier(option.id),
label: boundInlineText(option.label, JOURNAL_PROMPT_OPTION_LIMITS).text
}))
}
function boundPromptText(value: string): string {
return boundInlineText(value, DEFAULT_JOURNAL_PAYLOAD_LIMITS).text
}
function boundPromptIdentifier(value: string): string {
if (Buffer.byteLength(value, 'utf8') <= JOURNAL_PROMPT_ID_MAX_BYTES) {
return value
}
const bounded = boundPayload(value, {
inlineHeadBytes: JOURNAL_PROMPT_ID_MAX_BYTES - 33,
maxSessionBytes: Number.MAX_SAFE_INTEGER,
maxAppendsPerWindow: Number.MAX_SAFE_INTEGER,
appendWindowMs: Number.MAX_SAFE_INTEGER
})
return `${bounded.head}#${bounded.digest.slice(0, 32)}`
}
@@ -10,6 +10,7 @@ import { structuredAgentSessionPayloadFingerprint } from '../../../shared/struct
import {
applyJournalRow,
createJournalReducerState,
MAX_JOURNAL_APPLIED_SETTLEMENT_IDS,
referencedBlobDigests,
renderJournalState,
type JournalReducerState
@@ -361,6 +362,23 @@ describe('submission and dispatch state machine', () => {
})
})
describe('lifecycle settlement deduplication', () => {
it('retains only the newest bounded settlement ids', () => {
const state = createJournalReducerState('session-1', EPOCH)
for (let index = 0; index <= MAX_JOURNAL_APPLIED_SETTLEMENT_IDS; index += 1) {
applyJournalRow(state, {
kind: 'lifecycle-batch',
settlementId: `settlement-${index}`,
mutations: [{ kind: 'tombstone', itemId: 'item', revision: index + 1 }],
...base(index + 1)
})
}
expect(state.appliedSettlementIds.size).toBe(MAX_JOURNAL_APPLIED_SETTLEMENT_IDS)
expect(state.appliedSettlementIds.has('settlement-0')).toBe(false)
expect(state.appliedSettlementIds.has('settlement-1')).toBe(true)
})
})
describe('blob retention', () => {
it('reports the digests live rows still reference', () => {
const state = fold([
@@ -21,6 +21,8 @@ import {
import { structuredAgentSessionPayloadFingerprint } from '../../../shared/structured-agent-session-mutation'
import type { JournalRow } from './journal-row-schema'
export const MAX_JOURNAL_APPLIED_SETTLEMENT_IDS = 4_096
export type JournalReducerState = {
sessionId: string
epoch: string
@@ -36,6 +38,7 @@ export type JournalReducerState = {
/** Provider item id → the submission slot that adopted it. Stops an accepted
* echo from appending a second copy of the user's own message. */
aliases: Map<string, string>
appliedSettlementIds: Set<string>
}
export function createJournalReducerState(sessionId: string, epoch: string): JournalReducerState {
@@ -49,7 +52,8 @@ export function createJournalReducerState(sessionId: string, epoch: string): Jou
tombstones: new Map(),
submissions: new Map(),
receipts: new Map(),
aliases: new Map()
aliases: new Map(),
appliedSettlementIds: new Set()
}
}
@@ -75,6 +79,28 @@ export function applyJournalRow(state: JournalReducerState, row: JournalRow): vo
removeItem(state, resolveItemId(state, row.itemId), row.revision)
return
}
if (row.kind === 'lifecycle-batch') {
if (state.appliedSettlementIds.has(row.settlementId)) {
return
}
for (const mutation of row.mutations) {
if (mutation.kind === 'item') {
const itemId = resolveJournalItemId(state, mutation.itemId, mutation.body)
upsertItem(state, itemId, mutation.revision, {
itemId,
revision: mutation.revision,
body: mutation.body,
sequence: row.seq,
observedAt: row.ts,
...(row.recovered ? { recovered: row.recovered } : {})
})
} else {
removeItem(state, resolveItemId(state, mutation.itemId), mutation.revision)
}
}
rememberAppliedSettlementId(state, row.settlementId)
return
}
if (row.kind === 'submission') {
applySubmission(state, row)
return
@@ -82,6 +108,20 @@ export function applyJournalRow(state: JournalReducerState, row: JournalRow): vo
applyDispatch(state, row)
}
export function rememberAppliedSettlementId(
state: JournalReducerState,
settlementId: string
): void {
state.appliedSettlementIds.add(settlementId)
while (state.appliedSettlementIds.size > MAX_JOURNAL_APPLIED_SETTLEMENT_IDS) {
const oldest = state.appliedSettlementIds.values().next().value
if (oldest === undefined) {
return
}
state.appliedSettlementIds.delete(oldest)
}
}
export function resolveJournalItemId(
state: JournalReducerState,
itemId: string,
@@ -11,9 +11,15 @@ import type { JournalReducerState } from './journal-reducer'
import type {
JournalDispatchRow,
JournalItemRow,
JournalLifecycleBatchRow,
JournalLifecycleMutation,
JournalSubmissionRow,
JournalTombstoneRow
} from './journal-row-schema'
import {
MAX_JOURNAL_LIFECYCLE_BATCH_BYTES,
MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS
} from './journal-row-schema'
import type { ResolveDispatchInput } from './journal-store-contracts'
type RowBuilder<T> = (seq: number, ts: number) => T
@@ -78,6 +84,50 @@ export function journalDispatchRowBuilder(
})
}
export type JournalLifecycleMutationInput =
| { kind: 'item'; identity: AgentJournalItemIdentity; body: AgentJournalItemBody }
| { kind: 'tombstone'; identity: AgentJournalItemIdentity }
export function journalLifecycleBatchRowBuilder(
state: () => JournalReducerState,
settlementId: string,
mutations: readonly JournalLifecycleMutationInput[],
options: { fence: number; recovered?: true }
): RowBuilder<JournalLifecycleBatchRow> {
return (seq, ts) => {
if (mutations.length === 0 || mutations.length > MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS) {
throw new Error('journal_lifecycle_batch_mutation_bound_exceeded')
}
const current = state()
const revisions = new Map<string, number>()
const built: JournalLifecycleMutation[] = mutations.map((mutation) => {
const itemId = agentJournalItemKey(mutation.identity)
const resolved = current.aliases.get(itemId) ?? itemId
const revision =
(revisions.get(resolved) ??
Math.max(
current.items.get(resolved)?.revision ?? 0,
current.tombstones.get(resolved) ?? 0
)) + 1
revisions.set(resolved, revision)
return mutation.kind === 'item'
? { kind: 'item', itemId, revision, body: mutation.body }
: { kind: 'tombstone', itemId, revision }
})
const row: JournalLifecycleBatchRow = {
kind: 'lifecycle-batch',
settlementId,
mutations: built,
...journalRowBase(current.epoch, seq, options.fence, ts),
...(options.recovered ? { recovered: options.recovered } : {})
}
if (Buffer.byteLength(JSON.stringify(row), 'utf8') + 1 > MAX_JOURNAL_LIFECYCLE_BATCH_BYTES) {
throw new Error('journal_lifecycle_batch_byte_bound_exceeded')
}
return row
}
}
export function journalRowBase(
epoch: string,
seq: number,
@@ -1,5 +1,6 @@
import { describe, expect, it } from 'vitest'
import { parseJournalRow } from './journal-row-schema'
import { AGENT_SESSION_JOURNAL_SCHEMA_VERSION } from '../../../shared/agent-session-journal-types'
import { MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS, parseJournalRow } from './journal-row-schema'
const BASE = { v: 1, epoch: 'epoch-1', seq: 1, fence: 1, ts: 1 }
@@ -8,6 +9,41 @@ function parse(row: Record<string, unknown>): boolean {
}
describe('journal row validation', () => {
it('upcasts v1 rows to the current schema without changing their body', () => {
const parsed = parseJournalRow(
JSON.stringify({
...BASE,
kind: 'item',
itemId: 'i-1',
revision: 1,
body: { kind: 'status', text: 'from schema v1' }
})
)
expect(parsed).toEqual({
ok: true,
row: expect.objectContaining({
v: AGENT_SESSION_JOURNAL_SCHEMA_VERSION,
body: { kind: 'status', text: 'from schema v1' }
})
})
})
it('treats future-version rows as unreadable before validating future body shapes', () => {
expect(
parseJournalRow(
JSON.stringify({
...BASE,
v: AGENT_SESSION_JOURNAL_SCHEMA_VERSION + 1,
kind: 'item',
itemId: 'future',
revision: 1,
body: { kind: 'future-render-kind', payload: { anything: true } }
})
)
).toEqual({ ok: false, unreadable: true })
})
it('accepts every fully-formed row shape this build writes', () => {
expect(
parse({
@@ -189,4 +225,16 @@ describe('journal row validation', () => {
})
).toBe(true)
})
it('rejects lifecycle batches beyond the persisted mutation bound', () => {
const mutation = { kind: 'tombstone', itemId: 'i-1', revision: 1 }
expect(
parse({
...BASE,
kind: 'lifecycle-batch',
settlementId: 'settlement-1',
mutations: Array.from({ length: MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS + 1 }, () => mutation)
})
).toBe(false)
})
})
@@ -80,12 +80,32 @@ export type JournalDispatchRow = JournalRowBase & {
reason: string | null
}
export type JournalLifecycleMutation =
| {
kind: 'item'
itemId: string
revision: number
body: AgentJournalItemBody
}
| { kind: 'tombstone'; itemId: string; revision: number }
/** One durable append whose nested mutations share the outer ordering facts. */
export type JournalLifecycleBatchRow = JournalRowBase & {
kind: 'lifecycle-batch'
settlementId: string
mutations: JournalLifecycleMutation[]
}
export const MAX_JOURNAL_LIFECYCLE_BATCH_BYTES = 1_500_000
export const MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS = 200
export type JournalRow =
| JournalEpochRow
| JournalItemRow
| JournalTombstoneRow
| JournalSubmissionRow
| JournalDispatchRow
| JournalLifecycleBatchRow
export type JournalRowParse =
| { ok: true; row: JournalRow }
@@ -94,7 +114,14 @@ export type JournalRowParse =
/** A future schema version. The host must not write or compact this journal. */
| { ok: false; unreadable: true }
const ROW_KINDS = new Set(['epoch', 'item', 'tombstone', 'submission', 'dispatch'])
const ROW_KINDS = new Set([
'epoch',
'item',
'tombstone',
'submission',
'dispatch',
'lifecycle-batch'
])
export function serializeJournalRow(row: JournalRow): string {
return JSON.stringify(row)
@@ -191,9 +218,34 @@ function isJournalRow(record: Record<string, unknown>): record is JournalRow {
(record.reason === null || typeof record.reason === 'string')
)
}
if (record.kind === 'lifecycle-batch') {
return (
typeof record.settlementId === 'string' &&
record.settlementId.length > 0 &&
Array.isArray(record.mutations) &&
record.mutations.length > 0 &&
record.mutations.length <= MAX_JOURNAL_LIFECYCLE_BATCH_MUTATIONS &&
Buffer.byteLength(JSON.stringify(record), 'utf8') + 1 <= MAX_JOURNAL_LIFECYCLE_BATCH_BYTES &&
record.mutations.every(isLifecycleMutation)
)
}
return typeof record.reason === 'string' && isPlainObject(record.providerHandle)
}
function isLifecycleMutation(value: unknown): value is JournalLifecycleMutation {
if (!isPlainObject(value) || typeof value.itemId !== 'string') {
return false
}
if (value.kind === 'tombstone') {
return Number.isInteger(value.revision)
}
return (
value.kind === 'item' &&
Number.isInteger(value.revision) &&
isAdmissibleAgentJournalItemBody(value.body)
)
}
/** Approximate on-disk cost of a row, used for the per-session size bound. */
export function journalRowByteLength(row: JournalRow): number {
return Buffer.byteLength(serializeJournalRow(row), 'utf8') + 1
@@ -0,0 +1,362 @@
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import type { AgentJournalItemBody } from '../../../shared/agent-session-journal-types'
import { AGENT_SESSION_JOURNAL_SCHEMA_VERSION } from '../../../shared/agent-session-journal-types'
import { readJournalBlob } from './journal-blob-store'
import { appendJournalRows } from './journal-log-file'
import { JournalLifecycleAdmission } from './journal-lifecycle-admission'
import { JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES } from './journal-lifecycle-capacity'
import { loadJournal } from './journal-open'
import { boundPayload, DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
import { journalRowByteLength, type JournalRow } from './journal-row-schema'
import { JournalRowWriter } from './journal-row-writer'
import { JournalAppendBudget } from './journal-write-guards'
const SESSION_ID = 'session-1'
function row(seq: number, ts: number): JournalRow {
return {
v: AGENT_SESSION_JOURNAL_SCHEMA_VERSION,
epoch: 'epoch-1',
seq,
fence: 0,
ts,
kind: 'item',
itemId: 'item-1',
revision: 1,
body: { kind: 'status', text: 'ambiguous append' }
}
}
function rowWithBlob(seq: number, ts: number, output: ReturnType<typeof boundPayload>): JournalRow {
return {
v: AGENT_SESSION_JOURNAL_SCHEMA_VERSION,
epoch: 'epoch-1',
seq,
fence: 0,
ts,
kind: 'item',
itemId: 'item-with-blob',
revision: 1,
body: {
kind: 'tool-call',
name: 'shell',
input: {},
state: 'completed',
output
}
}
}
function runningToolRow(seq: number, ts: number, itemId = 'running-tool'): JournalRow {
return {
v: AGENT_SESSION_JOURNAL_SCHEMA_VERSION,
epoch: 'epoch-1',
seq,
fence: 0,
ts,
kind: 'item',
itemId,
revision: 1,
body: runningToolBody()
}
}
function runningToolBody(): AgentJournalItemBody {
return { kind: 'tool-call', name: 'shell', input: {}, state: 'running' }
}
describe('journal row writer read-only latch', () => {
let root: string
let readOnly = false
beforeEach(async () => {
root = await mkdtemp(join(tmpdir(), 'orca-journal-row-writer-'))
readOnly = false
})
afterEach(async () => {
await rm(root, { recursive: true, force: true })
})
it('enforces the lifecycle append rate and allows a retry after the window', () => {
const appendWindowMs = 100
const budget = new JournalAppendBudget(SESSION_ID, {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
maxAppendsPerWindow: 1,
appendWindowMs
})
budget.assertLifecycle(row(1, 1), 0)
expect(() => budget.assertLifecycle(row(2, 1), 0)).toThrow(
expect.objectContaining({ code: 'journal_rate_exceeded' })
)
expect(() => budget.assertLifecycle(row(2, appendWindowMs + 1), 0)).not.toThrow()
})
it('refuses lifecycle reservations once aggregate append capacity is saturated', () => {
const admission = new JournalLifecycleAdmission(SESSION_ID, 1_000_000, (itemId) => itemId, 2)
expect(admission.reserve({ id: 'first', bytes: 1, appendSlots: 1 }, 0)).toBe(true)
expect(admission.reserve({ id: 'second', bytes: 1, appendSlots: 1 }, 0)).toBe(true)
expect(admission.reserve({ id: 'third', bytes: 1, appendSlots: 1 }, 0)).toBe(false)
})
function writerHarness(
overrides: {
limits?: typeof DEFAULT_JOURNAL_PAYLOAD_LIMITS
physicalBytes?: number
appendRows?: (journalDir: string, rows: readonly JournalRow[]) => Promise<void>
commit?: (row: JournalRow, physicalBytes: number) => void
} = {}
) {
const limits = overrides.limits ?? DEFAULT_JOURNAL_PAYLOAD_LIMITS
const lifecycleAdmission = new JournalLifecycleAdmission(
SESSION_ID,
limits.maxSessionBytes,
(itemId) => itemId
)
let physicalBytes = overrides.physicalBytes ?? 0
let nextSequence = 1
const committedRows: JournalRow[] = []
const writer = new JournalRowWriter({
journalDir: root,
sessionId: SESSION_ID,
budget: new JournalAppendBudget(SESSION_ID, limits),
lifecycleAdmission,
autoCompact: false,
compaction: { minTailRows: 0, retainTailMs: 0 },
now: () => 1,
serialize: (run) => run(),
readOnly: () => readOnly,
setReadOnly: (value) => {
readOnly = value
},
physicalBytes: () => physicalBytes,
highestFence: () => 0,
nextSequence: () => nextSequence,
tailRows: () => committedRows,
referencedBlobDigests: () => new Set(),
compact: async () => undefined,
commit: (row, nextPhysicalBytes) => {
overrides.commit?.(row, nextPhysicalBytes)
committedRows.push(row)
physicalBytes = nextPhysicalBytes
nextSequence = row.seq + 1
},
...(overrides.appendRows ? { appendRows: overrides.appendRows } : {})
})
return { writer, lifecycleAdmission, committedRows }
}
it('latches read-only when a post-append failure makes durability ambiguous', async () => {
let committed = false
const writer = new JournalRowWriter({
journalDir: root,
sessionId: 'session-1',
budget: new JournalAppendBudget('session-1', DEFAULT_JOURNAL_PAYLOAD_LIMITS),
lifecycleAdmission: new JournalLifecycleAdmission(
'session-1',
DEFAULT_JOURNAL_PAYLOAD_LIMITS.maxSessionBytes,
(itemId) => itemId
),
autoCompact: false,
compaction: { minTailRows: 0, retainTailMs: 0 },
now: () => 1,
serialize: (run) => run(),
readOnly: () => readOnly,
setReadOnly: (value) => {
readOnly = value
},
physicalBytes: () => 0,
highestFence: () => 0,
nextSequence: () => 1,
tailRows: () => [],
referencedBlobDigests: () => new Set(),
compact: async () => undefined,
commit: () => {
committed = true
},
appendRows: async (journalDir, rows) => {
await appendJournalRows(journalDir, rows)
throw new Error('fsync failed after append')
}
})
await expect(writer.enqueue(row)).rejects.toThrow('fsync failed after append')
expect(readOnly).toBe(true)
expect(committed).toBe(false)
await expect(writer.enqueue(row)).rejects.toMatchObject({ code: 'journal_read_only' })
})
it('keeps blobs for a durable row when a post-append crash is reported', async () => {
const payload = 'durable blob payload'.repeat(2_000)
const bounded = boundPayload(payload, {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
inlineHeadBytes: 32
})
const writer = new JournalRowWriter({
journalDir: root,
sessionId: 'session-1',
budget: new JournalAppendBudget('session-1', DEFAULT_JOURNAL_PAYLOAD_LIMITS),
lifecycleAdmission: new JournalLifecycleAdmission(
'session-1',
DEFAULT_JOURNAL_PAYLOAD_LIMITS.maxSessionBytes,
(itemId) => itemId
),
autoCompact: false,
compaction: { minTailRows: 0, retainTailMs: 0 },
now: () => 1,
serialize: (run) => run(),
readOnly: () => readOnly,
setReadOnly: (value) => {
readOnly = value
},
physicalBytes: () => 0,
highestFence: () => 0,
nextSequence: () => 1,
tailRows: () => [],
referencedBlobDigests: () => new Set(),
compact: async () => undefined,
commit: () => undefined,
appendRows: async (journalDir, rows) => {
await appendJournalRows(journalDir, rows)
throw new Error('crash after row append')
}
})
await expect(
writer.enqueue(
(seq, ts) => rowWithBlob(seq, ts, bounded),
[{ digest: bounded.digest, payload }]
)
).rejects.toThrow('crash after row append')
expect(readOnly).toBe(true)
await expect(writer.enqueue(row)).rejects.toMatchObject({ code: 'journal_read_only' })
expect(await readJournalBlob(root, bounded.digest)).toBe(payload)
const reopened = await loadJournal(root, 'session-1')
const item = reopened?.state.items.get('item-with-blob')
expect(item?.body).toMatchObject({
kind: 'tool-call',
output: { digest: bounded.digest, truncated: true }
})
})
it('does not leak a lifecycle reservation after budget refusal', async () => {
const probe = runningToolRow(1, 1)
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
maxSessionBytes: JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES + journalRowByteLength(probe) - 1
}
const { writer, lifecycleAdmission, committedRows } = writerHarness({ limits })
await expect(writer.enqueue((seq, ts) => runningToolRow(seq, ts))).rejects.toMatchObject({
code: 'journal_bound_exceeded'
})
expect(lifecycleAdmission.state).toEqual({ reservedBytes: 0, reservedAppendSlots: 0 })
await expect(writer.enqueue(row)).resolves.toMatchObject({ kind: 'item', itemId: 'item-1' })
expect(
committedRows.map((entry) => (entry.kind === 'item' ? entry.itemId : 'non-item'))
).toEqual(['item-1'])
})
it('preflights existing durable-write temps before creating a blob or row', async () => {
const tempBytes = 512
const tempPath = join(root, 'log.jsonl.existing-write.tmp')
await writeFile(tempPath, 't'.repeat(tempBytes), 'utf8')
const probe = row(1, 1)
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
maxSessionBytes: tempBytes + journalRowByteLength(probe) - 1
}
const { writer, committedRows } = writerHarness({ limits })
await expect(writer.enqueue((seq, ts) => row(seq, ts))).rejects.toMatchObject({
code: 'journal_bound_exceeded'
})
expect(committedRows).toHaveLength(0)
expect(await readJournalBlob(root, 'a'.repeat(64))).toBeNull()
})
it('does not leak a lifecycle reservation after blob lookup failure', async () => {
const { writer, lifecycleAdmission } = writerHarness()
const digest = 'a'.repeat(64)
await writeFile(join(root, 'blobs'), 'not a directory', 'utf8')
await expect(
writer.enqueue((seq, ts) => runningToolRow(seq, ts), [{ digest, payload: 'payload' }])
).rejects.toThrow()
expect(lifecycleAdmission.state).toEqual({ reservedBytes: 0, reservedAppendSlots: 0 })
await rm(join(root, 'blobs'), { force: true })
await expect(writer.enqueue((seq, ts) => runningToolRow(seq, ts))).resolves.toMatchObject({
kind: 'item',
itemId: 'running-tool'
})
expect(lifecycleAdmission.state).toEqual({
reservedBytes: JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES,
reservedAppendSlots: 1
})
})
it('rolls back ordinary append-rate reservation after blob preflight failure', async () => {
const limits = {
...DEFAULT_JOURNAL_PAYLOAD_LIMITS,
maxAppendsPerWindow: 1,
appendWindowMs: 100
}
const { writer, committedRows } = writerHarness({ limits })
const payload = 'retryable blob payload'.repeat(100)
const bounded = boundPayload(payload, limits)
await writeFile(join(root, 'blobs'), 'not a directory', 'utf8')
await expect(
writer.enqueue(
(seq, ts) => rowWithBlob(seq, ts, bounded),
[{ digest: bounded.digest, payload }]
)
).rejects.toThrow()
await rm(join(root, 'blobs'), { force: true })
await expect(
writer.enqueue(
(seq, ts) => rowWithBlob(seq, ts, bounded),
[{ digest: bounded.digest, payload }]
)
).resolves.toMatchObject({ kind: 'item', itemId: 'item-with-blob' })
expect(committedRows).toHaveLength(1)
})
it('does not leak a lifecycle reservation after durable append failure', async () => {
const { writer, lifecycleAdmission } = writerHarness({
appendRows: async () => {
throw new Error('append failed before a durable row existed')
}
})
await expect(writer.enqueue((seq, ts) => runningToolRow(seq, ts))).rejects.toThrow(
'append failed before a durable row existed'
)
expect(readOnly).toBe(true)
expect(lifecycleAdmission.state).toEqual({ reservedBytes: 0, reservedAppendSlots: 0 })
})
it('does not leak a lifecycle reservation after reducer commit failure', async () => {
const { writer, lifecycleAdmission } = writerHarness({
commit: () => {
throw new Error('commit failed after durable append')
}
})
await expect(writer.enqueue((seq, ts) => runningToolRow(seq, ts))).rejects.toThrow(
'commit failed after durable append'
)
expect(lifecycleAdmission.state).toEqual({ reservedBytes: 0, reservedAppendSlots: 0 })
})
})
@@ -0,0 +1,188 @@
import {
budgetPressurePolicy,
journalTailCanShedRows,
journalTailIsReadyToCompact,
type JournalCompactionPolicy
} from './journal-compaction'
import { journalBlobFileSize, putJournalBlob, removeJournalBlob } from './journal-blob-store'
import { appendJournalRows } from './journal-log-file'
import { blobDigestsInBody } from './journal-reducer'
import { journalDirectoryBytes } from './journal-physical-quota'
import type { JournalLifecycleAdmission } from './journal-lifecycle-admission'
import { journalRowByteLength, type JournalRow } from './journal-row-schema'
import {
AgentSessionJournalError,
assertJournalFence,
assertJournalWritable,
type JournalAppendBudget
} from './journal-write-guards'
type JournalBlob = { digest: string; payload: string }
export type JournalRowWriterDeps = {
journalDir: string
sessionId: string
budget: JournalAppendBudget
lifecycleAdmission: JournalLifecycleAdmission
autoCompact: boolean
compaction: JournalCompactionPolicy
now: () => number
serialize: <T>(run: () => Promise<T>) => Promise<T>
readOnly: () => boolean
setReadOnly: (readOnly: boolean) => void
physicalBytes: () => number
highestFence: () => number
nextSequence: () => number
tailRows: () => readonly JournalRow[]
referencedBlobDigests: () => ReadonlySet<string>
compact: (now: number, policy: JournalCompactionPolicy) => Promise<void>
commit: (row: JournalRow, physicalBytes: number) => void
appendRows?: (journalDir: string, rows: readonly JournalRow[]) => Promise<void>
}
export class JournalRowWriter {
constructor(private readonly deps: JournalRowWriterDeps) {}
enqueue(
build: (seq: number, ts: number) => JournalRow,
blobs: readonly JournalBlob[] = []
): Promise<JournalRow> {
return this.deps.serialize(async () => {
assertJournalWritable(this.deps.readOnly(), this.deps.sessionId)
const ts = this.deps.now()
const row = build(this.deps.nextSequence(), ts)
assertJournalFence(row.fence, this.deps.highestFence())
// The in-memory counter is an optimization, not the quota source of
// truth: a prior crash may have left a durable-write temp beside the
// finals, and a concurrent/retried opener may have materialized files
// after the last commit callback. Recount before any speculative write
// so the peak check includes those bytes.
let physicalBytes = Math.max(
this.deps.physicalBytes(),
await journalDirectoryBytes(this.deps.journalDir)
)
const admission = this.deps.lifecycleAdmission.prepare(row, physicalBytes)
const newBlobs = await uniqueNewBlobs(this.deps.journalDir, blobs)
const blobBytes = newBlobs.reduce(
(total, blob) => total + Buffer.byteLength(blob.payload, 'utf8'),
0
)
const budgetCompaction = budgetPressurePolicy(this.deps.compaction)
let effectiveSize = physicalBytes + blobBytes + admission.protectedBytes
if (
this.deps.autoCompact &&
this.deps.budget.wouldExceedSize(row, effectiveSize) &&
journalTailCanShedRows(this.deps.tailRows(), budgetCompaction, ts)
) {
await this.deps.compact(ts, budgetCompaction)
physicalBytes = this.deps.physicalBytes()
effectiveSize = physicalBytes + blobBytes + admission.protectedBytes
}
const lifecycleRateCheckpoint = admission.lifecycleCovered
? this.deps.budget.checkpoint()
: null
const appendRateCheckpoint = this.deps.budget.checkpoint()
let committed = false
let appendMayHaveLanded = false
try {
if (admission.lifecycleCovered) {
this.deps.budget.assertReservedLifecycle(row, effectiveSize)
} else {
this.deps.budget.assert(row, ts, effectiveSize)
}
const appendedBytes = blobBytes + journalRowByteLength(row)
if (
physicalBytes + appendedBytes >
this.deps.budget.maxSessionBytes - admission.protectedBytes
) {
throw new AgentSessionJournalError(
'journal_bound_exceeded',
`agent-session journal for ${this.deps.sessionId} reached its ${this.deps.budget.maxSessionBytes}-byte physical bound`
)
}
await this.commitFiles(row, newBlobs, () => {
appendMayHaveLanded = true
})
physicalBytes += appendedBytes
this.deps.commit(row, physicalBytes)
this.deps.lifecycleAdmission.commit(admission)
committed = true
} catch (error) {
if (!committed && lifecycleRateCheckpoint) {
this.deps.budget.restore(lifecycleRateCheckpoint)
}
if (!committed && !appendMayHaveLanded) {
this.deps.budget.restore(appendRateCheckpoint)
}
throw error
}
if (
this.deps.autoCompact &&
journalTailIsReadyToCompact(this.deps.tailRows(), this.deps.compaction, ts)
) {
await this.deps.compact(ts, this.deps.compaction)
}
return row
})
}
private async commitFiles(
row: JournalRow,
blobs: readonly JournalBlob[],
markAppendLanded: () => void
): Promise<void> {
const persisted: string[] = []
let appendMayHaveLanded = false
try {
for (const blob of blobs) {
await putJournalBlob(this.deps.journalDir, blob.digest, blob.payload)
persisted.push(blob.digest)
}
appendMayHaveLanded = true
markAppendLanded()
await (this.deps.appendRows ?? appendJournalRows)(this.deps.journalDir, [row])
} catch (error) {
if (appendMayHaveLanded) {
this.deps.setReadOnly(true)
throw error
}
const retained = this.referencedBlobDigestsIncludingTail()
for (const digest of persisted) {
if (!retained.has(digest)) {
await removeJournalBlob(this.deps.journalDir, digest)
}
}
throw error
}
}
private referencedBlobDigestsIncludingTail(): Set<string> {
const retained = new Set(this.deps.referencedBlobDigests())
for (const row of this.deps.tailRows()) {
if (row.kind === 'item') {
blobDigestsInBody(row.body, retained)
} else if (row.kind === 'lifecycle-batch') {
for (const mutation of row.mutations) {
if (mutation.kind === 'item') {
blobDigestsInBody(mutation.body, retained)
}
}
}
}
return retained
}
}
async function uniqueNewBlobs(
journalDir: string,
blobs: readonly JournalBlob[]
): Promise<JournalBlob[]> {
const unique = new Map(blobs.map((blob) => [blob.digest, blob]))
const result: JournalBlob[] = []
for (const blob of unique.values()) {
if ((await journalBlobFileSize(journalDir, blob.digest)) === null) {
result.push(blob)
}
}
return result
}
@@ -1,12 +1,15 @@
import type {
AgentJournalCursor,
AgentJournalItemBody,
AgentJournalItemIdentity,
AgentJournalMessageItem,
AgentJournalResetReason,
AgentSessionJournalIdentity
} from '../../../shared/agent-session-journal-types'
import type { JournalCompactionPolicy } from './journal-compaction'
import type { JournalLoad } from './journal-open'
import type { JournalPayloadLimits } from './journal-payload-bounds'
import type { JournalLifecycleMutationInput } from './journal-row-builders'
import type { JournalRow } from './journal-row-schema'
export type AgentSessionJournalOptions = {
@@ -40,3 +43,27 @@ export type JournalAppendResult = {
itemId: string
revision: number
}
export type JournalItemAppendOptions = { fence: number; observedAt?: number; recovered?: true }
export type JournalBlobInput = { digest: string; payload: string }
export type JournalTombstoneInput = { fence: number }
export type JournalLifecycleBatchInput = {
settlementId: string
mutations: readonly JournalLifecycleMutationInput[]
fence: number
recovered?: true
}
export type JournalSubmissionInput = {
clientMessageId: string
payloadFingerprint: string
body: AgentJournalMessageItem
fence: number
}
export type JournalItemAppendInput = {
identity: AgentJournalItemIdentity
body: AgentJournalItemBody
options: JournalItemAppendOptions
}
@@ -0,0 +1,10 @@
import type { AgentSessionJournalOptions } from './journal-store-contracts'
import { AgentSessionJournal } from './journal-store'
export async function openAgentSessionJournal(
options: AgentSessionJournalOptions
): Promise<AgentSessionJournal> {
const journal = new AgentSessionJournal(options)
await journal.open()
return journal
}
@@ -0,0 +1,77 @@
import type { AgentJournalSnapshot } from '../../../shared/agent-session-journal-types'
import { malformedRowsDisclosure, quarantineCorruptSuffix } from './journal-corruption-quarantine'
import { ensureJournalDir } from './journal-log-file'
import { loadJournal, type JournalLoad } from './journal-open'
import { assertJournalPhysicalCapacity, journalDirectoryBytes } from './journal-physical-quota'
import type { JournalRow } from './journal-row-schema'
export function journalStoreLoadedFields(loaded: JournalLoad) {
return {
state: loaded.state,
tailRows: loaded.tailRows,
compactedThrough: loaded.compactedThrough,
sizeBytes: loaded.sizeBytes,
readOnly: loaded.readOnly,
malformedRows: loaded.malformedRows
}
}
export async function openJournalStoreState(input: {
journalDir: string
sessionId: string
maxBytes: number
loaded: JournalLoad | null | undefined
start: () => Promise<void>
adopt: (loaded: JournalLoad) => void
tailRows: () => readonly JournalRow[]
snapshot: () => AgentJournalSnapshot
rebuildLifecycle: (snapshot: AgentJournalSnapshot, physicalBytes: number) => void
appendDisclosure: (
identity: ReturnType<typeof malformedRowsDisclosure>['identity'],
body: ReturnType<typeof malformedRowsDisclosure>['body'],
fence: number
) => Promise<unknown>
highestFence: () => number
malformedRows: () => number
readOnly: () => boolean
setPhysicalBytes: (bytes: number) => void
}): Promise<void> {
await ensureJournalDir(input.journalDir)
input.setPhysicalBytes(
await assertJournalPhysicalCapacity({
journalDir: input.journalDir,
sessionId: input.sessionId,
maxBytes: input.maxBytes
})
)
const loaded =
input.loaded !== undefined
? input.loaded
: await loadJournal(input.journalDir, input.sessionId, { maxBytes: input.maxBytes })
if (!loaded) {
await input.start()
input.setPhysicalBytes(await journalDirectoryBytes(input.journalDir))
return
}
input.adopt(loaded)
if (loaded.corrupt && !loaded.readOnly) {
await quarantineCorruptSuffix(input.journalDir, input.tailRows(), loaded.quarantineRemainder, {
sessionId: input.sessionId,
maxBytes: input.maxBytes
})
}
let physicalBytes = await journalDirectoryBytes(input.journalDir)
input.setPhysicalBytes(physicalBytes)
// A future-schema/read-only journal is inspection-only. Its reduced state is
// intentionally empty, and rebuilding reservations from it would mutate the
// in-memory quota model (and could influence later admission decisions).
if (!loaded.readOnly) {
input.rebuildLifecycle(input.snapshot(), physicalBytes)
}
if (input.malformedRows() > 0 && !input.readOnly()) {
const disclosure = malformedRowsDisclosure(input.malformedRows())
await input.appendDisclosure(disclosure.identity, disclosure.body, input.highestFence())
}
physicalBytes = await journalDirectoryBytes(input.journalDir)
input.setPhysicalBytes(physicalBytes)
}
@@ -0,0 +1,313 @@
import { mkdtemp, readdir, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import type {
AgentJournalItemBody,
AgentJournalItemIdentity,
AgentSessionJournalIdentity
} from '../../../shared/agent-session-journal-types'
import { JOURNAL_LOG_FILE, JOURNAL_SNAPSHOT_FILE } from './journal-log-file'
import { openAgentSessionJournal } from './journal-store'
const IDENTITY: AgentSessionJournalIdentity = {
sessionId: 'session-1',
workspaceId: 'ws-1',
hostId: 'host-1',
agent: 'codex',
providerHandle: { kind: 'codex', threadId: 'thread-1' }
}
let root: string
let clock = 1_000
function tick(): number {
clock += 1
return clock
}
function item(ordinal: number): AgentJournalItemIdentity {
return { provider: 'codex', threadId: 'thread-1', turnId: 'turn-1', ordinal }
}
function body(value: string): AgentJournalItemBody {
return { kind: 'message', role: 'assistant', blocks: [{ type: 'text', text: value }] }
}
async function open(overrides: Partial<Parameters<typeof openAgentSessionJournal>[0]> = {}) {
return openAgentSessionJournal({
identity: IDENTITY,
journalDir: root,
now: tick,
mintEpoch: () => `epoch-${clock}`,
...overrides
})
}
beforeEach(async () => {
root = await mkdtemp(join(tmpdir(), 'orca-journal-'))
clock = 1_000
})
afterEach(async () => {
await rm(root, { recursive: true, force: true })
})
describe('schema', () => {
it('quarantines an invalid compacted snapshot without replacing its tail', async () => {
const journal = await open({ compaction: { minTailRows: 2, retainTailMs: 0 } })
for (let index = 0; index < 6; index += 1) {
await journal.appendItem(item(index), body(`m${index}`), { fence: 1 })
}
await journal.compact()
const epoch = journal.epoch
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const logPath = join(root, JOURNAL_LOG_FILE)
const invalidSnapshot = '{"folded history":'
await writeFile(snapshotPath, invalidSnapshot, 'utf-8')
const retainedTail = await readFile(logPath, 'utf-8')
expect(retainedTail).not.toContain('"kind":"epoch"')
const reopened = await open()
expect(reopened.epoch).toBe(epoch)
expect(await readFile(logPath, 'utf-8')).toBe(retainedTail)
const quarantined = (await readdir(root)).find((name) =>
name.startsWith('quarantine-snapshot-')
)
expect(quarantined).toBeDefined()
expect(await readFile(join(root, quarantined!), 'utf-8')).toBe(invalidSnapshot)
})
it('degrades to read-only on a row from a newer build, without skipping or deleting it', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
const future = JSON.stringify({
v: 99,
kind: 'item',
epoch: journal.epoch,
seq: 99,
fence: 1,
ts: 1,
itemId: 'future',
revision: 1,
body: { kind: 'status', text: 'from a newer host' }
})
const before = await readFile(logPath, 'utf-8')
await writeFile(logPath, `${before}${future}\n`, 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(true)
await expect(reopened.appendItem(item(1), body('b'), { fence: 1 })).rejects.toMatchObject({
code: 'journal_read_only'
})
await expect(reopened.compact()).rejects.toMatchObject({ code: 'journal_read_only' })
expect(reopened.readSince({ epoch: reopened.epoch, sequence: 0 })).toEqual({
ok: false,
reset: 'schema_unreadable'
})
// The unreadable row is still on disk, and nothing was compacted past it.
expect(await readFile(logPath, 'utf-8')).toContain('"v":99')
})
it('skips a malformed line without giving up the journal, and discloses the skip', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}{not json\n`, 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(false)
const items = reopened.snapshot().items
// The surviving row is untouched…
expect(items.some((entry) => entry.body.kind === 'message')).toBe(true)
// …and the skip is visible in the timeline instead of silently swallowed.
expect(
items.some(
(entry) => entry.body.kind === 'status' && entry.body.text.includes('could not be read')
)
).toBe(true)
})
it('keeps one disclosure row across reopens instead of stacking duplicates', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}{not json\n`, 'utf-8')
await open()
const reopened = await open()
expect(
reopened
.snapshot()
.items.filter(
(entry) => entry.body.kind === 'status' && entry.body.text.includes('could not be read')
)
).toHaveLength(1)
})
it('repairs a torn tail before acknowledging the next append', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
const intact = await readFile(logPath, 'utf-8')
await writeFile(logPath, intact.slice(0, -1), 'utf-8')
await journal.appendItem(item(1), body('b'), { fence: 1 })
const reopened = await open()
expect(reopened.snapshot().items.map((entry) => entry.body)).toEqual([body('a'), body('b')])
})
// Transcripts are full of emoji and CJK, so the repair's file offsets must be
// bytes: string indices would truncate mid-character and corrupt the prefix.
it('repairs a torn tail whose rows contain multi-byte characters', async () => {
const journal = await open()
await journal.appendItem(item(0), body('안녕하세요 🌊 café'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
const intact = await readFile(logPath)
// Kill mid-row: keep the complete first row plus a fragment of the second.
const torn = Buffer.concat([intact, Buffer.from('{"seq":2,"kind":"it', 'utf-8')])
await writeFile(logPath, torn)
await journal.appendItem(item(1), body('b'), { fence: 1 })
const reopened = await open()
expect(reopened.snapshot().items.map((entry) => entry.body)).toEqual([
body('안녕하세요 🌊 café'),
body('b')
])
})
it('degrades to read-only when the snapshot comes from a newer schema', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(true)
await expect(reopened.appendItem(item(1), body('b'), { fence: 1 })).rejects.toMatchObject({
code: 'journal_read_only'
})
})
it('preserves a future-version snapshot with an unknown body kind in place instead of quarantining it', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
// The version advances because bodies changed: a valid newer snapshot
// carries kinds this build cannot parse and must stay unreadable in place.
snapshot.items = [
{
itemId: 'codex:thread-1:turn-1:1',
revision: 1,
body: { kind: 'future-render-kind', payload: { anything: true } },
sequence: 1,
observedAt: 1_000
}
]
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
const entries = await readdir(root)
expect(entries.some((name) => name.startsWith('quarantine-'))).toBe(false)
expect(entries.includes(JOURNAL_SNAPSHOT_FILE)).toBe(true)
expect(reopened.isReadOnly).toBe(true)
expect(reopened.snapshot().items).toHaveLength(0)
await expect(reopened.appendItem(item(1), body('b'), { fence: 1 })).rejects.toMatchObject({
code: 'journal_read_only'
})
})
it('keeps the future-version snapshot bytes when the schema escape hatch rolls the epoch', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
snapshot.items = [
{
itemId: 'codex:thread-1:turn-1:1',
revision: 1,
body: { kind: 'future-render-kind', payload: { anything: true } },
sequence: 1,
observedAt: 1_000
}
]
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
// Still live in place before the explicit escape hatch runs.
expect((await readdir(root)).some((name) => name.startsWith('quarantine-'))).toBe(false)
await reopened.rollEpoch('schema_unreadable', 2)
expect(reopened.isReadOnly).toBe(false)
const quarantine = (await readdir(root)).find((name) => name.startsWith('quarantine-'))
expect(quarantine).toBeDefined()
expect(await readFile(join(root, quarantine!), 'utf-8')).toContain('future-render-kind')
})
it('reopens a log holding an admitted malformed-percent item id without throwing', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
// `parseJournalRow` admits any string itemId, so replay must degrade a
// malformed percent key to an opaque id instead of throwing URIError.
const malformedKeyRow = JSON.stringify({
v: 1,
epoch: journal.epoch,
seq: journal.cursor().sequence + 1,
fence: 1,
ts: 1,
kind: 'item',
itemId: '%',
revision: 1,
body: { kind: 'message', role: 'user', blocks: [{ type: 'text', text: 'hi' }] }
})
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}${malformedKeyRow}\n`, 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(false)
expect(reopened.snapshot().items.some((entry) => entry.itemId === '%')).toBe(true)
})
it('allows the explicit schema-unreadable epoch escape hatch while preserving the old files', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
await reopened.rollEpoch('schema_unreadable', 2)
expect(reopened.isReadOnly).toBe(false)
expect(reopened.snapshot().items).toHaveLength(0)
expect((await readdir(root)).some((name) => name.startsWith('quarantine-'))).toBe(true)
})
it('keeps the unreadable log suffix in the schema escape quarantine', async () => {
const journal = await open()
const logPath = join(root, JOURNAL_LOG_FILE)
const future = JSON.stringify({
v: 99,
kind: 'item',
epoch: journal.epoch,
seq: 2,
fence: 1,
ts: 1,
itemId: 'future',
revision: 1,
body: { kind: 'status', text: 'preserve these bytes' }
})
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}${future}\n`, 'utf-8')
const reopened = await open()
await reopened.rollEpoch('schema_unreadable', 2)
const quarantine = (await readdir(root)).find((name) => name.startsWith('quarantine-'))
expect(quarantine).toBeDefined()
expect(await readFile(join(root, quarantine!), 'utf-8')).toContain('preserve these bytes')
})
})
@@ -20,11 +20,18 @@ import {
DEFAULT_JOURNAL_PAYLOAD_LIMITS
} from './journal-payload-bounds'
import { journalDirectoryFor, journalPathSegment } from './journal-paths'
import { journalDirectoryBytes } from './journal-physical-quota'
import type { JournalLifecycleMutationInput } from './journal-row-builders'
import {
AgentSessionJournalError,
openAgentSessionJournal,
type AgentSessionJournal
} from './journal-store'
import {
JOURNAL_DISPATCH_RESERVATION_BYTES,
JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES,
JOURNAL_TURN_TERMINAL_RESERVATION_BYTES
} from './journal-lifecycle-capacity'
const IDENTITY: AgentSessionJournalIdentity = {
sessionId: 'session-1',
@@ -428,7 +435,7 @@ describe('bounds', () => {
it('refuses a single row larger than the per-session size bound', async () => {
const journal = await open({
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 400 }
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 2_000 }
})
// Shedding the whole tail still cannot make room, so the bound holds.
await expect(
@@ -439,7 +446,7 @@ describe('bounds', () => {
it('refuses an append past the per-session size bound when compaction is off', async () => {
const journal = await open({
autoCompact: false,
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 400 }
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 2_000 }
})
await expect(
(async () => {
@@ -462,264 +469,323 @@ describe('bounds', () => {
})()
).rejects.toMatchObject({ code: 'journal_rate_exceeded' })
})
it('charges unique blobs and abandoned staging files to one physical quota', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 8_000 }
const journal = await open({ limits, autoCompact: false })
const payload = 'z'.repeat(1_200)
const bounded = boundPayload(payload, { ...limits, inlineHeadBytes: 8 })
const toolBody: AgentJournalItemBody = {
kind: 'tool-call',
name: 'command',
input: {},
state: 'completed',
output: bounded
}
await journal.appendItemWithBlobs(item(1), toolBody, [{ digest: bounded.digest, payload }], {
fence: 1
})
const afterFirst = await journalDirectoryBytes(root)
await journal.appendItemWithBlobs(item(2), toolBody, [{ digest: bounded.digest, payload }], {
fence: 1
})
const afterDuplicate = await journalDirectoryBytes(root)
expect(afterDuplicate - afterFirst).toBeLessThan(payload.length)
expect(await readdir(join(root, 'blobs'))).toEqual([bounded.digest])
expect(afterDuplicate).toBeLessThanOrEqual(limits.maxSessionBytes)
await writeFile(join(root, 'log.jsonl.abandoned.tmp'), 's'.repeat(2_000), 'utf8')
const physical = await journalDirectoryBytes(root)
await expect(
open({ limits: { ...limits, maxSessionBytes: physical - 1 } })
).rejects.toMatchObject({ code: 'journal_bound_exceeded' })
})
it('uses a running tool reservation when its authoritative blob cannot fit', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 220 * 1024 }
const journal = await open({ limits, autoCompact: false })
await journal.appendItem(
item(1),
{ kind: 'tool-call', name: 'command', input: {}, state: 'running' },
{ fence: 1 }
)
for (let ordinal = 10; ordinal < 100; ordinal += 1) {
try {
await journal.appendItem(item(ordinal), body('f'.repeat(4_000)), { fence: 1 })
} catch (error) {
expect(error).toMatchObject({ code: 'journal_bound_exceeded' })
break
}
}
const payload = 'o'.repeat(100 * 1024)
const bounded = boundPayload(payload, { ...limits, inlineHeadBytes: 16 * 1024 })
await journal.appendItemWithBlobs(
item(1),
{
kind: 'tool-call',
name: 'command',
input: {},
state: 'completed',
output: bounded
},
[{ digest: bounded.digest, payload }],
{ fence: 1 }
)
const tool = journal.snapshot().items.find((entry) => entry.itemId.includes('turn-1:1'))
expect(tool?.body).toEqual({
kind: 'tool-call',
name: 'command',
input: {},
state: 'completed'
})
expect(
journal
.snapshot()
.items.some(
(entry) =>
entry.body.kind === 'status' && entry.body.text.includes('could not be retained')
)
).toBe(true)
expect(await readJournalBlob(root, bounded.digest)).toBeNull()
expect(journal.lifecycleCapacityState()).toEqual({ reservedBytes: 0, reservedAppendSlots: 0 })
})
it('keeps cached physical bytes aligned after blob dedupe and compaction', async () => {
const limits = { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxSessionBytes: 45_000 }
const journal = await open({
limits,
autoCompact: false,
compaction: { minTailRows: 0, retainTailMs: 0 }
})
const payload = 'p'.repeat(20_000)
const bounded = boundPayload(payload, { ...limits, inlineHeadBytes: 8 })
const toolBody: AgentJournalItemBody = {
kind: 'tool-call',
name: 'command',
input: {},
state: 'completed',
output: bounded
}
await journal.appendItemWithBlobs(item(1), toolBody, [{ digest: bounded.digest, payload }], {
fence: 1
})
await journal.appendItemWithBlobs(item(2), toolBody, [{ digest: bounded.digest, payload }], {
fence: 1
})
await journal.compact(tick() + 10, { minTailRows: 0, retainTailMs: 0 })
const compactedBytes = await journalDirectoryBytes(root)
expect(compactedBytes).toBeLessThan(limits.maxSessionBytes)
await journal.appendItem(item(3), body('after compaction'), { fence: 1 })
expect(await journalDirectoryBytes(root)).toBeLessThanOrEqual(limits.maxSessionBytes)
expect(await readdir(join(root, 'blobs'))).toEqual([bounded.digest])
})
})
describe('schema', () => {
it('quarantines an invalid compacted snapshot without replacing its tail', async () => {
const journal = await open({ compaction: { minTailRows: 2, retainTailMs: 0 } })
for (let index = 0; index < 6; index += 1) {
await journal.appendItem(item(index), body(`m${index}`), { fence: 1 })
describe('lifecycle batches', () => {
it('uses a reserved append slot after ordinary rate pressure', async () => {
const journal = await open({
autoCompact: false,
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxAppendsPerWindow: 1, appendWindowMs: 60_000 }
})
const identity: AgentJournalItemIdentity = {
provider: 'orca',
clientMessageId: 'reserved-prompt'
}
const pending: AgentJournalItemBody = {
kind: 'approval',
title: 'Run a command?',
detail: null,
options: [{ id: 'accept', label: 'Allow' }],
resolution: { state: 'pending', selectedOptionId: null, resolvedBy: null, resolvedAt: null }
}
await journal.compact()
const epoch = journal.epoch
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const logPath = join(root, JOURNAL_LOG_FILE)
const invalidSnapshot = '{"folded history":'
await writeFile(snapshotPath, invalidSnapshot, 'utf-8')
const retainedTail = await readFile(logPath, 'utf-8')
expect(retainedTail).not.toContain('"kind":"epoch"')
const reopened = await open()
expect(reopened.epoch).toBe(epoch)
expect(await readFile(logPath, 'utf-8')).toBe(retainedTail)
const quarantined = (await readdir(root)).find((name) =>
name.startsWith('quarantine-snapshot-')
)
expect(quarantined).toBeDefined()
expect(await readFile(join(root, quarantined!), 'utf-8')).toBe(invalidSnapshot)
})
it('degrades to read-only on a row from a newer build, without skipping or deleting it', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
const future = JSON.stringify({
v: 99,
kind: 'item',
epoch: journal.epoch,
seq: 99,
// The pending row spends the only ordinary slot while reserving its
// terminal append slot for recovery.
await journal.appendLifecycleBatch({
settlementId: 'reserved-start',
fence: 1,
ts: 1,
itemId: 'future',
revision: 1,
body: { kind: 'status', text: 'from a newer host' }
mutations: [{ kind: 'item', identity, body: pending }]
})
const before = await readFile(logPath, 'utf-8')
await writeFile(logPath, `${before}${future}\n`, 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(true)
await expect(reopened.appendItem(item(1), body('b'), { fence: 1 })).rejects.toMatchObject({
code: 'journal_read_only'
})
await expect(reopened.compact()).rejects.toMatchObject({ code: 'journal_read_only' })
expect(reopened.readSince({ epoch: reopened.epoch, sequence: 0 })).toEqual({
ok: false,
reset: 'schema_unreadable'
})
// The unreadable row is still on disk, and nothing was compacted past it.
expect(await readFile(logPath, 'utf-8')).toContain('"v":99')
})
it('skips a malformed line without giving up the journal, and discloses the skip', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}{not json\n`, 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(false)
const items = reopened.snapshot().items
// The surviving row is untouched…
expect(items.some((entry) => entry.body.kind === 'message')).toBe(true)
// …and the skip is visible in the timeline instead of silently swallowed.
expect(
items.some(
(entry) => entry.body.kind === 'status' && entry.body.text.includes('could not be read')
await expect(
journal.appendItem(
identity,
{
...pending,
resolution: {
state: 'resolved',
selectedOptionId: 'accept',
resolvedBy: 'test',
resolvedAt: 1
}
},
{ fence: 1 }
)
).toBe(true)
).resolves.toBeDefined()
})
it('keeps one disclosure row across reopens instead of stacking duplicates', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}{not json\n`, 'utf-8')
it('rate-limits an unreserved lifecycle batch', async () => {
const journal = await open({
autoCompact: false,
limits: { ...DEFAULT_JOURNAL_PAYLOAD_LIMITS, maxAppendsPerWindow: 1, appendWindowMs: 60_000 }
})
const mutation = (id: string): JournalLifecycleMutationInput => ({
kind: 'item',
identity: { provider: 'orca', clientMessageId: id },
body: { kind: 'status', text: 'provider diagnostic' }
})
await journal.appendLifecycleBatch({
settlementId: 'unreserved-1',
fence: 1,
mutations: [mutation('one')]
})
await expect(
journal.appendLifecycleBatch({
settlementId: 'unreserved-2',
fence: 1,
mutations: [mutation('two')]
})
).rejects.toMatchObject({ code: 'journal_rate_exceeded' })
})
await open()
const reopened = await open()
it('rebuilds dispatch and turn reservations for pending submissions after reopen', async () => {
const journal = await open({ autoCompact: false })
await journal.appendSubmission({
clientMessageId: 'pending-send',
payloadFingerprint: 'fingerprint',
body: { kind: 'message', role: 'user', blocks: [{ type: 'text', text: 'hello' }] },
fence: 1
})
const reopened = await open({ autoCompact: false })
expect(reopened.lifecycleCapacityState()).toEqual({
reservedBytes: JOURNAL_DISPATCH_RESERVATION_BYTES + JOURNAL_TURN_TERMINAL_RESERVATION_BYTES,
reservedAppendSlots: 2
})
})
it('deduplicates concurrent submissions before appending a second row', async () => {
const journal = await open()
const input = {
settlementId: 'concurrent-settlement',
fence: 1,
mutations: [{ kind: 'item' as const, identity: item(1), body: body('settled') }]
}
const [first, replay] = await Promise.all([
journal.appendLifecycleBatch(input),
journal.appendLifecycleBatch(input)
])
expect([replay, first.sequence]).toEqual([first, 2])
})
it('applies every mutation at one sequence and deduplicates a replay across reopen', async () => {
const journal = await open({ autoCompact: false })
const turn: AgentJournalItemIdentity = {
provider: 'legacy',
agent: 'codex',
sessionId: 'session-1',
recordId: 'turn-lifecycle:turn-1'
}
await journal.appendItem(turn, { kind: 'status', text: 'working' }, { fence: 1 })
const settled = await journal.appendLifecycleBatch({
settlementId: 'exit:turn-1',
fence: 1,
mutations: [
{ kind: 'item', identity: item(1), body: body('tool settled') },
{
kind: 'item',
identity: { provider: 'orca', clientMessageId: 'exit-status' },
body: { kind: 'status', text: 'Provider exited' }
},
{ kind: 'tombstone', identity: turn }
]
})
const atSettlement = journal
.snapshot()
.items.filter((entry) => entry.sequence === settled.sequence)
expect(atSettlement).toHaveLength(2)
expect(
journal
.snapshot()
.items.some((entry) => entry.body.kind === 'status' && entry.body.text === 'working')
).toBe(false)
await journal.compact(tick() + 10, { minTailRows: 0, retainTailMs: 0 })
const reopened = await open({ autoCompact: false })
const beforeReplay = reopened.cursor()
const replay = await reopened.appendLifecycleBatch({
settlementId: 'exit:turn-1',
fence: 1,
mutations: [{ kind: 'item', identity: item(9), body: body('must not appear') }]
})
expect(replay).toEqual(beforeReplay)
expect(
reopened
.snapshot()
.items.filter(
(entry) => entry.body.kind === 'status' && entry.body.text.includes('could not be read')
.items.some(
(entry) =>
entry.body.kind === 'message' &&
entry.body.blocks.some(
(block) => block.type === 'text' && block.text === 'must not appear'
)
)
).toHaveLength(1)
).toBe(false)
})
it('repairs a torn tail before acknowledging the next append', async () => {
it('reserves and releases terminal prompts created inside lifecycle batches', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
const intact = await readFile(logPath, 'utf-8')
await writeFile(logPath, intact.slice(0, -1), 'utf-8')
await journal.appendItem(item(1), body('b'), { fence: 1 })
const reopened = await open()
expect(reopened.snapshot().items.map((entry) => entry.body)).toEqual([body('a'), body('b')])
})
// Transcripts are full of emoji and CJK, so the repair's file offsets must be
// bytes: string indices would truncate mid-character and corrupt the prefix.
it('repairs a torn tail whose rows contain multi-byte characters', async () => {
const journal = await open()
await journal.appendItem(item(0), body('안녕하세요 🌊 café'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
const intact = await readFile(logPath)
// Kill mid-row: keep the complete first row plus a fragment of the second.
const torn = Buffer.concat([intact, Buffer.from('{"seq":2,"kind":"it', 'utf-8')])
await writeFile(logPath, torn)
await journal.appendItem(item(1), body('b'), { fence: 1 })
const reopened = await open()
expect(reopened.snapshot().items.map((entry) => entry.body)).toEqual([
body('안녕하세요 🌊 café'),
body('b')
])
})
it('degrades to read-only when the snapshot comes from a newer schema', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(true)
await expect(reopened.appendItem(item(1), body('b'), { fence: 1 })).rejects.toMatchObject({
code: 'journal_read_only'
})
})
it('preserves a future-version snapshot with an unknown body kind in place instead of quarantining it', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
// The version advances because bodies changed: a valid newer snapshot
// carries kinds this build cannot parse and must stay unreadable in place.
snapshot.items = [
{
itemId: 'codex:thread-1:turn-1:1',
revision: 1,
body: { kind: 'future-render-kind', payload: { anything: true } },
sequence: 1,
observedAt: 1_000
const identity: AgentJournalItemIdentity = { provider: 'orca', clientMessageId: 'prompt-1' }
const pending: AgentJournalItemBody = {
kind: 'approval',
title: 'Run a command?',
detail: null,
options: [{ id: 'accept', label: 'Allow' }],
resolution: {
state: 'pending',
selectedOptionId: null,
resolvedBy: null,
resolvedAt: null
}
]
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
}
const reopened = await open()
const entries = await readdir(root)
expect(entries.some((name) => name.startsWith('quarantine-'))).toBe(false)
expect(entries.includes(JOURNAL_SNAPSHOT_FILE)).toBe(true)
expect(reopened.isReadOnly).toBe(true)
expect(reopened.snapshot().items).toHaveLength(0)
await expect(reopened.appendItem(item(1), body('b'), { fence: 1 })).rejects.toMatchObject({
code: 'journal_read_only'
await journal.appendLifecycleBatch({
settlementId: 'prompt-start',
fence: 1,
mutations: [{ kind: 'item', identity, body: pending }]
})
})
it('keeps the future-version snapshot bytes when the schema escape hatch rolls the epoch', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
snapshot.items = [
expect(journal.lifecycleCapacityState()).toEqual({
reservedBytes: JOURNAL_ITEM_TERMINAL_RESERVATION_BYTES,
reservedAppendSlots: 1
})
await journal.appendItem(
identity,
{
itemId: 'codex:thread-1:turn-1:1',
revision: 1,
body: { kind: 'future-render-kind', payload: { anything: true } },
sequence: 1,
observedAt: 1_000
}
]
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
// Still live in place before the explicit escape hatch runs.
expect((await readdir(root)).some((name) => name.startsWith('quarantine-'))).toBe(false)
...pending,
resolution: {
state: 'resolved',
selectedOptionId: 'accept',
resolvedBy: 'test',
resolvedAt: tick()
}
},
{ fence: 1 }
)
await reopened.rollEpoch('schema_unreadable', 2)
expect(reopened.isReadOnly).toBe(false)
const quarantine = (await readdir(root)).find((name) => name.startsWith('quarantine-'))
expect(quarantine).toBeDefined()
expect(await readFile(join(root, quarantine!), 'utf-8')).toContain('future-render-kind')
})
it('reopens a log holding an admitted malformed-percent item id without throwing', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const logPath = join(root, JOURNAL_LOG_FILE)
// `parseJournalRow` admits any string itemId, so replay must degrade a
// malformed percent key to an opaque id instead of throwing URIError.
const malformedKeyRow = JSON.stringify({
v: 1,
epoch: journal.epoch,
seq: journal.cursor().sequence + 1,
fence: 1,
ts: 1,
kind: 'item',
itemId: '%',
revision: 1,
body: { kind: 'message', role: 'user', blocks: [{ type: 'text', text: 'hi' }] }
expect(journal.lifecycleCapacityState()).toEqual({
reservedBytes: 0,
reservedAppendSlots: 0
})
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}${malformedKeyRow}\n`, 'utf-8')
const reopened = await open()
expect(reopened.isReadOnly).toBe(false)
expect(reopened.snapshot().items.some((entry) => entry.itemId === '%')).toBe(true)
})
it('allows the explicit schema-unreadable epoch escape hatch while preserving the old files', async () => {
const journal = await open()
await journal.appendItem(item(0), body('a'), { fence: 1 })
const snapshotPath = join(root, JOURNAL_SNAPSHOT_FILE)
const snapshot = JSON.parse(await readFile(snapshotPath, 'utf-8')) as Record<string, unknown>
snapshot.v = 99
await writeFile(snapshotPath, JSON.stringify(snapshot), 'utf-8')
const reopened = await open()
await reopened.rollEpoch('schema_unreadable', 2)
expect(reopened.isReadOnly).toBe(false)
expect(reopened.snapshot().items).toHaveLength(0)
expect((await readdir(root)).some((name) => name.startsWith('quarantine-'))).toBe(true)
})
it('keeps the unreadable log suffix in the schema escape quarantine', async () => {
const journal = await open()
const logPath = join(root, JOURNAL_LOG_FILE)
const future = JSON.stringify({
v: 99,
kind: 'item',
epoch: journal.epoch,
seq: 2,
fence: 1,
ts: 1,
itemId: 'future',
revision: 1,
body: { kind: 'status', text: 'preserve these bytes' }
})
await writeFile(logPath, `${await readFile(logPath, 'utf-8')}${future}\n`, 'utf-8')
const reopened = await open()
await reopened.rollEpoch('schema_unreadable', 2)
const quarantine = (await readdir(root)).find((name) => name.startsWith('quarantine-'))
expect(quarantine).toBeDefined()
expect(await readFile(join(root, quarantine!), 'utf-8')).toContain('preserve these bytes')
})
})
@@ -6,30 +6,19 @@ import type {
AgentJournalCursor,
AgentJournalItemBody,
AgentJournalItemIdentity,
AgentJournalMessageItem,
AgentJournalSnapshot,
AgentJournalSubmission,
AgentSessionJournalIdentity
} from '../../../shared/agent-session-journal-types'
import { agentJournalItemKey } from '../../../shared/agent-session-journal-item-key'
import {
budgetPressurePolicy,
compactJournal,
DEFAULT_JOURNAL_COMPACTION_POLICY,
journalTailCanShedRows,
journalTailIsReadyToCompact,
type JournalCompactionPolicy
} from './journal-compaction'
import { replaceJournalEpoch, type JournalReplacementItem } from './journal-epoch-replacement'
import type { JournalReplacementItem } from './journal-epoch-replacement'
import { readJournalSince } from './journal-cursor'
import { publishNewEpoch } from './journal-epoch-rollover'
import { appendJournalRows, ensureJournalDir } from './journal-log-file'
import {
malformedRowsDisclosure,
quarantineCorruptSuffix,
quarantineUnreadableSchema
} from './journal-corruption-quarantine'
import { loadJournal, type JournalLoad } from './journal-open'
import type { JournalLoad } from './journal-open'
import { DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
import { markJournalPendingSubmissionsUnknown } from './journal-pending-submission-recovery'
import {
@@ -42,36 +31,34 @@ import {
} from './journal-reducer'
import {
journalDispatchRowBuilder,
journalItemRowBuilder,
journalSubmissionRowBuilder,
journalTombstoneRowBuilder
} from './journal-row-builders'
import type {
AgentSessionJournalOptions,
JournalAppendResult,
JournalBlobInput,
JournalItemAppendOptions,
JournalLifecycleBatchInput,
JournalReadSince,
JournalSubmissionInput,
JournalTombstoneInput,
ResolveDispatchInput
} from './journal-store-contracts'
import {
journalRowByteLength,
type AgentJournalEpochReason,
type JournalRow
} from './journal-row-schema'
import {
assertJournalFence,
assertJournalWritable,
JournalAppendBudget
} from './journal-write-guards'
import type { AgentJournalEpochReason, JournalRow } from './journal-row-schema'
import { assertJournalWritable, JournalAppendBudget } from './journal-write-guards'
import { journalDirectoryBytes } from './journal-physical-quota'
import type { JournalLifecycleReservation } from './journal-lifecycle-capacity'
import { JournalLifecycleAdmission } from './journal-lifecycle-admission'
import { JournalRowWriter } from './journal-row-writer'
import { JournalEpochController } from './journal-epoch-controller'
import { journalStoreLoadedFields, openJournalStoreState } from './journal-store-open'
import { JournalItemAppender } from './journal-item-appender'
import { JournalLifecycleBatchAppender } from './journal-lifecycle-batch-appender'
export { AgentSessionJournalError } from './journal-write-guards'
export async function openAgentSessionJournal(
options: AgentSessionJournalOptions
): Promise<AgentSessionJournal> {
const journal = new AgentSessionJournal(options)
await journal.open()
return journal
}
export { openAgentSessionJournal } from './journal-store-factory'
export class AgentSessionJournal {
private readonly identity: AgentSessionJournalIdentity
@@ -89,6 +76,11 @@ export class AgentSessionJournal {
private sizeBytes = 0
private readOnly = false
private malformedRows = 0
private readonly lifecycleAdmission: JournalLifecycleAdmission
private readonly rowWriter: JournalRowWriter
private readonly epochController: JournalEpochController
private readonly itemAppender: JournalItemAppender
private readonly lifecycleBatchAppender: JournalLifecycleBatchAppender
/** Serializes sequence assignment with the durable write behind it. */
private writes: Promise<unknown> = Promise.resolve()
@@ -105,6 +97,63 @@ export class AgentSessionJournal {
this.mintEpoch = options.mintEpoch ?? randomUUID
this.loaded = options.loaded
this.state = createJournalReducerState(options.identity.sessionId, '')
this.lifecycleAdmission = new JournalLifecycleAdmission(
options.identity.sessionId,
this.budget.maxSessionBytes,
(itemId) => resolveJournalItemId(this.state, itemId),
this.budget.maxAppendsPerWindow
)
this.rowWriter = new JournalRowWriter({
journalDir: this.journalDir,
sessionId: options.identity.sessionId,
budget: this.budget,
lifecycleAdmission: this.lifecycleAdmission,
autoCompact: this.autoCompact,
compaction: this.compaction,
now: this.now,
serialize: (run) => this.serializeWrite(run),
readOnly: () => this.readOnly,
setReadOnly: (readOnly) => {
this.readOnly = readOnly
},
physicalBytes: () => this.sizeBytes,
highestFence: () => this.state.highestFence,
nextSequence: () => this.state.lastSequence + 1,
tailRows: () => this.tailRows,
referencedBlobDigests: () => referencedBlobDigests(this.state),
compact: (now, policy) => this.compact(now, policy),
commit: (row, physicalBytes) => {
applyJournalRow(this.state, row)
this.tailRows.push(row)
this.sizeBytes = physicalBytes
}
})
this.epochController = new JournalEpochController({
identity: this.identity,
journalDir: this.journalDir,
budget: this.budget,
compaction: this.compaction,
now: this.now,
mintEpoch: this.mintEpoch,
serialize: (run) => this.serializeWrite(run),
readOnly: () => this.readOnly,
setReadOnly: (readOnly) => {
this.readOnly = readOnly
},
highestFence: () => this.state.highestFence,
cursor: this.cursor,
adopt: (loaded) => this.adoptLoadedJournal(loaded)
})
this.itemAppender = new JournalItemAppender({
journal: () => this,
state: () => this.state,
enqueue: (build, blobs) => this.enqueue(build, blobs)
})
this.lifecycleBatchAppender = new JournalLifecycleBatchAppender({
state: () => this.state,
cursor: this.cursor,
enqueue: (build) => this.enqueue(build)
})
}
get isReadOnly(): boolean {
@@ -126,26 +175,24 @@ export class AgentSessionJournal {
}
async open(): Promise<void> {
await ensureJournalDir(this.journalDir)
const loaded =
this.loaded !== undefined
? this.loaded
: await loadJournal(this.journalDir, this.identity.sessionId)
if (!loaded) {
await this.startEpoch('session_created', 0)
return
}
this.adoptLoadedJournal(loaded)
if (loaded.corrupt && !loaded.readOnly) {
// The epoch stays put: no intact history is discarded to recover.
await quarantineCorruptSuffix(this.journalDir, this.tailRows, loaded.quarantineRemainder)
}
if (this.malformedRows > 0 && !this.readOnly) {
const disclosure = malformedRowsDisclosure(this.malformedRows)
await this.appendItem(disclosure.identity, disclosure.body, {
fence: this.state.highestFence
})
}
await openJournalStoreState({
journalDir: this.journalDir,
sessionId: this.identity.sessionId,
maxBytes: this.budget.maxSessionBytes,
loaded: this.loaded,
start: () => this.epochController.start('session_created', 0),
adopt: (loaded) => this.adoptLoadedJournal(loaded),
tailRows: () => this.tailRows,
snapshot: this.snapshot,
rebuildLifecycle: (snapshot, bytes) => this.lifecycleAdmission.rebuild(snapshot, bytes),
appendDisclosure: (identity, body, fence) => this.appendItem(identity, body, { fence }),
highestFence: () => this.state.highestFence,
malformedRows: () => this.malformedRows,
readOnly: () => this.readOnly,
setPhysicalBytes: (bytes) => {
this.sizeBytes = bytes
}
})
}
cursor = (): AgentJournalCursor => ({
@@ -162,16 +209,29 @@ export class AgentSessionJournal {
/** The durable answer to "did my send land?" — a reconnecting client asking
* again gets this instead of re-sending. */
receiptFor(clientMessageId: string): AgentJournalAcceptanceReceipt | null {
return this.state.receipts.get(clientMessageId) ?? null
}
receiptFor = (clientMessageId: string): AgentJournalAcceptanceReceipt | null =>
this.state.receipts.get(clientMessageId) ?? null
canonicalItemId = (itemId: string): string => resolveJournalItemId(this.state, itemId)
referencedBlobDigests(): Set<string> {
return referencedBlobDigests(this.state)
reserveLifecycleCapacity(token: JournalLifecycleReservation): Promise<boolean> {
return this.serializeCapacityMutation(async () => {
this.sizeBytes = await journalDirectoryBytes(this.journalDir)
return this.lifecycleAdmission.reserve(token, this.sizeBytes)
})
}
transferLifecycleCapacity(fromId: string, toId: string): Promise<boolean> {
return this.serializeCapacityMutation(() => this.lifecycleAdmission.transfer(fromId, toId))
}
releaseLifecycleCapacity(id: string): Promise<void> {
return this.serializeCapacityMutation(() => this.lifecycleAdmission.release(id))
}
lifecycleCapacityState = (): { reservedBytes: number; reservedAppendSlots: number } =>
this.lifecycleAdmission.state
readSince(cursor: AgentJournalCursor): JournalReadSince {
return readJournalSince(
{ state: this.state, tailRows: this.tailRows, readOnly: this.readOnly },
@@ -185,21 +245,24 @@ export class AgentSessionJournal {
appendItem(
identity: AgentJournalItemIdentity,
body: AgentJournalItemBody,
options: { fence: number; observedAt?: number; recovered?: true } = { fence: 0 }
options: JournalItemAppendOptions = { fence: 0 }
): Promise<JournalAppendResult> {
const itemId = agentJournalItemKey(identity)
return this.enqueue(journalItemRowBuilder(() => this.state, identity, body, options)).then(
(row) => ({
cursor: { epoch: row.epoch, sequence: row.seq },
itemId,
revision: (row as Extract<JournalRow, { kind: 'item' }>).revision
})
)
return this.itemAppender.append(identity, body, options)
}
/** Blob-before-row admission on the same serialized path as sequence assignment. */
appendItemWithBlobs(
identity: AgentJournalItemIdentity,
body: AgentJournalItemBody,
blobs: readonly JournalBlobInput[],
options: JournalItemAppendOptions = { fence: 0 }
): Promise<JournalAppendResult> {
return this.itemAppender.appendWithBlobs(identity, body, blobs, options)
}
appendTombstone(
identity: AgentJournalItemIdentity,
options: { fence: number }
options: JournalTombstoneInput
): Promise<AgentJournalCursor> {
const itemId = agentJournalItemKey(identity)
return this.enqueue(journalTombstoneRowBuilder(() => this.state, itemId, options.fence)).then(
@@ -207,17 +270,16 @@ export class AgentSessionJournal {
)
}
appendLifecycleBatch(input: JournalLifecycleBatchInput): Promise<AgentJournalCursor> {
return this.lifecycleBatchAppender.append(input)
}
/**
* Write-ahead submission row. It is durable before the caller dispatches
* anything, and it doubles as the optimistic user bubble so an accepted echo
* reconciles into an existing slot instead of appending a second copy.
*/
appendSubmission(input: {
clientMessageId: string
payloadFingerprint: string
body: AgentJournalMessageItem
fence: number
}): Promise<AgentJournalCursor> {
appendSubmission(input: JournalSubmissionInput): Promise<AgentJournalCursor> {
return this.enqueue(
journalSubmissionRowBuilder(() => this.state, this.identity.providerHandle, input)
).then((row) => ({ epoch: row.epoch, sequence: row.seq }))
@@ -254,25 +316,19 @@ export class AgentSessionJournal {
tailRows: this.tailRows,
policy,
now,
maxSessionBytes: this.budget.maxSessionBytes
maxSessionBytes: this.budget.maxSessionBytes,
sessionId: this.identity.sessionId
})
this.tailRows = result.tailRows
this.compactedThrough = result.compactedThrough
this.state.oldestSequence = result.oldestSequence
this.sizeBytes = this.tailRows.reduce((total, row) => total + journalRowByteLength(row), 0)
this.sizeBytes = await journalDirectoryBytes(this.journalDir)
}
/** The escape hatch for corruption, an unreconcilable prefix, a forked handle,
* and an unreadable schema. It invalidates every cursor; clients reload. */
async rollEpoch(reason: AgentJournalEpochReason, fence: number): Promise<AgentJournalCursor> {
if (reason !== 'schema_unreadable') {
assertJournalWritable(this.readOnly, this.identity.sessionId)
} else if (this.readOnly) {
await quarantineUnreadableSchema(this.journalDir)
}
await this.startEpoch(reason, fence)
this.readOnly = false
return this.cursor()
return this.epochController.roll(reason, fence)
}
replaceEpochItems(
@@ -280,48 +336,11 @@ export class AgentSessionJournal {
fence: number,
items: readonly JournalReplacementItem[]
): Promise<AgentJournalCursor> {
const run = this.writes.then(async () => {
assertJournalWritable(this.readOnly, this.identity.sessionId)
assertJournalFence(fence, this.state.highestFence)
await replaceJournalEpoch({
journalDir: this.journalDir,
identity: this.identity,
reason,
fence,
items,
budget: this.budget.fork(),
compaction: this.compaction,
now: this.now,
mintEpoch: this.mintEpoch,
onSnapshotPublished: (loaded) => this.adoptLoadedJournal(loaded)
})
return this.cursor()
})
this.writes = run.catch(() => undefined)
return run
}
private async startEpoch(reason: AgentJournalEpochReason, fence: number): Promise<void> {
this.adoptLoadedJournal(
await publishNewEpoch({
journalDir: this.journalDir,
sessionId: this.identity.sessionId,
providerHandle: this.identity.providerHandle,
epoch: this.mintEpoch(),
reason,
fence,
now: this.now()
})
)
return this.epochController.replace(reason, fence, items)
}
private adoptLoadedJournal(loaded: JournalLoad): void {
this.state = loaded.state
this.tailRows = loaded.tailRows
this.compactedThrough = loaded.compactedThrough
this.sizeBytes = loaded.sizeBytes
this.readOnly = loaded.readOnly
this.malformedRows = loaded.malformedRows
Object.assign(this, journalStoreLoadedFields(loaded))
}
/**
@@ -329,32 +348,18 @@ export class AgentSessionJournal {
* SAME reducer replay uses — all inside one serialized step, so concurrent
* callers cannot interleave and mint the same sequence.
*/
private enqueue(build: (seq: number, ts: number) => JournalRow): Promise<JournalRow> {
const run = this.writes.then(async () => {
assertJournalWritable(this.readOnly, this.identity.sessionId)
const ts = this.now()
const row = build(this.state.lastSequence + 1, ts)
assertJournalFence(row.fence, this.state.highestFence)
const budgetCompaction = budgetPressurePolicy(this.compaction)
if (
this.autoCompact &&
this.budget.wouldExceedSize(row, this.sizeBytes) &&
journalTailCanShedRows(this.tailRows, budgetCompaction, ts)
) {
await this.compact(ts, budgetCompaction)
}
this.budget.assert(row, ts, this.sizeBytes)
await appendJournalRows(this.journalDir, [row])
applyJournalRow(this.state, row)
this.tailRows.push(row)
this.sizeBytes += journalRowByteLength(row)
// Nothing else calls compact(), so without this the log only ever grows —
// until the size bound refuses every append for the rest of the session.
if (this.autoCompact && journalTailIsReadyToCompact(this.tailRows, this.compaction, ts)) {
await this.compact(ts)
}
return row
})
private enqueue(
build: (seq: number, ts: number) => JournalRow,
blobs: readonly JournalBlobInput[] = []
): Promise<JournalRow> {
return this.rowWriter.enqueue(build, blobs)
}
private serializeCapacityMutation = <T>(runMutation: () => Promise<T> | T): Promise<T> =>
this.serializeWrite(async () => runMutation())
private serializeWrite<T>(runWrite: () => Promise<T>): Promise<T> {
const run = this.writes.then(runWrite)
this.writes = run.catch(() => undefined)
return run
}
@@ -0,0 +1,58 @@
import type {
AgentJournalItemBody,
AgentJournalItemIdentity
} from '../../../shared/agent-session-journal-types'
import type { AgentSessionJournal } from './journal-store'
import type { JournalAppendResult } from './journal-store-contracts'
import { AgentSessionJournalError } from './journal-write-guards'
import { boundToolInput, DEFAULT_JOURNAL_PAYLOAD_LIMITS } from './journal-payload-bounds'
export async function appendToolOutputFallback(input: {
journal: AgentSessionJournal
error: unknown
identity: AgentJournalItemIdentity
body: AgentJournalItemBody
blobs: readonly { digest: string; payload: string }[]
itemId: string
fence: number
}): Promise<JournalAppendResult> {
if (
!(input.error instanceof AgentSessionJournalError) ||
input.error.code !== 'journal_bound_exceeded' ||
input.body.kind !== 'tool-call' ||
input.body.state === 'running' ||
input.blobs.length === 0
) {
throw input.error
}
const digest = input.blobs[0]?.digest ?? 'unknown'
const cursor = await input.journal.appendLifecycleBatch({
settlementId: `tool-output-unavailable:${input.itemId}:${digest}`,
fence: input.fence,
mutations: [
{
kind: 'item',
identity: input.identity,
body: {
kind: 'tool-call',
name: input.body.name,
input: boundToolInput(input.body.input, DEFAULT_JOURNAL_PAYLOAD_LIMITS),
state: input.body.state
}
},
{
kind: 'item',
identity: { provider: 'orca', clientMessageId: `output-unavailable:${input.itemId}` },
body: {
kind: 'status',
text: 'The tool completed, but its output could not be retained within the session storage limit.'
}
}
]
})
const item = input.journal.snapshot().items.find((entry) => entry.itemId === input.itemId)
if (!item) {
throw new Error('journal_tool_output_fallback_lost')
}
return { cursor, itemId: input.itemId, revision: item.revision }
}
@@ -60,6 +60,20 @@ export class JournalAppendBudget {
return this.limits.maxSessionBytes
}
get maxAppendsPerWindow(): number {
return this.limits.maxAppendsPerWindow
}
/** Capture rate state so a speculative append can be rolled back safely. */
checkpoint(): { windowStart: number; appendsInWindow: number } {
return { windowStart: this.windowStart, appendsInWindow: this.appendsInWindow }
}
restore(checkpoint: { windowStart: number; appendsInWindow: number }): void {
this.windowStart = checkpoint.windowStart
this.appendsInWindow = checkpoint.appendsInWindow
}
wouldExceedSize(row: JournalRow, sizeBytes: number): boolean {
return sizeBytes + journalRowByteLength(row) > this.limits.maxSessionBytes
}
@@ -71,16 +85,50 @@ export class JournalAppendBudget {
`agent-session journal for ${this.sessionId} reached its ${this.limits.maxSessionBytes}-byte bound`
)
}
if (ts - this.windowStart >= this.limits.appendWindowMs) {
this.windowStart = ts
this.appendsInWindow = 0
this.assertRate(ts)
}
/** Lifecycle capacity cannot bypass the session-wide append rate. */
assertLifecycle(row: JournalRow, sizeBytes: number): void {
if (this.wouldExceedSize(row, sizeBytes)) {
throw new AgentSessionJournalError(
'journal_bound_exceeded',
`agent-session journal for ${this.sessionId} reached its ${this.limits.maxSessionBytes}-byte bound`
)
}
this.appendsInWindow += 1
if (this.appendsInWindow > this.limits.maxAppendsPerWindow) {
this.assertRate(row.ts)
}
/**
* Consume a lifecycle row covered by a pre-reserved append slot. Reserved
* rows still observe the physical quota, but do not spend ordinary window
* rate headroom that may be needed by unrelated traffic.
*/
assertReservedLifecycle(row: JournalRow, sizeBytes: number): void {
if (this.wouldExceedSize(row, sizeBytes)) {
throw new AgentSessionJournalError(
'journal_bound_exceeded',
`agent-session journal for ${this.sessionId} reached its ${this.limits.maxSessionBytes}-byte bound`
)
}
}
private assertRate(ts: number): void {
let windowStart = this.windowStart
let appendsInWindow = this.appendsInWindow
if (ts - windowStart >= this.limits.appendWindowMs) {
windowStart = ts
appendsInWindow = 0
}
appendsInWindow += 1
if (appendsInWindow > this.limits.maxAppendsPerWindow) {
// A refusal must not consume a slot, so a later retry can succeed.
throw new AgentSessionJournalError(
'journal_rate_exceeded',
`agent-session journal for ${this.sessionId} exceeded ${this.limits.maxAppendsPerWindow} appends per ${this.limits.appendWindowMs}ms`
)
}
this.windowStart = windowStart
this.appendsInWindow = appendsInWindow
}
}
@@ -73,6 +73,18 @@ describe('agent-session delta coalescer', () => {
])
})
it('appends ten thousand deltas without rebuilding the retained prefix per token', () => {
const clock = manualClock()
const { instance, emitted } = coalescer(clock)
for (let index = 0; index < 10_000; index += 1) {
instance.append('item-1', 'x')
}
clock.fire()
expect(emitted).toEqual([['item-1', 'x'.repeat(10_000)]])
})
it('does not re-emit a stream with no new text', () => {
const clock = manualClock()
const { instance, emitted } = coalescer(clock)
@@ -95,6 +107,53 @@ describe('agent-session delta coalescer', () => {
expect(clock.pendingCount()).toBe(0)
})
it('retains dirty state and surfaces admission failure for retry', () => {
const clock = manualClock()
let reject = true
const emitted: string[] = []
const instance = createAgentSessionDeltaCoalescer({
schedule: clock.schedule,
emit: (_key, text) => {
if (reject) {
return false
}
emitted.push(text)
return true
}
})
instance.append('item-1', 'retry me')
expect(instance.flush('item-1')).toBe(false)
expect(emitted).toEqual([])
reject = false
expect(instance.flush('item-1')).toBe(true)
expect(emitted).toEqual(['retry me'])
})
it('does not evict buffered output when flushing the oldest stream is backpressured', () => {
const clock = manualClock()
let reject = true
const emitted: [string, string][] = []
const instance = createAgentSessionDeltaCoalescer({
maxStreams: 1,
schedule: clock.schedule,
emit: (key, text) => {
if (reject) {
return false
}
emitted.push([key, text])
return true
}
})
instance.append('first', 'preserve me')
expect(instance.append('second', 'new stream')).toBe(false)
expect(instance.snapshot('first')?.text).toBe('preserve me')
reject = false
expect(instance.append('second', 'new stream')).toBe(true)
expect(emitted).toEqual([['first', 'preserve me']])
})
it('drops a forgotten stream without emitting it, because its final body already landed', () => {
const clock = manualClock()
const { instance, emitted } = coalescer(clock)
@@ -127,4 +186,61 @@ describe('agent-session delta coalescer', () => {
expect(clock.windows()).toEqual([5])
})
it('bounds retained UTF-8 text while continuing to count observed bytes', () => {
const clock = manualClock()
const emitted: { text: string; observedBytes: number; truncated: boolean }[] = []
const instance = createAgentSessionDeltaCoalescer({
maxRetainedBytes: 40,
schedule: clock.schedule,
emit: (_key, _text, snapshot) => emitted.push(snapshot)
})
instance.append('item-1', 'éé')
instance.append('item-1', `${'é'.repeat(20)}more`)
clock.fire()
instance.append('item-1', 'ignored')
clock.fire()
expect(emitted).toEqual([
{
text: 'ééé\n[Orca: streamed output truncated]',
observedBytes: 48,
truncated: true
}
])
expect(instance.snapshot('item-1')).toEqual({
text: 'ééé\n[Orca: streamed output truncated]',
observedBytes: 55,
truncated: true
})
})
it('bounds aggregate retained stream text across independent items', () => {
const clock = manualClock()
const emitted = new Map<string, { text: string; observedBytes: number; truncated: boolean }>()
const instance = createAgentSessionDeltaCoalescer({
maxRetainedBytes: 80,
maxTotalRetainedBytes: 120,
schedule: clock.schedule,
emit: (key, _text, snapshot) => emitted.set(key, snapshot)
})
instance.append('item-1', 'a'.repeat(80))
instance.append('item-2', 'b'.repeat(80))
instance.append('item-3', 'c'.repeat(80))
clock.fire()
instance.append('item-3', 'c'.repeat(80))
clock.fire()
const snapshots = ['item-1', 'item-2', 'item-3'].map((key) => instance.snapshot(key))
const retainedBytes = snapshots.reduce(
(total, snapshot) => total + Buffer.byteLength(snapshot?.text ?? '', 'utf8'),
0
)
expect(retainedBytes).toBeLessThanOrEqual(120)
expect(snapshots.map((snapshot) => snapshot?.observedBytes)).toEqual([80, 80, 160])
expect(snapshots.map((snapshot) => snapshot?.truncated)).toEqual([false, true, true])
expect(emitted.get('item-3')).toEqual({ text: '', observedBytes: 80, truncated: true })
})
})
@@ -15,24 +15,45 @@
* text still reads as streaming. */
export const AGENT_SESSION_DELTA_COALESCE_MS = 60
/** Matches the admitted live provider-record envelope. The framer owns record
* admission; this independently prevents many legal deltas from rebuilding an
* unbounded string after they have crossed that boundary. */
export const AGENT_SESSION_STREAMED_TEXT_MAX_BYTES = 16 * 1024 * 1024
export const AGENT_SESSION_STREAMED_TEXT_TOTAL_MAX_BYTES = 32 * 1024 * 1024
export const AGENT_SESSION_STREAMED_TEXT_TRUNCATION_MARKER = '\n[Orca: streamed output truncated]'
export const AGENT_SESSION_MAX_STREAMS = 256
export type AgentSessionDeltaSnapshot = {
text: string
observedBytes: number
truncated: boolean
}
export type AgentSessionDeltaCoalescerDeps = {
/** Called with the FULL text accumulated for the key, not the increment. */
emit: (key: string, text: string) => void
emit: (key: string, text: string, snapshot: AgentSessionDeltaSnapshot) => unknown
windowMs?: number
maxRetainedBytes?: number
maxTotalRetainedBytes?: number
/** Maximum distinct item streams retained at once. */
maxStreams?: number
/** Injected by tests so a window can be driven without real time. */
schedule?: (run: () => void, ms: number) => () => void
}
export type AgentSessionDeltaCoalescer = {
append: (key: string, delta: string) => void
/** Returns false when a full stream cannot be flushed to admit this new key. */
append: (key: string, delta: string) => boolean
/** Emit one stream now, if it has unflushed text. */
flush: (key: string) => void
flush: (key: string) => boolean
/** Emit every stream now. The lifecycle bypass. */
flushAll: () => void
flushAll: () => boolean
/** Drop a stream without emitting — its authoritative body arrived, so the
* accumulated text is now the stale copy. */
forget: (key: string) => void
dispose: () => void
/** Bounded last-known state for terminalizing a rejected completion. */
snapshot: (key: string) => AgentSessionDeltaSnapshot | null
}
function defaultSchedule(run: () => void, ms: number): () => void {
@@ -46,48 +67,179 @@ export function createAgentSessionDeltaCoalescer(
): AgentSessionDeltaCoalescer {
const windowMs = deps.windowMs ?? AGENT_SESSION_DELTA_COALESCE_MS
const schedule = deps.schedule ?? defaultSchedule
const streams = new Map<string, { text: string; dirty: boolean }>()
const maxRetainedBytes = deps.maxRetainedBytes ?? AGENT_SESSION_STREAMED_TEXT_MAX_BYTES
const maxTotalRetainedBytes =
deps.maxTotalRetainedBytes ?? AGENT_SESSION_STREAMED_TEXT_TOTAL_MAX_BYTES
const maxStreams = Math.max(1, deps.maxStreams ?? AGENT_SESSION_MAX_STREAMS)
const streams = new Map<
string,
{
chunks: string[]
retainedBytes: number
observedBytes: number
truncated: boolean
dirty: boolean
}
>()
let totalRetainedBytes = 0
let cancelTimer: (() => void) | null = null
const streamOrder = new Map<string, number>()
let nextOrder = 0
const flushKey = (key: string): void => {
const flushKey = (key: string): boolean => {
const stream = streams.get(key)
if (!stream?.dirty) {
return
return true
}
const text = stream.chunks.join('')
const emitted = deps.emit(key, text, {
text,
observedBytes: stream.observedBytes,
truncated: stream.truncated
})
if (emitted === false) {
return false
}
stream.dirty = false
deps.emit(key, stream.text)
return true
}
const flushAll = (): void => {
const scheduleFlush = (): void => {
cancelTimer ??= schedule(() => {
cancelTimer = null
flushAll()
}, windowMs)
}
const flushAll = (): boolean => {
cancelTimer?.()
cancelTimer = null
let emitted = true
for (const key of streams.keys()) {
flushKey(key)
emitted = flushKey(key) && emitted
}
if (!emitted) {
scheduleFlush()
}
return emitted
}
return {
append: (key, delta) => {
const stream = streams.get(key) ?? { text: '', dirty: false }
stream.text += delta
stream.dirty = true
let stream = streams.get(key)
if (!stream) {
// Evict the oldest stream before admitting a new attacker-controlled
// id. Flush first so the retained prefix is durably visible.
if (streams.size >= maxStreams) {
const oldest = [...streamOrder.entries()].sort((a, b) => a[1] - b[1])[0]?.[0]
if (oldest) {
// Under sink backpressure the oldest stream must remain available
// for a later retry; dropping it would lose already-observed output.
if (!flushKey(oldest)) {
return false
}
const evicted = streams.get(oldest)
if (evicted) {
totalRetainedBytes -= evicted.retainedBytes
}
streams.delete(oldest)
streamOrder.delete(oldest)
}
}
stream = {
chunks: [],
retainedBytes: 0,
observedBytes: 0,
truncated: false,
dirty: false
}
streamOrder.set(key, nextOrder++)
}
stream.observedBytes += Buffer.byteLength(delta, 'utf8')
if (!stream.truncated) {
const availableTotal = Math.max(0, maxTotalRetainedBytes - totalRetainedBytes)
const streamLimit = Math.min(maxRetainedBytes, stream.retainedBytes + availableTotal)
const next = appendWithinUtf8ByteLimit(
stream.chunks,
stream.retainedBytes,
delta,
streamLimit
)
totalRetainedBytes += next.retainedBytes - stream.retainedBytes
stream.chunks = next.chunks
stream.retainedBytes = next.retainedBytes
stream.truncated = next.truncated
stream.dirty = true
}
streams.set(key, stream)
// One timer for every stream: a shared deadline bounds latency the same
// way and costs one wakeup per window instead of one per stream.
cancelTimer ??= schedule(() => {
cancelTimer = null
flushAll()
}, windowMs)
scheduleFlush()
return true
},
flush: flushKey,
flushAll,
forget: (key) => {
streams.delete(key)
const stream = streams.get(key)
if (stream) {
totalRetainedBytes -= stream.retainedBytes
streams.delete(key)
streamOrder.delete(key)
}
},
dispose: () => {
cancelTimer?.()
cancelTimer = null
streams.clear()
streamOrder.clear()
totalRetainedBytes = 0
},
snapshot: (key) => {
const stream = streams.get(key)
return stream
? {
text: stream.chunks.join(''),
observedBytes: stream.observedBytes,
truncated: stream.truncated
}
: null
}
}
}
function appendWithinUtf8ByteLimit(
current: string[],
currentBytes: number,
delta: string,
maxBytes: number
): { chunks: string[]; retainedBytes: number; truncated: boolean } {
const available = Math.max(0, maxBytes - currentBytes)
const deltaBuffer = Buffer.from(delta, 'utf8')
if (deltaBuffer.byteLength <= available) {
// The caller owns the per-stream array; append in place so each token is
// amortized O(1) instead of copying the complete prefix on every delta.
current.push(delta)
return {
chunks: current,
retainedBytes: currentBytes + deltaBuffer.byteLength,
truncated: false
}
}
const marker = Buffer.from(AGENT_SESSION_STREAMED_TEXT_TRUNCATION_MARKER, 'utf8')
const headBytes = Math.max(0, maxBytes - marker.byteLength)
const combined = Buffer.concat([
...current.map((chunk) => Buffer.from(chunk, 'utf8')),
deltaBuffer
])
let end = Math.min(combined.byteLength, headBytes)
while (end > 0 && (combined[end] & 0b1100_0000) === 0b1000_0000) {
end -= 1
}
const visibleMarker = marker.subarray(0, Math.min(marker.byteLength, maxBytes - end))
const text = combined.subarray(0, end).toString('utf8') + visibleMarker.toString('utf8')
return {
chunks: text ? [text] : [],
retainedBytes: Buffer.byteLength(text, 'utf8'),
truncated: true
}
}
@@ -0,0 +1,108 @@
import {
agentJournalSubmissionKey,
boundJournalKeyComponent
} from '../../../shared/agent-session-journal-item-key'
import type {
AgentJournalRenderItem,
AgentJournalSubmission
} from '../../../shared/agent-session-journal-types'
import { REMOTE_RUNTIME_MAX_OUTBOUND_JSON_BYTES } from '../../../shared/remote-runtime-memory-limits'
export const AGENT_SESSION_HISTORY_MAX_PAGE_BYTES = REMOTE_RUNTIME_MAX_OUTBOUND_JSON_BYTES / 2
const HISTORY_PAGE_ENVELOPE_RESERVE_BYTES = 64 * 1024
export const HISTORY_PAGE_CONTENT_BUDGET_BYTES =
AGENT_SESSION_HISTORY_MAX_PAGE_BYTES - HISTORY_PAGE_ENVELOPE_RESERVE_BYTES
export function historyEntryBytes(
item: AgentJournalRenderItem,
submissionBytes: ReadonlyMap<string, number>
): number {
return Buffer.byteLength(JSON.stringify(item), 'utf8') + (submissionBytes.get(item.itemId) ?? 0)
}
export function submissionBytesByItemId(
submissions: readonly AgentJournalSubmission[]
): Map<string, number> {
return new Map(
submissions.map((submission) => [
agentJournalSubmissionKey(submission.clientMessageId),
Buffer.byteLength(JSON.stringify(submission), 'utf8')
])
)
}
export function oversizedHistoryItem(
item: AgentJournalRenderItem,
byteLength: number
): AgentJournalRenderItem {
return {
...item,
itemId: boundJournalKeyComponent(item.itemId),
body: {
kind: 'status',
text: `[Orca: item truncated — ${byteLength} bytes exceeds the history page budget]`
}
}
}
export function boundHistoryItemsByBytes(
items: AgentJournalRenderItem[],
keep: 'newest' | 'oldest',
submissionBytes: ReadonlyMap<string, number>,
maxBytes: number
): { items: AgentJournalRenderItem[]; dropped: number } {
const groups = groupItemsBySequence(items)
const ordered = keep === 'newest' ? groups.toReversed() : groups
const kept: AgentJournalRenderItem[][] = []
let total = 0
for (const group of ordered) {
const bytes = group.reduce((sum, item) => sum + historyEntryBytes(item, submissionBytes), 0)
if (kept.length === 0 && bytes > maxBytes) {
kept.push(group.map((item) => oversizedHistoryItem(item, bytes)))
break
}
if (total + bytes > maxBytes) {
break
}
kept.push(group)
total += bytes
}
return {
items: (keep === 'newest' ? kept.toReversed() : kept).flat(),
dropped: items.length - kept.reduce((count, group) => count + group.length, 0)
}
}
function groupItemsBySequence(
items: readonly AgentJournalRenderItem[]
): AgentJournalRenderItem[][] {
const groups: AgentJournalRenderItem[][] = []
for (const item of items) {
const current = groups.at(-1)
if (current?.[0]?.sequence === item.sequence) {
current.push(item)
} else {
groups.push([item])
}
}
return groups
}
export function newestWholeSequenceGroups(
items: readonly AgentJournalRenderItem[],
limit: number
): AgentJournalRenderItem[] {
const groups = groupItemsBySequence(items)
const selected: AgentJournalRenderItem[][] = []
let count = 0
for (const group of groups.toReversed()) {
if (selected.length > 0 && count + group.length > limit) {
break
}
selected.push(group)
count += group.length
}
return selected.toReversed().flat()
}
@@ -312,6 +312,49 @@ describe('history page byte ceiling', () => {
})
describe('projectJournalBatch', () => {
it('publishes every nested lifecycle mutation atomically at the outer cursor', async () => {
const turn: AgentJournalItemIdentity = {
provider: 'legacy',
agent: 'codex',
sessionId: 'session-1',
recordId: 'turn-lifecycle:turn-1'
}
await journal.appendItem(turn, { kind: 'status', text: 'working' }, { fence: 1 })
const cursor = journal.cursor()
await journal.appendLifecycleBatch({
settlementId: 'settlement-1',
fence: 1,
mutations: [
{ kind: 'item', identity: item(1), body: body('one') },
{ kind: 'item', identity: item(2), body: body('two') },
{ kind: 'tombstone', identity: turn }
]
})
const page = readAgentSessionHistory(journal, {
sessionId: 'session-1',
direction: 'after',
cursor,
limit: 1
})
if (!page.ok) {
throw new Error(`expected a page, got reset ${page.reset}`)
}
expect(page.page.items.map((entry) => entry.body)).toEqual([body('one'), body('two')])
expect(page.page.removedItemIds).toHaveLength(1)
expect(new Set(page.page.items.map((entry) => entry.sequence)).size).toBe(1)
const tail = readAgentSessionHistory(journal, {
sessionId: 'session-1',
direction: 'tail',
limit: 1
})
if (!tail.ok) {
throw new Error(`expected a page, got reset ${tail.reset}`)
}
expect(tail.page.items.map((entry) => entry.body)).toEqual([body('one'), body('two')])
})
it('reports a hole in the row sequence as journal_gap', async () => {
await appendItems(3)
const since = journal.readSince({ epoch: journal.epoch, sequence: 0 })
@@ -7,17 +7,12 @@
// read would silently skip that revision. Rows carry the revision, which is why
// `after` is the only direction that can answer `cursor_compacted`.
import {
agentJournalSubmissionKey,
boundJournalKeyComponent
} from '../../../shared/agent-session-journal-item-key'
import { agentJournalSubmissionKey } from '../../../shared/agent-session-journal-item-key'
import type {
AgentJournalCursor,
AgentJournalRenderItem,
AgentJournalSnapshot,
AgentJournalSubmission
AgentJournalSnapshot
} from '../../../shared/agent-session-journal-types'
import { REMOTE_RUNTIME_MAX_OUTBOUND_JSON_BYTES } from '../../../shared/remote-runtime-memory-limits'
import {
AGENT_SESSION_HISTORY_DEFAULT_LIMIT,
AGENT_SESSION_HISTORY_MAX_LIMIT,
@@ -28,95 +23,16 @@ import {
} from '../../../shared/agent-session-wire'
import type { AgentSessionJournal } from '../agent-session-journal/journal-store'
import { projectJournalBatch } from './agent-session-journal-batch'
import {
boundHistoryItemsByBytes,
HISTORY_PAGE_CONTENT_BUDGET_BYTES,
historyEntryBytes,
newestWholeSequenceGroups,
oversizedHistoryItem,
submissionBytesByItemId
} from './agent-session-history-page-bounds'
/** Byte budget for one history page. Half the outbound channel cap, so the RPC
* envelope and page framing always fit beside the items: row counts alone
* cannot protect the channel — forty legal 256 KiB messages serialize past the
* 4 MiB outbound cap, and an overflow closes the client's socket on every
* reopen. Pages degrade to fewer rows instead; `hasOlder`/`hasNewer` keep the
* client paging. */
export const AGENT_SESSION_HISTORY_MAX_PAGE_BYTES = REMOTE_RUNTIME_MAX_OUTBOUND_JSON_BYTES / 2
/** Reserved for everything the page carries beyond its items and removal ids:
* cursors, session/epoch ids, and the RPC envelope. Charged up front so the
* content budget bounds the COMPLETE serialized result, not just the rows. */
const HISTORY_PAGE_ENVELOPE_RESERVE_BYTES = 64 * 1024
const HISTORY_PAGE_CONTENT_BUDGET_BYTES =
AGENT_SESSION_HISTORY_MAX_PAGE_BYTES - HISTORY_PAGE_ENVELOPE_RESERVE_BYTES
/** Item bytes plus the submission the page would carry alongside it. */
function historyEntryBytes(
item: AgentJournalRenderItem,
submissionBytes: ReadonlyMap<string, number>
): number {
return Buffer.byteLength(JSON.stringify(item), 'utf8') + (submissionBytes.get(item.itemId) ?? 0)
}
function submissionBytesByItemId(
submissions: readonly AgentJournalSubmission[]
): Map<string, number> {
return new Map(
submissions.map((submission) => [
agentJournalSubmissionKey(submission.clientMessageId),
Buffer.byteLength(JSON.stringify(submission), 'utf8')
])
)
}
/** Visible stand-in for an item whose body alone exceeds the page budget. The
* full body stays in the journal — this bounds what ONE PAGE carries, it never
* rewrites the record. */
function oversizedHistoryItem(
item: AgentJournalRenderItem,
byteLength: number
): AgentJournalRenderItem {
return {
...item,
// A pre-bounding id can exceed the budget by itself; the stand-in must not
// re-inflate the page it exists to bound. Bounding is deterministic, so
// re-reads keep deduplicating on the same key.
itemId: boundJournalKeyComponent(item.itemId),
body: {
kind: 'status',
text: `[Orca: item truncated — ${byteLength} bytes exceeds the history page budget]`
}
}
}
/**
* Keep the edge of the window nearest the requested position within the byte
* budget: `newest` for tail/backward pages, `oldest` for forward catch-up. The
* page stays contiguous, so the dropped remainder is exactly what the next page
* serves. Never empties a non-empty window — a single over-budget item degrades
* to a visible marker so the client always makes progress.
*/
function boundHistoryItemsByBytes(
items: AgentJournalRenderItem[],
keep: 'newest' | 'oldest',
submissionBytes: ReadonlyMap<string, number>,
maxBytes: number
): { items: AgentJournalRenderItem[]; dropped: number } {
const ordered = keep === 'newest' ? items.toReversed() : items
const kept: AgentJournalRenderItem[] = []
let total = 0
for (const item of ordered) {
const bytes = historyEntryBytes(item, submissionBytes)
if (kept.length === 0 && bytes > maxBytes) {
kept.push(oversizedHistoryItem(item, bytes))
break
}
if (total + bytes > maxBytes) {
break
}
kept.push(item)
total += bytes
}
return {
items: keep === 'newest' ? kept.toReversed() : kept,
dropped: items.length - kept.length
}
}
export { AGENT_SESSION_HISTORY_MAX_PAGE_BYTES } from './agent-session-history-page-bounds'
/** Clamped, never rejected: a client asking for more than the host will serve
* should get a smaller page and keep paging, not an error mid-scroll. */
@@ -151,7 +67,7 @@ export function readAgentSessionHistory(
const older = cursor
? snapshot.items.filter((item) => item.sequence < cursor.sequence)
: snapshot.items
const windowed = older.slice(Math.max(0, older.length - limit))
const windowed = newestWholeSequenceGroups(older, limit)
const { items, dropped } = boundHistoryItemsByBytes(
windowed,
'newest',
@@ -185,7 +101,7 @@ function buildHydrationPage(
snapshot: AgentJournalSnapshot,
fence?: number
): AgentSessionHistoryPage {
const items = snapshot.items.slice(-AGENT_SESSION_HISTORY_MAX_LIMIT)
const items = newestWholeSequenceGroups(snapshot.items, AGENT_SESSION_HISTORY_MAX_LIMIT)
const bounded = boundHistoryItemsByBytes(
items,
'newest',
@@ -35,6 +35,16 @@ export function projectJournalBatch(input: {
const touchedItemIds = new Set<string>()
const touchedClientMessageIds = new Set<string>()
for (const row of input.rows) {
if (row.kind === 'lifecycle-batch') {
for (const mutation of row.mutations) {
touchedItemIds.add(
input.canonicalItemId?.(mutation.itemId) ??
aliases.get(mutation.itemId) ??
mutation.itemId
)
}
continue
}
if (row.kind === 'item' || row.kind === 'tombstone') {
touchedItemIds.add(
input.canonicalItemId?.(row.itemId) ?? aliases.get(row.itemId) ?? row.itemId
@@ -44,6 +44,9 @@ export class AgentSessionAcquisitionExitUnprovenError extends Error {
export type AgentSessionAcquisition = {
process: AgentSessionProcessIdentity
link: AgentSessionProviderHandleLink
/** Host-local identity for this exact provider child, distinct even when the durable fence is
* reused by a superseding acquisition. */
acquisitionGeneration?: string
}
/** Acquisition validation failed before the adapter attempted to spawn. */
@@ -65,6 +68,17 @@ export type AgentSessionDispatchOutcome =
/** The call did not settle. Never re-send on the user's behalf. */
| { state: 'unknown'; reason: string }
export type StructuredAgentSessionLifecycleEvent = {
type: 'ended'
sessionId: string
reason: string
cause: 'unexpected-exit' | 'requested-close'
fence: number
acquisitionGeneration: string
/** Translator could not admit terminal rows; host recovery must append its bounded fallback. */
settlementRetryRequired?: boolean
}
export type StructuredAgentSessionAcquireInput = {
identity: AgentSessionJournalIdentity
fence: number
@@ -125,6 +139,8 @@ export type StructuredAgentSessionAdapter = {
/** Gracefully stops the structured owner after its event stream is drained. */
/** Returns true only after the provider child exit is proven. */
closeSession?(sessionId: string): Promise<boolean>
/** Stops a provider after a sink failure; the resulting exit is recovered as unexpected. */
forceCloseSession?(sessionId: string): Promise<boolean>
/** Stops a provider child for teardown without requiring a future-resume cursor. */
disposeSession?(sessionId: string): Promise<boolean>
}
@@ -5,6 +5,7 @@
import type { AgentSessionWireRefusal } from '../../../shared/agent-session-wire'
import type { AgentJournalResetReason } from '../../../shared/agent-session-journal-types'
import type { AgentSessionAttachParams } from './structured-agent-session-attach'
import type { AgentSessionJournal } from '../agent-session-journal/journal-store'
import type {
StructuredAgentSessionHostDeps,
@@ -29,6 +30,8 @@ export type StructuredAgentSessionAttachContext = {
}
tasks: StructuredAgentSessionTaskQueue
reconcileLeases: (sessionId: string) => Promise<AgentSessionWireRefusal | null>
/** Retries a durable provider-exit journal settlement before a new owner is reserved. */
retryPendingSettlement?: (sessionId: string, params: AgentSessionAttachParams) => Promise<boolean>
serialize: <T>(sessionId: string, task: () => Promise<T>) => Promise<T>
now: () => number
}
@@ -47,7 +47,10 @@ export type AttachFlowInput = {
/** Registers the opened journal and fans out to subscribers before the caller
* sees the result, so no client can send against a session the host has not
* finished publishing. */
onAttached: (attached: AttachedJournal) => void
onAttached: (
attached: AttachedJournal,
acquisitionGeneration: string | null
) => Promise<void> | void
/** Handed to the adapter so it can journal what the provider streams. The
* host owns it and binds it to the journal inside `onAttached`. */
eventSink?: StructuredAgentSessionEventSink
@@ -70,6 +73,7 @@ export async function performAttach(
}
let record: AgentSessionRecord
let acquisitionGeneration: string | null = null
let reservedRecord: AgentSessionRecord | null = null
let replayed = false
try {
@@ -101,7 +105,9 @@ export async function performAttach(
}
reservedRecord = record
if (!agentSessionLeaseAdmitsWriter(record.lease)) {
record = await acquireOwner(input, record)
const acquired = await acquireOwner(input, record)
record = acquired.record
acquisitionGeneration = acquired.acquisitionGeneration
}
} catch (error) {
const spawnToken = reservedRecord?.lease.reservedSpawnToken
@@ -171,7 +177,7 @@ export async function performAttach(
journalRoot: input.journalRoot,
adapter: input.adapter
})
input.onAttached(attached)
await input.onAttached(attached, acquisitionGeneration)
await store.recordOperationOutcome({
callerKey: input.callerKey,
operationId: params.envelope.clientOperationId,
@@ -240,7 +246,7 @@ async function settlePostAcquisitionAttachFailure(
async function acquireOwner(
input: AttachFlowInput,
record: AgentSessionRecord
): Promise<AgentSessionRecord> {
): Promise<{ record: AgentSessionRecord; acquisitionGeneration: string | null }> {
const fence = record.lease.runtimeFence
const spawnToken = record.lease.reservedSpawnToken
if (!spawnToken) {
@@ -284,13 +290,17 @@ async function acquireOwner(
} else if (!isDeepStrictEqual(record.lease.ownerProcess, acquired.process)) {
throw new Error('agent_session_ownership_unknown')
}
return await input.store.proveOwner({
const proved = await input.store.proveOwner({
sessionId: record.sessionId,
fence,
link: acquired.link,
now: input.now(),
...(options ? { options } : {})
})
return {
record: proved,
acquisitionGeneration: acquired.acquisitionGeneration ?? null
}
} catch (error) {
if (isAgentSessionPreSpawnError(error)) {
throw error
@@ -22,25 +22,43 @@ import type { StructuredAgentSessionAttachContext } from './structured-agent-ses
export function attachStructuredAgentSession(
context: StructuredAgentSessionAttachContext,
callerKey: string,
params: AgentSessionAttachParams
params: AgentSessionAttachParams,
admitRecoveryTicket?: () => boolean
): Promise<AgentSessionMutationResult<AgentSessionAttachResult>> {
const sessionId = params.envelope.sessionId
const attaching = context.serialize(sessionId, async () => {
if (admitRecoveryTicket && !admitRecoveryTicket()) {
return refuseAgentSessionMutation({
code: 'agent_session_checkpoint_stale',
message: 'The provider-exit recovery ticket is no longer current.'
})
}
const unreconciled = await context.reconcileLeases(sessionId)
if (unreconciled) {
return refuseAgentSessionMutation(unreconciled)
}
await context.runtimeState.resolveRecovery(sessionId)
if (context.retryPendingSettlement) {
const settled = await context.retryPendingSettlement(sessionId, params)
if (!settled) {
return refuseAgentSessionMutation({
code: 'agent_session_ownership_unknown',
message: 'The provider-exit terminal journal settlement is still pending; retry attach.'
})
}
}
const eventSink = context.runtimeState.eventSinkFor(sessionId)
const attached = await performAttach({
store: context.deps.store,
adapter: context.deps.adapter,
journalRoot: context.deps.journalRoot,
eventSink: eventSink.sink,
onAcquiring: () => eventSink.unbind(),
beforeJournalOpen: async () => {
onAcquiring: async () => {
const barrier = await eventSink.drained()
if (!barrier.ok) {
throw barrier.error
}
eventSink.unbind()
await eventSink.drained()
},
authority: {
spawnToken: () => context.deps.mintSpawnToken?.() ?? randomUUID(),
@@ -58,14 +76,25 @@ export function attachStructuredAgentSession(
eventSink.close()
context.runtimeState.discardEventSink(sessionId)
},
onAttached: (attached) => {
onAttached: async (attached, acquisitionGeneration) => {
const fence = context.deps.store.getRecord(sessionId)?.lease.runtimeFence ?? 0
const previousFence = context.sessions.get(sessionId)?.fence
const previous = context.sessions.get(sessionId)
const previousFence = previous?.fence
eventSink.bind({
journal: attached.journal,
fence,
publish: () => context.subscribers.publish(sessionId, attached.journal)
})
const barrier = await eventSink.drained()
if (!barrier.ok) {
throw barrier.error
}
context.sessions.set(sessionId, {
journal: attached.journal,
params,
fence,
hasProviderChild: true
hasProviderChild: true,
acquisitionGeneration: acquisitionGeneration ?? previous?.acquisitionGeneration ?? null
})
if (attached.recovery) {
context.subscribers.reset(sessionId, attached.journal, attached.recovery.reset, fence)
@@ -74,11 +103,6 @@ export function attachStructuredAgentSession(
} else {
context.subscribers.publish(sessionId, attached.journal)
}
eventSink.bind({
journal: attached.journal,
fence,
publish: () => context.subscribers.publish(sessionId, attached.journal)
})
}
})
// Why: a failed attach that left no session behind must not strand a bound sink; the runtime

Some files were not shown because too many files have changed in this diff Show More