fix(native-chat): terminal restore no longer waits for chats an older Orca saved (#43527)

* fix(native-chat): terminal restore no longer waits for chats an older Orca saved

The open that terminal restore and SSH reconnect wait for now only reads and writes per-chat records. A converted chat an older Orca continued gets its copy record at once; the copy's history converts from the original like any unconverted chat, paged on a reader. A converted chat with no saved revision is compared in the background, one chat per idle step. A chat that fails is recorded with its error and absorbed, so it no longer reruns the pass at every launch; storage failures pause the pass for the next launch instead.

Tracked in #42783.

* fix(native-chat): keep copies an older build readied early, and pending work across pauses

A copy's mapping now clears when its history converts here, so a copy another build readied first is found and kept again from the original. Copy-due markers clear per chat, so a paused pass loses none. A pending copy is found by its original's id, however many times the original moved. Forced copies use a set lookup.

* fix(native-chat): bound and absorb the remaining recovery work

A copy whose history fails to convert is recorded and left to an open instead of every idle pass. The background compare restores at most 20,000 original rows per chat; a longer chat keeps its turns as a copy. Corruption fails only its own chat. Compare markers are written with the pass, and a later absorbed revision clears an earlier failure.

* fix(native-chat): never mark a copy failed for a quit or a lost reader

Only a failure of the original's own content records a copy as failed, and never while maintenance is stopping. Corruption after the original's revision read pauses the pass instead of absorbing chats, since it may be this file's.

* fix(native-chat): compare a whole chat per idle step and absorb one that crashes its reader

Per the council's ruling, the row budget is removed: it forced duplicate copies of long chats whose conversion then did the same full restore. A compare that takes its reader down is recorded and absorbed instead of retried every launch; stopping and contention still leave it for the next launch.

* test: give the forgotten-chats maintenance fake the legacy compare step

* fix(agent-session-journal): forgotten chats absorb their pending compare; this file's damage leaves a compare pending

A chat forgotten while its older-Orca compare or copy waited takes that revision, so the next open never brings it back as a copy. Compare reads this file before the original, and only the original's failures settle as the chat's answer; this file's damage, contention and stopping stay pending. A reader exit gets one retry before it is absorbed, and a compare error no longer skips that launch's idle conversion.

* fix(agent-session-journal): a damaged original settles its compare instead of staying pending

* fix(agent-session-journal): a message an older Orca queued makes its compare keep a copy

* chore: count the compare's card read and the stranded-copy point read in the sqlite baseline

* test(agent-session-journal): the copy keeps the older Orca's queued message beside the chat's own

* perf(agent-session-journal): a copy due on an unchanged original no longer reads or replays every chat

* chore: prune the blocking baselines main's worker-only rules cleared; count the reconcile's split-out files

* perf(agent-session-journal): keep the older-Orca reconcile on workers and off whole-profile reads

The chat-store path moves beside the journal's own, so the main thread no longer reaches the
reconcile chain. A pass over copies due reads only those chats' claims and looks tabs up one at a
time, and a compare with no original reads nothing from this file.

* perf(agent-session-journal): recording the tab index keeps this file's tabs as a set

* perf(agent-session-journal): a missing original ends the compare pass instead of asking about every chat

* docs(agent-session-journal): say what a pass over copies due still scans

* fix(agent-session-journal): damage to one chat here leaves only its compare pending

* docs(agent-session-journal): a pass over copies due scans tabs once per due chat

* fix(agent-session-journal): a copy too large for one reader reply is recorded, not retried every launch

* chore: drop the tab lookup's cast; prune the sync-fs rows the worker-only move cleared

* fix(agent-session-journal): an original that won't open ends the compare pass; a stranded rollback stops the reconcile

Drops the statement-count test: the repo doesn't test work counts.

* fix(agent-session-journal): a forgotten copy drops its mapping and failure keys

* test(e2e): take the chat-store path from journal-host-database

* fix(agent-session-journal): name the older-Orca compare in reader crash diagnostics
This commit is contained in:
Brennan Benson
2026-11-18 11:38:39 -08:00
committed by GitHub
parent b779ed213d
commit 9f8ea51b79
39 changed files with 1229 additions and 242 deletions
-17
View File
@@ -826,22 +826,13 @@
4 src/main/native-chat/agent-session-attachments/agent-session-attachment-claims.ts
1 src/main/native-chat/agent-session-journal/context-epochs/journal-context-cursor.ts
12 src/main/native-chat/agent-session-journal/database/journal-database.ts
13 src/main/native-chat/agent-session-journal/database/journal-domain-database.ts # store-open tail fold (one-shot startup); same SQL already in main-blocking-sqlite-baseline
5 src/main/native-chat/agent-session-journal/database/journal-domain-migrations.ts # tail table create at store open (one-shot startup); worker-only: +2 create the retired-epoch pending table in a transaction on the journal SQL worker
8 src/main/native-chat/agent-session-journal/database/journal-domain-profile-import.ts
3 src/main/native-chat/agent-session-journal/database/journal-host-database.ts
40 src/main/native-chat/agent-session-journal/database/journal-legacy-reconcile.ts
1 src/main/native-chat/agent-session-journal/database/journal-legacy-source-receipt.ts
4 src/main/native-chat/agent-session-journal/database/journal-legacy-source.ts
1 src/main/native-chat/agent-session-journal/durable-rows/journal-row-table.ts
2 src/main/native-chat/agent-session-journal/history-import/adopted-transcript-source.ts
1 src/main/native-chat/agent-session-journal/history-import/journal-hermes-import.ts
5 src/main/native-chat/agent-session-journal/history-import/journal-legacy-import.ts
2 src/main/native-chat/agent-session-journal/input-order/journal-input-conversion.ts
1 src/main/native-chat/agent-session-journal/journal-domain-import-discard.ts
10 src/main/native-chat/agent-session-journal/journal-domain-import-rows.ts
1 src/main/native-chat/agent-session-journal/journal-domain-import-submissions.ts
7 src/main/native-chat/agent-session-journal/journal-domain-import.ts
1 src/main/native-chat/agent-session-journal/journal-domain-input-eligibility.ts
2 src/main/native-chat/agent-session-journal/journal-domain-item-display.ts
2 src/main/native-chat/agent-session-journal/journal-domain-maintenance.ts # worker-only: the retired prune settles its pending entry on the journal SQL worker
@@ -849,7 +840,6 @@
2 src/main/native-chat/agent-session-journal/journal-domain-queue-pauses.ts
6 src/main/native-chat/agent-session-journal/journal-domain-queue-reads.ts
1 src/main/native-chat/agent-session-journal/journal-domain-schema.ts
2 src/main/native-chat/agent-session-journal/journal-domain-stop-notes.ts
2 src/main/native-chat/agent-session-journal/journal-domain-submission-consume.ts
1 src/main/native-chat/agent-session-journal/journal-forgotten-chats.ts
11 src/main/native-chat/agent-session-journal/journal-sql-queue-read.ts
@@ -860,12 +850,7 @@
5 src/main/native-chat/agent-session-journal/journal-stored-output-rows.ts # streamed-text tail (#43716): same SQL already in main-blocking-sqlite-baseline; journal SQL path, not a timer or handler
10 src/main/native-chat/agent-session-journal/journal-text-tail.ts # streamed-text tail (#43716): same SQL already in main-blocking-sqlite-baseline; journal SQL path, not a timer or handler
2 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-format.ts
2 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-table.ts
1 src/main/native-chat/agent-session-journal/queued-messages/journal-queue-import.ts
1 src/main/native-chat/agent-session-journal/replay/journal-historical-inputs.ts
13 src/main/native-chat/agent-session-journal/replay/journal-open.ts
4 src/main/native-chat/agent-session-journal/replay/journal-read-row.ts
2 src/main/native-chat/agent-session-journal/replay/journal-retained-inputs.ts
3 src/main/native-chat/agent-session-journal/replay/journal-stored-cursor.ts
3 src/main/native-chat/agent-session-journal/writing/command-receipt-table.ts
1 src/main/native-chat/agent-session-journal/writing/command-receipt-transaction.ts
@@ -1212,8 +1197,6 @@
1 src/main/runtime/agent-session/model-catalog/structured-agent-model-catalog-discovery.ts
1 src/main/runtime/agent-session/ownership/agent-session-claim-identity.ts
4 src/main/runtime/agent-session/ownership/agent-session-process-identity-probe.ts
5 src/main/runtime/agent-session/records/agent-session-background-meta.ts
5 src/main/runtime/agent-session/records/agent-session-record-rows.ts
1 src/main/runtime/agent-session/records/agent-session-record-store-slot.ts
8 src/main/runtime/agent-session/records/agent-session-runtime-end-record.ts
2 src/main/runtime/agent-session/records/agent-session-store-transactions.ts
-16
View File
@@ -29,42 +29,28 @@
19 src/main/native-chat/agent-session-attachments/agent-session-attachment-claims.ts
5 src/main/native-chat/agent-session-journal/database/journal-checkpoint-owner.ts
22 src/main/native-chat/agent-session-journal/database/journal-database.ts
14 src/main/native-chat/agent-session-journal/database/journal-domain-database.ts
8 src/main/native-chat/agent-session-journal/database/journal-domain-migrations.ts # worker-only: +1 is the reply outbox table migration, run on the journal SQL worker
22 src/main/native-chat/agent-session-journal/database/journal-domain-profile-import.ts
1 src/main/native-chat/agent-session-journal/database/journal-host-database.ts
32 src/main/native-chat/agent-session-journal/database/journal-legacy-reconcile.ts
18 src/main/native-chat/agent-session-journal/database/journal-legacy-source.ts
5 src/main/native-chat/agent-session-journal/database/journal-retire-pending.ts # worker-only: retired-epoch pending table, triggers and settle, run on the journal SQL worker
5 src/main/native-chat/agent-session-journal/database/journal-text-tail-schema.ts
12 src/main/native-chat/agent-session-journal/durable-rows/journal-row-table.ts
18 src/main/native-chat/agent-session-journal/history-import/journal-pi-active-branch.ts
6 src/main/native-chat/agent-session-journal/journal-domain-import-discard.ts
24 src/main/native-chat/agent-session-journal/journal-domain-import-rows.ts
4 src/main/native-chat/agent-session-journal/journal-domain-import-submissions.ts
2 src/main/native-chat/agent-session-journal/journal-domain-import.ts
4 src/main/native-chat/agent-session-journal/journal-domain-item-display.ts
4 src/main/native-chat/agent-session-journal/journal-domain-item-names.ts
12 src/main/native-chat/agent-session-journal/journal-domain-maintenance.ts
10 src/main/native-chat/agent-session-journal/journal-domain-queue-failures.ts
6 src/main/native-chat/agent-session-journal/journal-domain-queue-pauses.ts
8 src/main/native-chat/agent-session-journal/journal-domain-queue-reads.ts
4 src/main/native-chat/agent-session-journal/journal-domain-queue-revision.ts
10 src/main/native-chat/agent-session-journal/journal-domain-queue-rows.ts
6 src/main/native-chat/agent-session-journal/journal-domain-stop-notes.ts
2 src/main/native-chat/agent-session-journal/journal-domain-submission-consume.ts
4 src/main/native-chat/agent-session-journal/journal-domain-submission-records.ts
20 src/main/native-chat/agent-session-journal/journal-forgotten-chats.ts
4 src/main/native-chat/agent-session-journal/journal-sql-accepted-inputs.ts
4 src/main/native-chat/agent-session-journal/journal-stored-output-rows.ts
24 src/main/native-chat/agent-session-journal/journal-text-tail.ts
15 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-format.ts
6 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-table.ts
2 src/main/native-chat/agent-session-journal/queued-messages/journal-queue-import.ts
5 src/main/native-chat/agent-session-journal/queued-messages/queued-message-schema.ts
10 src/main/native-chat/agent-session-journal/replay/journal-historical-inputs.ts
4 src/main/native-chat/agent-session-journal/replay/journal-read-row.ts
8 src/main/native-chat/agent-session-journal/replay/journal-retained-inputs.ts
1 src/main/native-chat/agent-session-journal/writing/command-receipt-schema.ts
8 src/main/native-chat/agent-session-journal/writing/command-receipt-table.ts
2 src/main/native-chat/agent-session-journal/writing/journal-goal-attachment-inverse.ts
@@ -95,9 +81,7 @@
4 src/main/persistence/profile-state/legacy-json/profile-state-json-acceptance.ts
2 src/main/persistence/profile-state/recovery/profile-state-section-repair.ts
4 src/main/rate-limits/providers/opencode-go/opencode-go-api-key-source.ts
4 src/main/runtime/agent-session/records/agent-session-background-meta.ts
2 src/main/runtime/agent-session/records/agent-session-record-lineage.ts
38 src/main/runtime/agent-session/records/agent-session-record-rows.ts
23 src/main/runtime/orchestration/db/dispatch-context/presentation-completion-source.ts
6 src/main/runtime/orchestration/db/dispatch-row-writer.ts
1 src/main/runtime/orchestration/db/federation/federated-stub-home-run-backfill.ts
-2
View File
@@ -316,9 +316,7 @@
1 src/main/muse/hook-config-json.ts
4 src/main/muse/hook-service.ts
2 src/main/native-chat/agent-session-attachments/agent-session-attachment-claims.ts
8 src/main/native-chat/agent-session-journal/database/journal-domain-database.ts
1 src/main/native-chat/agent-session-journal/database/journal-host-database.ts
3 src/main/native-chat/agent-session-journal/database/journal-legacy-source-receipt.ts
1 src/main/native-chat/transcript-discovery/host-readable-transcript-path.ts
2 src/main/native-chat/transcript-filesystem/process/wsl-transcript-fs-process-spawn.ts
2 src/main/node-runtime/managed-runtime-release-install.ts
@@ -21,10 +21,9 @@ import {
createAgentSessionRecordTablesSql
} from './journal-database-schema'
import { openJournalDatabase } from './journal-database'
import { journalDatabasePath } from './journal-host-database'
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
import {
initializeJournalDomainDatabase,
journalDomainDatabasePath,
openJournalDomainDatabase
} from './journal-domain-database'
import { readMailWithdrawal } from '../writing/mail-withdrawal-receipts'
@@ -9,7 +9,7 @@ import {
journalPragmaNumber,
type OpenJournalDatabase
} from './journal-database'
import { journalDatabasePath } from './journal-host-database'
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
import { tryFoldJournalTextTails } from '../journal-text-tail'
import {
JOURNAL_DOMAIN_SCHEMA_VERSION,
@@ -26,10 +26,6 @@ import {
JOURNAL_LEGACY_SOURCE_RECEIPT
} from './journal-legacy-source-receipt'
export function journalDomainDatabasePath(stateDirectory: string): string {
return join(stateDirectory, 'agent-session-domain.db')
}
/** Run once on the writer worker, before opening its steady-state writable connection. */
export async function initializeJournalDomainDatabase(stateDirectory: string): Promise<string> {
const destination = journalDomainDatabasePath(stateDirectory)
@@ -19,6 +19,10 @@ export function journalDatabasePath(stateDirectory: string): string {
return join(stateDirectory, JOURNAL_DATABASE_FILE)
}
export function journalDomainDatabasePath(stateDirectory: string): string {
return join(stateDirectory, 'agent-session-domain.db')
}
/** What journal collaborators need from a connection owner; the SQL writer supplies its own. */
export type JournalTransactionalDatabase = Pick<
JournalHostDatabase,
@@ -0,0 +1,80 @@
import type Database from '../../../sqlite/sync-database'
import type { SqliteRow } from '../../../sqlite/sqlite-statement'
import { AGENT_SESSION_TABS_RECORDED } from '../../../runtime/agent-session/records/agent-session-record-rows'
import { readMeta } from './journal-legacy-reconcile-marks'
export type Claim = SqliteRow
/** A Set over a walk's rows, or point reads over this file's own. */
type Lookup = {
has(value: string): boolean
add(value: string): unknown
delete(value: string): unknown
}
/** The original's tabs and claims, and this file's tabs, read once per walk. */
export type Adoption = {
savedTabs: Map<string, string>
claims: Map<string, Claim[]>
tabbed: Lookup
takenTabs: Lookup
nextPosition: number
tabsRecorded: boolean
/** Tabs this pass added, so a chat that fails can take back its own. */
added: [sessionId: string, tab: string][]
}
/**
* A pass over chats due a copy reads only those chats' claims and looks this file's tabs up one at a
* time; a copy never takes the original's saved tab. Neither table is indexed by chat, so that pass
* still scans the claims once and this file's tabs once per due chat, keeping only their rows.
*/
export function readAdoption(
source: Database.Database,
db: Database.Database,
only: readonly string[] | null
): Adoption {
const claims = new Map<string, Claim[]>()
for (const claim of source
.prepare(`SELECT upload_id,session_id,claimed_at FROM agent_session_attachment_claims
WHERE ?1 IS NULL OR session_id IN (SELECT value FROM json_each(?1))`)
.all(only && JSON.stringify(only))) {
const id = String(claim.session_id)
const list = claims.get(id) ?? []
list.push(claim)
claims.set(id, list)
}
const tail = db.prepare('SELECT COALESCE(MAX(position),-1) AS tail FROM agent_session_tabs').get()
const tabsRecorded = readMeta(db, AGENT_SESSION_TABS_RECORDED) !== undefined
// Recording the tab index adds one tab per record, so it takes this file's tabs as a set.
const pointed = only !== null && tabsRecorded
const tabs = pointed ? [] : db.prepare('SELECT tab_id,session_id FROM agent_session_tabs').all()
return {
savedTabs: new Map(
only
? []
: source
.prepare('SELECT tab_id,session_id FROM agent_session_tabs')
.all()
.map((row) => [String(row.session_id), String(row.tab_id)])
),
claims,
tabbed: pointed
? tabLookup(db, 'session_id')
: new Set(tabs.map((row) => String(row.session_id))),
takenTabs: pointed ? tabLookup(db, 'tab_id') : new Set(tabs.map((row) => String(row.tab_id))),
nextPosition: Number(tail?.tail ?? -1) + 1,
tabsRecorded,
added: []
}
}
/** This file's own rows answer, so a tab added or rolled back needs no bookkeeping here. */
function tabLookup(db: Database.Database, column: 'session_id' | 'tab_id'): Lookup {
const query = db.prepare(`SELECT 1 FROM agent_session_tabs WHERE ${column}=?`)
return {
has: (value) => query.get(value) !== undefined,
add: () => undefined,
delete: () => undefined
}
}
@@ -0,0 +1,170 @@
import type Database from '../../../sqlite/sync-database'
import { isSqliteCorruption, isSqliteStorageFailure } from '../../../sqlite/sqlite-read-failure'
import { isUnsettledQueuedMessage } from '../queued-messages/queued-message-table'
import { prepareJournalDomainImport, type JournalDomainImportPage } from '../journal-domain-import'
import {
journalLegacyChatRevision,
journalLegacyFailureKey,
recordJournalLegacyChatRevision,
settleJournalLegacyCopy
} from './journal-legacy-source'
import {
deleteMeta,
JOURNAL_LEGACY_COMPARE,
JOURNAL_LEGACY_COPY_DUE,
prefixEnd,
readMeta,
recordJournalLegacyFailure,
writeMeta
} from './journal-legacy-reconcile-marks'
export type JournalLegacyCompare = { sessionId: string; revision: string }
/** Converted chats awaiting a background compare, in pages after `after`. */
export function pendingJournalLegacyCompares(
db: Database.Database,
after: string | null
): JournalLegacyCompare[] {
return db
.prepare(`SELECT key,value FROM agent_session_store_meta WHERE key>? AND key<?
ORDER BY key LIMIT 16`)
.all(`${JOURNAL_LEGACY_COMPARE}${after ?? ''}`, prefixEnd(JOURNAL_LEGACY_COMPARE))
.map((row) => ({
sessionId: String(row.key).slice(JOURNAL_LEGACY_COMPARE.length),
revision: String(row.value)
}))
}
/**
* Whether this file already holds every item of the original chat. Runs on a reader, one chat per
* idle step, and stops at the first item this chat never held. Its unit is the chat's restore,
* the same one converting that chat takes; an interrupted compare restarts that chat. Only the
* original's failures are the chat's answer; this file's own failures throw and stay pending.
*/
export function compareJournalLegacyChat(
openSource: () => Database.Database | undefined,
db: Database.Database,
sessionId: string
): { revision: string; held: boolean } | { error: string } | null {
let source: Database.Database | undefined
try {
source = openSource()
} catch (error) {
// An original that won't open is no chat's answer; it ends the pass until it opens again.
if (isSqliteCorruption(error)) {
return null
}
return originalFailure(error)
}
// With no original there is nothing to compare, so this file isn't read.
if (!source) {
return null
}
const held = { items: heldItems(db, sessionId), cards: heldCards(db, sessionId) }
try {
const imported = prepareJournalDomainImport(source, sessionId)
if (!imported) {
return { revision: journalLegacyChatRevision(source, sessionId) ?? 'null', held: true }
}
return {
revision: imported.sourceRevision ?? 'null',
held: heldEveryItem(held, imported.pages)
}
} catch (error) {
return originalFailure(error)
}
}
function originalFailure(error: unknown): { error: string } {
if (isSqliteStorageFailure(error)) {
throw error
}
return { error: error instanceof Error ? error.message : String(error) }
}
/** Ignored when a later open marked a newer revision; a failed compare is absorbed like a chat. */
export function settleJournalLegacyCompare(
db: Database.Database,
compare: JournalLegacyCompare & ({ held: boolean } | { error: string })
): void {
const key = `${JOURNAL_LEGACY_COMPARE}${compare.sessionId}`
if (readMeta(db, key) !== compare.revision) {
return
}
deleteMeta(db, key)
if ('error' in compare) {
recordJournalLegacyFailure(db, {
sessionId: compare.sessionId,
revision: compare.revision,
error: new Error(compare.error)
})
} else if (compare.held) {
recordJournalLegacyChatRevision(db, compare.sessionId, compare.revision)
} else {
writeMeta(db, `${JOURNAL_LEGACY_COPY_DUE}${compare.sessionId}`, compare.revision)
}
}
/**
* Removing an item here leaves its tombstone, so every item this chat converted stays held, at
* its converted revision or a later one.
*/
function heldItems(db: Database.Database, id: string): Map<string, number> {
return new Map(
db
.prepare(`SELECT item_id AS id, MAX(revision) AS revision FROM (
SELECT item_id, revision FROM journal_items WHERE session_id=?1
UNION ALL SELECT item_id, ?2 FROM journal_item_tombstones WHERE session_id=?1)
GROUP BY item_id`)
.all(id, Number.MAX_SAFE_INTEGER)
.map((row) => [String(row.id), Number(row.revision)])
)
}
/** Every card this chat ever queued, whatever became of it. */
function heldCards(db: Database.Database, id: string): Set<string> {
return new Set(
db
.prepare('SELECT DISTINCT message_id AS id FROM journal_queue_cards WHERE session_id=?')
.all(id)
.map((row) => String(row.id))
)
}
/** A message the older Orca queued and never sent has no item, only its card. */
function heldEveryItem(
held: { items: Map<string, number>; cards: Set<string> },
pages: Iterable<JournalDomainImportPage>
): boolean {
for (const page of pages) {
if (
page.kind === 'items' &&
!page.rows.every(({ item }) => (held.items.get(item.itemId) ?? -1) >= item.revision)
) {
return false
}
if (
page.kind === 'cards' &&
page.rows.some((card) => isUnsettledQueuedMessage(card) && !held.cards.has(card.messageId))
) {
return false
}
}
return true
}
/**
* A forgotten chat takes the revision its compare or copy waited on, so an older Orca's unchanged
* copy never brings it back; a forgotten copy drops its mapping, so no key outlives it.
*/
export function absorbForgottenJournalLegacyChat(db: Database.Database, sessionId: string): void {
for (const prefix of [JOURNAL_LEGACY_COMPARE, JOURNAL_LEGACY_COPY_DUE]) {
const revision = readMeta(db, `${prefix}${sessionId}`)
if (revision !== undefined) {
deleteMeta(db, `${prefix}${sessionId}`)
recordJournalLegacyChatRevision(db, sessionId, revision)
}
}
settleJournalLegacyCopy(db, sessionId)
deleteMeta(db, journalLegacyFailureKey(sessionId))
}
@@ -0,0 +1,99 @@
import type Database from '../../../sqlite/sync-database'
import { runJournalTransaction } from './journal-database'
import {
journalLegacyCopySourceKey,
journalLegacyFailureKey,
journalLegacyPendingCopyKey,
recordJournalLegacyChatRevision,
settleJournalLegacyCopy
} from './journal-legacy-source'
import {
continuedCopyId,
deleteMeta,
holdsPendingCopy,
JOURNAL_LEGACY_COPY_DUE,
revisionHoldsHistory,
writeMeta
} from './journal-legacy-reconcile-marks'
/**
* A background compare found turns this file lacks, or a copy was `stranded`: readied by another
* build without the original's history.
*/
export type JournalLegacyForcedCopy = { stranded?: string }
/** Adds the original's record as this id, with its tab, catalog row and claims. */
type Adopt = (as: string) => void
/**
* The original's version of a converted chat it moved past becomes a second chat beside it. Only
* its record is made here; its history converts from the original chat when opened or idle. With
* no revision to compare (files copied before revisions were kept), a background compare decides.
*/
export function keepContinuedChat(
db: Database.Database,
id: string,
revision: string,
shown: string | undefined,
adopt: Adopt,
pass: { baselines: [string, string][]; compares: [string, string][] }
): void {
if (shown === undefined && revisionHoldsHistory(revision)) {
pass.compares.push([id, revision])
return
}
if (holdsPendingCopy(db, id)) {
pass.baselines.push([id, revision])
return
}
runJournalTransaction(db, () => keepCopy(db, id, revision, adopt))
}
/** Background conversion skips a copy whose history failed to convert; opening it still retries. */
export function recordJournalLegacyCopyFailure(
db: Database.Database,
copyId: string,
error: string
): void {
if (
db
.prepare('SELECT 1 FROM agent_session_store_meta WHERE key=?')
.get(journalLegacyCopySourceKey(copyId))
) {
writeMeta(db, journalLegacyFailureKey(copyId), JSON.stringify({ revision: null, error }))
}
}
/** In the caller's transaction. */
export function keepForcedCopy(
db: Database.Database,
id: string,
revision: string,
forced: JournalLegacyForcedCopy,
adopt: Adopt
): void {
deleteMeta(db, `${JOURNAL_LEGACY_COPY_DUE}${id}`)
if (forced.stranded === undefined) {
keepCopy(db, id, revision, adopt)
return
}
settleJournalLegacyCopy(db, forced.stranded)
keepCopy(db, id, revision, adopt, continuedCopyId(id, forced.stranded))
}
function keepCopy(
db: Database.Database,
id: string,
revision: string,
adopt: Adopt,
copy = continuedCopyId(id, revision)
): void {
if (!revisionHoldsHistory(revision)) {
recordJournalLegacyChatRevision(db, id, revision)
return
}
adopt(copy)
writeMeta(db, journalLegacyCopySourceKey(copy), id)
writeMeta(db, journalLegacyPendingCopyKey(id), copy)
recordJournalLegacyChatRevision(db, id, revision)
}
@@ -0,0 +1,112 @@
import { createHash } from 'node:crypto'
import type Database from '../../../sqlite/sync-database'
import {
journalLegacyCopySourceKey,
journalLegacyFailureKey,
journalLegacyPendingCopyKey,
recordJournalLegacyChatRevision
} from './journal-legacy-source'
/** A converted chat with no revision to compare whose original moved; compared in the background. */
export const JOURNAL_LEGACY_COMPARE = 'journal.legacy-compare:'
/** Compared and found to hold turns this file lacks; the next open keeps them as a copy. */
export const JOURNAL_LEGACY_COPY_DUE = 'journal.legacy-copy-due:'
export type JournalLegacyFailure = {
sessionId: string
revision: string | undefined
error: unknown
}
/** Absorbed with its error: retried only when the older Orca writes this chat again. */
export function recordJournalLegacyFailure(
db: Database.Database,
failure: JournalLegacyFailure
): void {
console.warn('[agent-session-journal] keeping an older Orca chat failed', failure.error)
if (failure.revision !== undefined) {
recordJournalLegacyChatRevision(db, failure.sessionId, failure.revision)
}
writeMeta(
db,
journalLegacyFailureKey(failure.sessionId),
JSON.stringify({ revision: failure.revision ?? null, error: failureMessage(failure.error) })
)
deleteMeta(db, `${JOURNAL_LEGACY_COPY_DUE}${failure.sessionId}`)
}
export function failureMessage(error: unknown): string {
return (error instanceof Error ? error.message : String(error)).slice(0, 500)
}
export function readMeta(db: Database.Database, key: string): string | undefined {
const row = db.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)
return row === undefined ? undefined : String(row.value)
}
export function metaKeys(db: Database.Database, prefix: string): string[] {
return db
.prepare('SELECT key FROM agent_session_store_meta WHERE key>? AND key<? ORDER BY key')
.all(prefix, prefixEnd(prefix))
.map((row) => String(row.key).slice(prefix.length))
}
/** Keys here are ASCII and every prefix ends in ':', so ';' bounds its range. */
export function prefixEnd(prefix: string): string {
return `${prefix.slice(0, -1)};`
}
export function deleteMeta(db: Database.Database, key: string): void {
db.prepare('DELETE FROM agent_session_store_meta WHERE key=?').run(key)
}
export function writeMeta(db: Database.Database, key: string, value: string): void {
db.prepare('INSERT OR REPLACE INTO agent_session_store_meta (key, value) VALUES (?, ?)').run(
key,
value
)
}
/** Derived from the revision, so a pass interrupted after making it finds the same chat. */
export function continuedCopyId(id: string, revision: string): string {
return createHash('sha256').update(`${id}\0${revision}`).digest('hex').slice(0, 32)
}
/** A copy of this original chat still waiting to convert reads the current original. */
export function holdsPendingCopy(db: Database.Database, id: string): boolean {
const copy = readMeta(db, journalLegacyPendingCopyKey(id))
return (
copy !== undefined &&
db
.prepare(`SELECT 1 FROM agent_session_records WHERE session_id=?1 AND NOT EXISTS (
SELECT 1 FROM journal_chat_state WHERE session_id=?1 AND epoch IS NOT NULL)`)
.get(copy) !== undefined
)
}
const COPY_SOURCE = journalLegacyCopySourceKey('')
/**
* Copies another build readied before this one converted them: an older build that can't read
* the mapping founds them empty. Conversion here clears the mapping, so only those remain.
*/
export function strandedCopies(db: Database.Database): [original: string, copy: string][] {
const mapped = db
.prepare('SELECT key, value FROM agent_session_store_meta WHERE key>? AND key<?')
.all(COPY_SOURCE, prefixEnd(COPY_SOURCE))
if (mapped.length === 0) {
return []
}
const founded = db.prepare(
'SELECT 1 FROM journal_chat_state WHERE session_id=? AND epoch IS NOT NULL'
)
return mapped
.map((row): [string, string] => [String(row.value), String(row.key).slice(COPY_SOURCE.length)])
.filter(([, copy]) => founded.get(copy) !== undefined)
}
/** An original chat whose current epoch saved no row has nothing to keep. */
export function revisionHoldsHistory(revision: string): boolean {
const parsed: unknown = JSON.parse(revision)
return Array.isArray(parsed) && parsed[1] !== 0
}
@@ -7,10 +7,9 @@ import type Database from '../../../sqlite/sync-database'
import { openJournalDatabase, runJournalTransaction } from './journal-database'
import {
initializeJournalDomainDatabase,
journalDomainDatabasePath,
openJournalDomainDatabase
} from './journal-domain-database'
import { journalDatabasePath } from './journal-host-database'
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
import { reconcileJournalLegacySource } from './journal-legacy-reconcile'
import { insertJournalRow, publishJournalSessionEpoch } from '../durable-rows/journal-row-table'
import { journalSqlRetryRows } from '../journal-sql-retry-rows.test-fixture'
@@ -19,6 +18,8 @@ import { JournalSqlConversions } from '../journal-sql-conversions'
import { journalSqlReaderPath } from '../journal-sql-worker-bundle.test-fixture'
import { applyJournalSqlMaintenance } from '../journal-sql-maintenance'
import type { JournalRow } from '../durable-rows/journal-row-schema'
import { AGENT_SESSION_JOURNAL_QUEUE_SCHEMA_VERSION } from '../../../../shared/agent-session/journal/agent-session-journal-types'
import { readJournalSqlDatabase } from '../journal-sql-database-read'
const cleanup: (() => Promise<void> | void)[] = []
afterEach(async () => {
@@ -73,6 +74,16 @@ async function upgrade(root: string) {
return {
db,
convert: (sessionId: string) => conversions.ensure(sessionId),
/** One background pass over converted chats awaiting a compare, as maintenance runs it. */
compare: async () => {
const page = await pool.database({ kind: 'legacy-compares', after: null })
for (const compare of page.kind === 'legacy-compares' ? page.compares : []) {
const result = await pool.compareLegacyChat(compare.sessionId)
runJournalTransaction(db, () =>
applyJournalSqlMaintenance(db, { kind: 'legacy-compare-settle', ...compare, ...result! })
)
}
},
close: async () => {
await conversions.close()
await pool.close()
@@ -126,17 +137,6 @@ it('keeps every chat and turn an older Orca saved after a downgrade, without cha
.update(readFileSync(journalDatabasePath(root)))
.digest('hex')
// Upgrade again; one chat fails to join the first time and joins on the next launch.
const blocked = openJournalDomainDatabase(journalDomainDatabasePath(root)).db
blocked.exec(`CREATE TRIGGER blocked BEFORE INSERT ON agent_session_records
WHEN NEW.session_id='started-older' BEGIN SELECT RAISE(ABORT,'disk full'); END`)
blocked.close()
const failing = openJournalDomainDatabase(await initializeJournalDomainDatabase(root)).db
expect(
failing.prepare("SELECT 1 FROM agent_session_records WHERE session_id='started-older'").get()
).toBeUndefined()
failing.exec('DROP TRIGGER blocked')
failing.close()
const second = await upgrade(root)
cleanup.push(second.close)
const { db } = second
@@ -146,6 +146,9 @@ it('keeps every chat and turn an older Orca saved after a downgrade, without cha
WHERE session_id NOT IN ('converted','unopened','started-older','closed')`)
.all()
.map((row) => String(row.session_id))
// Terminal restore waits for the open, so the copy's history converts later, from the original.
expect(items(db, copy)).toEqual([])
expect(await second.convert(copy)).toBe('epoch')
expect(items(db, copy)).toEqual([...converted, 'older-turn'])
expect(
JSON.parse(
@@ -215,6 +218,18 @@ it('fills in revisions for chats converted before they were kept, then still kee
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
})
original.pragma('wal_checkpoint(TRUNCATE)')
await (await upgrade(root)).close()
// Written twice more before the copy converted: that copy reads the current original, so no second.
for (const [index, itemId] of ['older-turn-2', 'older-turn-3'].entries()) {
runJournalTransaction(original, () => {
insertJournalRow(original, 'converted', turn(itemId, tip + 2 + index))
})
original.pragma('wal_checkpoint(TRUNCATE)')
if (index === 0) {
await (await upgrade(root)).close()
}
}
const later = await upgrade(root)
cleanup.push(later.close)
const copies = later.db
@@ -222,7 +237,10 @@ it('fills in revisions for chats converted before they were kept, then still kee
.all()
.map((row) => String(row.session_id))
expect(copies).toHaveLength(1)
expect(items(later.db, copies[0]!)).toContain('older-turn')
expect(await later.convert(copies[0]!)).toBe('epoch')
expect(items(later.db, copies[0]!)).toEqual(
expect.arrayContaining(['older-turn', 'older-turn-2', 'older-turn-3'])
)
expect(
later.db
.prepare("SELECT value FROM agent_session_store_meta WHERE key='session_tabs_recorded'")
@@ -302,9 +320,16 @@ it('keeps only what an older Orca wrote after a file from before revisions were
// The older Orca continues one converted chat and starts another.
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
olderOrcaChat(original, 'started-older', [rows[0], turn('started-turn', 2)])
const known = ['closed', 'converted', 'idle', 'open', 'started-older']
const compared = await upgrade(root)
// With no revision to compare, the open reads no history; a background pass compares it.
expect(
compared.db.prepare('SELECT COUNT(*) AS records FROM agent_session_records').get()
).toEqual({ records: known.length })
await compared.compare()
await compared.close()
const later = await upgrade(root)
cleanup.push(later.close)
const known = ['closed', 'converted', 'idle', 'open', 'started-older']
const [copy] = later.db
.prepare(
`SELECT session_id FROM agent_session_records WHERE session_id NOT IN (${known
@@ -313,8 +338,9 @@ it('keeps only what an older Orca wrote after a file from before revisions were
)
.all(...known)
.map((row) => String(row.session_id))
expect(await later.convert(copy!)).toBe('epoch')
expect(items(later.db, copy!)).toContain('older-turn')
expect(tabbedChats(later.db)).toEqual(['open', copy, 'started-older'])
expect(tabbedChats(later.db)).toEqual(['open', 'started-older', copy])
expect(later.db.prepare('SELECT COUNT(*) AS records FROM agent_session_records').get()).toEqual({
records: known.length + 1
})
@@ -331,6 +357,161 @@ it('keeps only what an older Orca wrote after a file from before revisions were
})
})
it('keeps a message an older Orca queued in a chat from before revisions were kept', async () => {
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
const { db: original } = openJournalDatabase(journalDatabasePath(root))
cleanup.push(() => original.close())
olderOrcaChat(original, 'queued', rows)
const first = await upgrade(root)
expect(await first.convert('queued')).toBe('epoch')
asOlderChatStore(first.db)
await first.close()
insertJournalRow(original, 'queued', {
v: AGENT_SESSION_JOURNAL_QUEUE_SCHEMA_VERSION,
epoch: 'epoch',
seq: tip + 1,
ts: 100,
fence: 1,
kind: 'queue',
messageId: 'older-queued',
change: {
op: 'queued',
body: { kind: 'message', role: 'user', blocks: [{ type: 'text', text: 'later' }] },
position: 1,
queuedAt: null
}
})
const compared = await upgrade(root)
await compared.compare()
await compared.close()
const later = await upgrade(root)
cleanup.push(later.close)
const copy = later.db
.prepare("SELECT session_id FROM agent_session_records WHERE session_id!='queued'")
.get()?.session_id
expect(await later.convert(String(copy))).toBe('epoch')
expect(
later.db
.prepare('SELECT message_id FROM journal_queue_cards WHERE session_id=?')
.all(String(copy))
).toContainEqual({ message_id: 'older-queued' })
})
it('records a chat it cannot keep and does not retry it until the older Orca writes it again', async () => {
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
const { db: original } = openJournalDatabase(journalDatabasePath(root))
cleanup.push(() => original.close())
olderOrcaChat(original, 'kept', rows)
await (await upgrade(root)).close()
olderOrcaChat(original, 'refused', [rows[0], turn('refused-turn', 2)])
original.pragma('wal_checkpoint(TRUNCATE)')
const blocked = openJournalDomainDatabase(journalDomainDatabasePath(root)).db
blocked.exec(`CREATE TRIGGER refused BEFORE INSERT ON agent_session_records
WHEN NEW.session_id='refused' BEGIN SELECT RAISE(ABORT,'record refused'); END`)
blocked.close()
const failing = openJournalDomainDatabase(await initializeJournalDomainDatabase(root)).db
const recorded = (key: string) =>
failing.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)?.value
expect(JSON.parse(String(recorded('journal.legacy-failed:refused')))).toMatchObject({
error: expect.stringContaining('record refused')
})
expect(recorded('journal.legacy-revisions-kept.v1')).toBe('1')
failing.exec('DROP TRIGGER refused')
failing.close()
const refusedRecord = (db: Database.Database) =>
db.prepare("SELECT 1 FROM agent_session_records WHERE session_id='refused'").get()
const absorbed = await upgrade(root)
expect(refusedRecord(absorbed.db)).toBeUndefined()
await absorbed.close()
insertJournalRow(original, 'refused', turn('refused-turn-2', 3))
original.pragma('wal_checkpoint(TRUNCATE)')
const retried = await upgrade(root)
cleanup.push(retried.close)
expect(refusedRecord(retried.db)).toEqual({ 1: 1 })
})
it('keeps a fresh copy when another build readied a copy before its history converted here', async () => {
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
const { db: original } = openJournalDatabase(journalDatabasePath(root))
cleanup.push(() => original.close())
olderOrcaChat(original, 'converted', rows)
const first = await upgrade(root)
expect(await first.convert('converted')).toBe('epoch')
await first.close()
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
original.pragma('wal_checkpoint(TRUNCATE)')
const copied = await upgrade(root)
const copies = (db: Database.Database) =>
db
.prepare("SELECT session_id FROM agent_session_records WHERE session_id!='converted'")
.all()
.map((row) => String(row.session_id))
const [stranded] = copies(copied.db)
// An older build that can't read the copy's source founds it empty, as a new chat.
copied.db
.prepare(
"INSERT INTO journal_chat_state(workspace_id,session_id,epoch) VALUES('workspace',?,'founded') ON CONFLICT(session_id) DO UPDATE SET epoch='founded'"
)
.run(stranded!)
await copied.close()
const again = await upgrade(root)
cleanup.push(again.close)
const kept = copies(again.db).find((id) => id !== stranded)
expect(copies(again.db)).toHaveLength(2)
expect(await again.convert(kept!)).toBe('epoch')
expect(items(again.db, kept!)).toContain('older-turn')
})
it('leaves a copy whose history failed to convert to an open, not to every idle pass', async () => {
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
const { db: original } = openJournalDatabase(journalDatabasePath(root))
cleanup.push(() => original.close())
olderOrcaChat(original, 'converted', rows)
const first = await upgrade(root)
expect(await first.convert('converted')).toBe('epoch')
await first.close()
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
original.pragma('wal_checkpoint(TRUNCATE)')
const later = await upgrade(root)
cleanup.push(later.close)
const { db } = later
const [copy] = db
.prepare("SELECT session_id FROM agent_session_records WHERE session_id!='converted'")
.all()
.map((row) => String(row.session_id))
const idle = () => {
const page = readJournalSqlDatabase(db, { kind: 'sessions', after: null })
return page.kind === 'sessions' ? page.sessionIds : []
}
expect(idle()).toContain(copy)
for (const sessionId of [copy!, 'converted']) {
runJournalTransaction(db, () =>
applyJournalSqlMaintenance(db, { kind: 'legacy-copy-failed', sessionId, error: 'damaged' })
)
}
expect(idle()).not.toContain(copy)
expect(idle()).toContain('converted')
expect(await later.convert(copy!)).toBe('epoch')
expect(items(db, copy!)).toContain('older-turn')
expect(
db
.prepare('SELECT 1 FROM agent_session_store_meta WHERE key=?')
.get(`journal.legacy-failed:${copy}`)
).toBeUndefined()
})
// Perf contract: an unchanged original only needs baselines, so it skips adoption at any fleet size.
it.each([10, 100])(
'fills in %i missing baselines without reading tabs, claims or chat state when the original is unchanged',
@@ -1,21 +1,32 @@
import { createHash } from 'node:crypto'
import type Database from '../../../sqlite/sync-database'
import type { SqliteRow } from '../../../sqlite/sqlite-statement'
import { structuredAgentSessionTabId } from '../../../../shared/agent-session/view/structured-agent-session-projection'
import { prepareJournalDomainImport, type JournalDomainImportPage } from '../journal-domain-import'
import {
finishJournalDomainImport,
insertJournalDomainImportPage
} from '../journal-domain-import-rows'
import { runJournalTransaction } from './journal-database'
import { isSqliteCorruption, isSqliteStorageFailure } from '../../../sqlite/sqlite-read-failure'
import { AGENT_SESSION_TABS_RECORDED } from '../../../runtime/agent-session/records/agent-session-record-rows'
import { readAdoption, type Adoption, type Claim } from './journal-legacy-adoption'
import { journalDatabasePath } from './journal-host-database'
import {
journalLegacyChatRevision,
journalLegacyChatRevisionKey,
openJournalLegacySource,
recordJournalLegacyChatRevision
recordJournalLegacyChatRevision,
settleJournalLegacyCopy
} from './journal-legacy-source'
import {
keepContinuedChat,
keepForcedCopy,
type JournalLegacyForcedCopy
} from './journal-legacy-copies'
import {
JOURNAL_LEGACY_COMPARE,
JOURNAL_LEGACY_COPY_DUE,
metaKeys,
readMeta,
recordJournalLegacyFailure,
strandedCopies,
writeMeta,
type JournalLegacyFailure
} from './journal-legacy-reconcile-marks'
import {
captureJournalLegacySourceReceipt,
JOURNAL_LEGACY_SOURCE_RECEIPT,
@@ -24,12 +35,13 @@ import {
/** Set after a pass absorbed every original chat, so each chat here carries its revision. */
export const JOURNAL_LEGACY_REVISIONS_KEPT = 'journal.legacy-revisions-kept.v1'
/**
* An older Orca keeps writing the original journal after a downgrade. When the original changed,
* each original chat's revision is compared with the one this file shows: a chat it started, or
* one not converted here yet, joins as itself; a converted chat it continued is kept as a second
* chat beside it, so no turn it saved goes missing. Nothing here deletes a chat or its turns.
* Terminal restore waits for this pass, so it reads no history: a copy's history converts later
* like any unconverted chat's, and a chat needing a history compare is compared in the background.
*/
export function reconcileJournalLegacySource(stateDirectory: string, db: Database.Database): void {
const sourcePath = journalDatabasePath(stateDirectory)
@@ -42,19 +54,29 @@ export function reconcileJournalLegacySource(stateDirectory: string, db: Databas
const saved = readMeta(db, JOURNAL_LEGACY_SOURCE_RECEIPT)
const unchanged = sameJournalLegacySource(saved, captured)
const kept = readMeta(db, JOURNAL_LEGACY_REVISIONS_KEPT) !== undefined
if (unchanged && kept) {
const forced = new Map<string, JournalLegacyForcedCopy>(
metaKeys(db, JOURNAL_LEGACY_COPY_DUE).map((id) => [id, {}])
)
for (const [id, copy] of strandedCopies(db)) {
forced.set(id, { stranded: copy })
}
if (unchanged && kept && forced.size === 0) {
return
}
const receipt = JSON.stringify(captured)
const pass: Pass = { kept, baselines: [] }
const pass: Pass = { kept, forced, baselines: [], compares: [], failures: [] }
const source = openJournalLegacySource(sourcePath)
let complete = true
try {
if (unchanged) {
// The original wrote nothing since the copy, so each chat is as shown here.
pass.baselines = missingBaselines(source, db)
// The original wrote nothing since the copy, so each chat is as shown here; once every
// chat carries its revision, only chats due a copy have work.
if (!kept) {
pass.baselines = missingBaselines(source, db).filter(([id]) => !forced.has(id))
}
complete = forced.size === 0 || absorbChats(source, db, pass, false)
} else {
complete = absorbChats(source, db, pass)
complete = absorbChats(source, db, pass, true)
}
} finally {
source.close()
@@ -71,7 +93,17 @@ export function reconcileJournalLegacySource(stateDirectory: string, db: Databas
for (const [id, revision] of pass.baselines) {
recordJournalLegacyChatRevision(db, id, revision)
}
// A failed chat leaves the original unabsorbed, so the next launch retries it.
for (const [id, revision] of pass.compares) {
writeMeta(db, `${JOURNAL_LEGACY_COMPARE}${id}`, revision)
}
for (const failure of pass.failures) {
recordJournalLegacyFailure(db, failure)
const { stranded } = pass.forced.get(failure.sessionId) ?? {}
if (stranded !== undefined) {
settleJournalLegacyCopy(db, stranded)
}
}
// A storage failure leaves the original unabsorbed; the next launch retries it.
if (complete) {
writeMeta(db, JOURNAL_LEGACY_SOURCE_RECEIPT, receipt)
writeMeta(db, JOURNAL_LEGACY_REVISIONS_KEPT, '1')
@@ -83,19 +115,12 @@ export function reconcileJournalLegacySource(stateDirectory: string, db: Databas
type Pass = {
/** Every chat in the original at the copy was given a revision here. */
kept: boolean
/** Chats that get a copy whatever their revision shows. */
forced: ReadonlyMap<string, JournalLegacyForcedCopy>
baselines: [string, string][]
}
type Claim = SqliteRow
/** The original's tabs and claims, and this file's tabs, read once per pass. */
type Adoption = {
savedTabs: Map<string, string>
claims: Map<string, Claim[]>
tabbed: Set<string>
takenTabs: Set<string>
nextPosition: number
tabsRecorded: boolean
/** Converted chats with no revision to compare, for the background compare. */
compares: [string, string][]
failures: JournalLegacyFailure[]
}
/**
@@ -120,64 +145,69 @@ function originalChatIds(source: Database.Database, order: 'saved' | 'any'): str
.map((row) => String(row.session_id))
}
/** True when every chat was absorbed; a failed chat is logged and retried next launch. */
function absorbChats(source: Database.Database, db: Database.Database, pass: Pass): boolean {
let adoption = readAdoption(source, db)
/**
* False only when a storage failure stopped the pass. A chat that fails otherwise is recorded with its
* error and taken as absorbed, so one bad chat can't rerun the pass at every launch. Without a walk,
* only chats due a copy are visited.
*/
function absorbChats(
source: Database.Database,
db: Database.Database,
pass: Pass,
walk: boolean
): boolean {
const adoption = readAdoption(source, db, walk ? null : [...pass.forced.keys()])
runJournalTransaction(db, () => {
for (const [sessionId, claims] of adoption.claims) {
insertClaims(db, sessionId, claims)
}
})
let complete = true
for (const id of originalChatIds(source, 'saved')) {
const chats = walk ? originalChatIds(source, 'saved') : [...pass.forced.keys()]
for (const id of chats) {
let revision: string | undefined
const added = adoption.added.length
const { nextPosition, tabsRecorded } = adoption
try {
absorbChat(source, db, id, adoption, pass)
// A chat with no history still has a value, so one deleted here stays deleted.
revision = journalLegacyChatRevision(source, id) ?? 'null'
absorbChat(source, db, id, revision, adoption, pass)
} catch (error) {
complete = false
// The failed chat's transaction rolled back; reread what this file now holds.
adoption = readAdoption(source, db)
console.warn('[agent-session-journal] keeping an older Orca chat failed', error)
// Corruption past the original's own revision read may be this file's: never absorb it. A
// rollback that failed leaves this connection mid-transaction for every later chat.
if (
isSqliteStorageFailure(error) ||
(revision !== undefined && isSqliteCorruption(error)) ||
db.isTransaction
) {
console.warn('[agent-session-journal] keeping chats an older Orca saved paused', error)
return false
}
pass.failures.push({ sessionId: id, revision, error })
// The failed chat's transaction rolled back, and with it any tab it added.
for (const [sessionId, tab] of adoption.added.splice(added)) {
adoption.tabbed.delete(sessionId)
adoption.takenTabs.delete(tab)
}
Object.assign(adoption, { nextPosition, tabsRecorded })
}
}
return complete
}
function readAdoption(source: Database.Database, db: Database.Database): Adoption {
const claims = new Map<string, Claim[]>()
for (const claim of source
.prepare('SELECT upload_id,session_id,claimed_at FROM agent_session_attachment_claims')
.all()) {
const id = String(claim.session_id)
const list = claims.get(id) ?? []
list.push(claim)
claims.set(id, list)
}
const tabs = db.prepare('SELECT tab_id,session_id FROM agent_session_tabs').all()
const tail = db.prepare('SELECT COALESCE(MAX(position),-1) AS tail FROM agent_session_tabs').get()
return {
savedTabs: new Map(
source
.prepare('SELECT tab_id,session_id FROM agent_session_tabs')
.all()
.map((row) => [String(row.session_id), String(row.tab_id)])
),
claims,
tabbed: new Set(tabs.map((row) => String(row.session_id))),
takenTabs: new Set(tabs.map((row) => String(row.tab_id))),
nextPosition: Number(tail?.tail ?? -1) + 1,
tabsRecorded: readMeta(db, AGENT_SESSION_TABS_RECORDED) !== undefined
}
return true
}
function absorbChat(
source: Database.Database,
db: Database.Database,
id: string,
revision: string,
adoption: Adoption,
pass: Pass
): void {
// A chat with no history still has a value, so one deleted here stays deleted.
const revision = journalLegacyChatRevision(source, id) ?? 'null'
const adopt = (copy: string) => adoptChat(source, db, id, copy, adoption)
const forced = pass.forced.get(id)
if (forced) {
runJournalTransaction(db, () => keepForcedCopy(db, id, revision, forced, adopt))
return
}
const shown = readMeta(db, journalLegacyChatRevisionKey(id))
if (revision === shown) {
return
@@ -197,7 +227,7 @@ function absorbChat(
return
}
if (converted) {
keepContinuedChat(source, db, id, revision, shown !== undefined, adoption, pass)
keepContinuedChat(db, id, revision, shown, adopt, pass)
return
}
// Not converted here yet: its conversion reads the current original, so it joins as itself.
@@ -238,10 +268,11 @@ function adoptChat(
const session = source
.prepare('SELECT workspace_id FROM journal_sessions WHERE session_id=?')
.get(id)
if (as === id && session) {
// A copy is cataloged unconverted too; its history converts from the original chat.
if (session) {
db.prepare(
'INSERT INTO journal_chat_state(session_id,workspace_id) VALUES(?,?) ON CONFLICT DO NOTHING'
).run(id, session.workspace_id)
).run(as, session.workspace_id)
}
if (as !== id) {
insertClaims(db, as, adoption.claims.get(id) ?? [])
@@ -274,73 +305,7 @@ function addTab(db: Database.Database, adoption: Adoption, sessionId: string, ta
adoption.nextPosition += 1
adoption.tabbed.add(sessionId)
adoption.takenTabs.add(tab)
}
/**
* The original's version of a converted chat it moved past becomes a second chat beside it. With
* no revision to compare (files copied before revisions were kept), the original moved past this
* chat only if its restored history has an item this chat never held.
*/
function keepContinuedChat(
source: Database.Database,
db: Database.Database,
id: string,
revision: string,
compared: boolean,
adoption: Adoption,
pass: Pass
): void {
const imported = prepareJournalDomainImport(source, id)
if (!imported) {
pass.baselines.push([id, revision])
return
}
const { pages: read, ...header } = imported
let pages: Iterable<JournalDomainImportPage> = read
if (!compared) {
const all = [...read]
if (heldEveryItem(db, id, all)) {
pass.baselines.push([id, revision])
return
}
pages = all
}
// Derived from the revision, so an interrupted import resumes into the same chat.
const copy = createHash('sha256').update(`${id}\0${revision}`).digest('hex').slice(0, 32)
const target = { sessionId: copy, epoch: header.epoch }
for (const page of pages) {
runJournalTransaction(db, () => insertJournalDomainImportPage(db, target, page))
}
runJournalTransaction(db, () => {
finishJournalDomainImport(db, { ...header, sessionId: copy, sourceRevision: null })
adoptChat(source, db, id, copy, adoption)
recordJournalLegacyChatRevision(db, id, revision)
})
}
/**
* Removing an item here leaves its tombstone, so every item this chat converted stays held, at
* its converted revision or a later one.
*/
function heldEveryItem(
db: Database.Database,
id: string,
pages: readonly JournalDomainImportPage[]
): boolean {
const held = new Map(
db
.prepare(`SELECT item_id AS id, MAX(revision) AS revision FROM (
SELECT item_id, revision FROM journal_items WHERE session_id=?1
UNION ALL SELECT item_id, ?2 FROM journal_item_tombstones WHERE session_id=?1)
GROUP BY item_id`)
.all(id, Number.MAX_SAFE_INTEGER)
.map((row) => [String(row.id), Number(row.revision)])
)
return pages.every(
(page) =>
page.kind !== 'items' ||
page.rows.every(({ item }) => (held.get(item.itemId) ?? -1) >= item.revision)
)
adoption.added.push([sessionId, tab])
}
/** Uploads the older Orca attached stay claimed, so the sweeper never removes them. */
@@ -351,15 +316,3 @@ function insertClaims(db: Database.Database, sessionId: string, claims: readonly
insert.run(claim.upload_id!, sessionId, claim.claimed_at!)
}
}
function readMeta(db: Database.Database, key: string): string | undefined {
const row = db.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)
return row === undefined ? undefined : String(row.value)
}
function writeMeta(db: Database.Database, key: string, value: string): void {
db.prepare('INSERT OR REPLACE INTO agent_session_store_meta (key, value) VALUES (?, ?)').run(
key,
value
)
}
@@ -6,10 +6,9 @@ import { afterEach, expect, it } from 'vitest'
import { openJournalDatabase, runJournalTransaction } from './journal-database'
import {
initializeJournalDomainDatabase,
journalDomainDatabasePath,
openJournalDomainDatabase
} from './journal-domain-database'
import { journalDatabasePath } from './journal-host-database'
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
import { insertJournalRow, publishJournalSessionEpoch } from '../durable-rows/journal-row-table'
import { journalSqlRetryRows } from '../journal-sql-retry-rows.test-fixture'
import { JournalSqlReaderPool } from '../journal-sql-reader-pool'
@@ -90,6 +90,7 @@ export function journalLegacyChatRevisionKey(sessionId: string): string {
return `journal.legacy-revision:${sessionId}`
}
/** Absorbing a revision clears any failure an earlier one recorded. */
export function recordJournalLegacyChatRevision(
db: Database.Database,
sessionId: string,
@@ -99,4 +100,47 @@ export function recordJournalLegacyChatRevision(
journalLegacyChatRevisionKey(sessionId),
revision
)
db.prepare('DELETE FROM agent_session_store_meta WHERE key=?').run(
journalLegacyFailureKey(sessionId)
)
}
/** A chat, or a copy's history, that could not be kept, with its error; not retried by itself. */
export function journalLegacyFailureKey(sessionId: string): string {
return `journal.legacy-failed:${sessionId}`
}
/** A copy kept for an original chat an older Orca continued converts its history from that chat. */
export function journalLegacyCopySourceKey(copyId: string): string {
return `journal.legacy-copy-source:${copyId}`
}
/** The original chat a copy converts from, or the chat itself. */
export function journalLegacyConversionSource(db: Database.Database, sessionId: string): string {
const row = db
.prepare('SELECT value FROM agent_session_store_meta WHERE key=?')
.get(journalLegacyCopySourceKey(sessionId))
return row === undefined ? sessionId : String(row.value)
}
/** The copy kept for an original chat that hasn't converted here yet. */
export function journalLegacyPendingCopyKey(sessionId: string): string {
return `journal.legacy-pending-copy:${sessionId}`
}
/** Once a copy is ready here its history no longer reads from the original. */
export function settleJournalLegacyCopy(db: Database.Database, copyId: string): void {
const key = journalLegacyCopySourceKey(copyId)
const original = db.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)
if (original === undefined) {
return
}
db.prepare('DELETE FROM agent_session_store_meta WHERE key IN (?,?)').run(
key,
journalLegacyFailureKey(copyId)
)
db.prepare('DELETE FROM agent_session_store_meta WHERE key=? AND value=?').run(
journalLegacyPendingCopyKey(String(original.value)),
copyId
)
}
@@ -15,8 +15,7 @@ import type { JournalWriterEffect, JournalWriterEffectResult } from '../journal-
import type { CommandReceiptScope } from '../writing/command-receipt-schema'
import { AgentSessionJournalError } from '../writing/journal-write-guards'
import { assertJournalChatReadable } from '../replay/journal-open-failure'
import { journalDatabasePath } from './journal-host-database'
import { journalDomainDatabasePath } from './journal-domain-database'
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
import type { JournalSqlStoredRecords } from '../journal-sql-database-read'
/** The runtime owns workers, never a writable connection or a retained chat projection. */
@@ -9,7 +9,10 @@ import type Database from '../../sqlite/sync-database'
import { insertJournalDomainQueueRow, journalDomainQueueRow } from './journal-domain-queue-rows'
import type { JournalDomainImportHeader, JournalDomainImportPage } from './journal-domain-import'
import { writeJournalDomainItemNames } from './journal-domain-item-names'
import { recordJournalLegacyChatRevision } from './database/journal-legacy-source'
import {
recordJournalLegacyChatRevision,
settleJournalLegacyCopy
} from './database/journal-legacy-source'
import { hydrateJournalStoredItem, journalStoredItemRow } from './journal-stored-output-rows'
export type JournalDomainImportState =
@@ -214,6 +217,7 @@ export function finishJournalDomainImport(
if (header.sourceRevision) {
recordJournalLegacyChatRevision(db, header.sessionId, header.sourceRevision)
}
settleJournalLegacyCopy(db, header.sessionId)
return state.epoch
}
@@ -305,6 +305,7 @@ it('finishes a retired pass during activity while an already-started conversion
}
},
{
compareLegacyChats: async () => {},
ensure: async (_sessionId, foreground) => {
if (foreground) {
throw new Error('background conversion was promoted')
@@ -122,7 +122,7 @@ it('deletes an idle chat with its tab, then every journal row in background page
inTransaction(db, () => applyJournalSqlMaintenance(db, operation))
},
{ database: async (operation) => readJournalSqlDatabase(db, operation) },
{ ensure: async () => null },
{ ensure: async () => null, compareLegacyChats: async () => {} },
new JournalSqlActivity(),
(error) => {
throw error
@@ -149,6 +149,43 @@ it('deletes an idle chat with its tab, then every journal row in background page
expect(count(db, "journal_chat_state WHERE session_id='kept-chat'")).toBe(1)
})
it('takes the revision a chat awaiting its older-Orca compare waited on, so no copy brings it back', () => {
const db = journal()
db.prepare(
`DELETE FROM agent_session_store_meta WHERE key='journal.legacy-revision:gone-chat'`
).run()
db.prepare(
`INSERT INTO agent_session_store_meta VALUES('journal.legacy-compare:gone-chat','["legacy",4]')`
).run()
expect(forget(db)).toBe(true)
expect(count(db, "agent_session_store_meta WHERE key LIKE 'journal.legacy-compare:%'")).toBe(0)
expect(
db
.prepare(
"SELECT value FROM agent_session_store_meta WHERE key='journal.legacy-revision:gone-chat'"
)
.get()?.value
).toBe('["legacy",4]')
})
it("drops a forgotten copy's mapping and failure, so none of its keys outlive it", () => {
const db = journal()
for (const [key, value] of [
['journal.legacy-copy-source:gone-chat', 'original-chat'],
['journal.legacy-failed:gone-chat', '{"revision":null,"error":"damaged"}'],
['journal.legacy-pending-copy:original-chat', 'gone-chat']
]) {
db.prepare('INSERT INTO agent_session_store_meta VALUES(?,?)').run(key, value)
}
expect(forget(db)).toBe(true)
expect(
count(
db,
"agent_session_store_meta WHERE key LIKE 'journal.legacy-copy-source:%' OR key LIKE 'journal.legacy-failed:%' OR key LIKE 'journal.legacy-pending-copy:%'"
)
).toBe(0)
})
const refusals: {
name: string
gone?: Partial<AgentSessionRecord>
@@ -28,7 +28,7 @@ export class JournalSqlBackgroundMaintenance {
constructor(
private readonly writer: Pick<JournalSqlOwner, 'maintain'>,
private readonly readers: Pick<JournalSqlReaderPool, 'database'>,
private readonly conversions: Pick<JournalSqlConversions, 'ensure'>,
private readonly conversions: Pick<JournalSqlConversions, 'ensure' | 'compareLegacyChats'>,
private readonly activity: JournalSqlActivity,
private readonly onError: (error: unknown, sessionId?: string) => void,
private readonly onTick?: () => void
@@ -114,6 +114,7 @@ export class JournalSqlBackgroundMaintenance {
if (this.discover) {
this.discover = false
try {
await this.conversions.compareLegacyChats(() => this.boundary())
await this.convertSavedChats()
} catch (error) {
this.report(error)
@@ -1,9 +1,20 @@
import { setImmediate } from 'node:timers/promises'
import type { JournalSqlOwner } from './journal-sql-owner'
import type { JournalSqlReaderPool } from './journal-sql-reader-pool'
import { compareJournalLegacyChatsWhenIdle } from './journal-sql-legacy-compares'
import { isJournalSqlSourceContentFailure } from './journal-sql-failure'
import { isJournalReaderReplyLimitFailure } from './journal-sql-reader-reply-limit'
type Conversion = { promotion: AbortController; result: Promise<string | null> }
function warnLegacy(step: string, sessionId: string, error: unknown): void {
console.warn(
`[agent-session-journal] ${step} a chat an older Orca saved failed`,
sessionId,
error
)
}
/** One lock per chat, shared by background work and opens; promotion wakes a paused conversion. */
export class JournalSqlConversions {
private readonly pending = new Map<string, Conversion>()
@@ -12,10 +23,26 @@ export class JournalSqlConversions {
constructor(
private readonly writer: Pick<JournalSqlOwner, 'maintain'>,
private readonly readers: Pick<JournalSqlReaderPool, 'importState' | 'prepareConversion'>,
private readonly readers: Pick<
JournalSqlReaderPool,
'importState' | 'prepareConversion' | 'database' | 'compareLegacyChat'
>,
private readonly waitForIdle: (promotion: AbortSignal) => Promise<void>
) {}
/** Converted chats an older Orca may have continued, compared before idle conversion. */
compareLegacyChats(boundary: () => Promise<boolean>): Promise<void> {
return compareJournalLegacyChatsWhenIdle(
this.writer,
this.readers,
boundary,
() => this.closed,
(error, sessionId) => warnLegacy('comparing', sessionId, error)
).catch((error: unknown) =>
console.warn('[agent-session-journal] comparing chats an older Orca saved stopped', error)
)
}
async ensure(sessionId: string, foreground = true): Promise<string | null> {
if (this.closed) {
throw new Error('journal conversions are closed')
@@ -42,6 +69,10 @@ export class JournalSqlConversions {
}
return epoch
})
.catch((error: unknown) => {
this.noteCopyFailure(sessionId, error)
throw error
})
.finally(() => {
this.pending.delete(sessionId)
})
@@ -49,6 +80,20 @@ export class JournalSqlConversions {
return result
}
/** A copy whose original content can't convert or fit a reply is left to an open, never to a quit's rejection. */
private noteCopyFailure(sessionId: string, error: unknown): void {
if (
this.closed ||
!(isJournalSqlSourceContentFailure(error) || isJournalReaderReplyLimitFailure(error))
) {
return
}
const message = (error instanceof Error ? error.message : String(error)).slice(0, 500)
void this.writer
.maintain({ kind: 'legacy-copy-failed', sessionId, error: message })
.catch((failure: unknown) => warnLegacy('recording', sessionId, failure))
}
/** A ready physical epoch survives context changes for the lifetime of the database owner. */
rememberReady(sessionId: string, epoch: string): void {
this.ready.set(sessionId, epoch)
@@ -12,6 +12,14 @@ import {
} from '../../runtime/agent-session/records/agent-session-tab-table'
import { readCommandReceipt } from './writing/command-receipt-table'
import type { CommandReceiptScope } from './writing/command-receipt-schema'
import {
journalLegacyCopySourceKey,
journalLegacyFailureKey
} from './database/journal-legacy-source'
import {
pendingJournalLegacyCompares,
type JournalLegacyCompare
} from './database/journal-legacy-compare'
import { readForgottenJournalChats } from './journal-forgotten-chats'
import { readJournalConnectorReplies } from './connector-replies/journal-connector-replies'
import type { JournalConnectorReplyRow } from '../../../shared/agent-session/journal/journal-connector-reply'
@@ -21,6 +29,12 @@ export type JournalSqlStoredRecords = Omit<AgentSessionStoreState, 'sessionTabs'
backgroundSessions: StoredAgentSessionBackground | null
}
/** Background conversion leaves a copy whose history failed to convert to an open. */
const UNFAILED_COPY = `NOT EXISTS (SELECT 1 FROM agent_session_store_meta f
WHERE f.key='${journalLegacyFailureKey('')}'||c.session_id
AND EXISTS (SELECT 1 FROM agent_session_store_meta m
WHERE m.key='${journalLegacyCopySourceKey('')}'||c.session_id))`
/** A pending chat; epoch is null once its head is gone, generation settles exactly this entry. */
export type JournalRetiredHead = { sessionId: string; epoch: string | null; generation: number }
@@ -33,6 +47,7 @@ export type JournalSqlDatabaseRead =
| { kind: 'retired-heads'; after: string | null; through: string | null }
| { kind: 'attachment-claims'; after: number }
| { kind: 'attachment-sweeps'; after: string | null }
| { kind: 'legacy-compares'; after: string | null }
| { kind: 'forgotten-chats'; after: string | null }
| { kind: 'connector-replies'; sessionId?: string }
@@ -42,6 +57,7 @@ export type JournalSqlDatabaseReadResult =
| { kind: 'receipt'; receipt: ReturnType<typeof readCommandReceipt> }
| { kind: 'epoch'; epoch: string | null }
| { kind: 'sessions'; sessionIds: string[] }
| { kind: 'legacy-compares'; compares: JournalLegacyCompare[] }
| { kind: 'retired-heads'; heads: JournalRetiredHead[]; through: string | null }
| {
kind: 'attachment-claims'
@@ -106,15 +122,22 @@ function read(
.get(operation.sessionId)
?.epoch?.toString() ?? null
}
case 'legacy-compares':
return {
kind: 'legacy-compares',
compares: pendingJournalLegacyCompares(db, operation.after)
}
case 'sessions': {
const rows =
operation.after === null
? db
.prepare('SELECT session_id FROM journal_chat_state ORDER BY session_id LIMIT 32')
.prepare(
`SELECT session_id FROM journal_chat_state c WHERE ${UNFAILED_COPY} ORDER BY session_id LIMIT 32`
)
.all()
: db
.prepare(`SELECT session_id FROM journal_chat_state
WHERE session_id>? ORDER BY session_id LIMIT 32`)
.prepare(`SELECT session_id FROM journal_chat_state c
WHERE session_id>? AND ${UNFAILED_COPY} ORDER BY session_id LIMIT 32`)
.all(operation.after)
return { kind: 'sessions', sessionIds: rows.map((row) => String(row.session_id)) }
}
@@ -101,8 +101,10 @@ it('expires replay during continuous activity even while conversion and sweeps w
}
}
},
{ database: async (operation) => readJournalSqlDatabase(db, operation) },
{ ensure: async () => null },
{
database: async (operation) => readJournalSqlDatabase(db, operation)
},
{ ensure: async () => null, compareLegacyChats: async () => {} },
activity,
(error) => {
throw error
@@ -44,6 +44,11 @@ function journalSqlSourceContent(error: unknown): JournalSourceContent | undefin
return undefined
}
/** The original journal's own content failed, not the transport or this process's lifecycle. */
export function isJournalSqlSourceContentFailure(error: unknown): boolean {
return journalSqlSourceContent(error) !== undefined
}
export function journalSqlSourceFailure(error: unknown): JournalSqlFailure {
const failure = journalSqlFailure(error)
if (isSqliteCorruption(error)) {
@@ -0,0 +1,119 @@
import { expect, it } from 'vitest'
import { compareJournalLegacyChatsWhenIdle } from './journal-sql-legacy-compares'
import type { JournalSqlMaintenance } from './journal-sql-maintenance'
async function runWith(failures: Error[], stopping = false): Promise<JournalSqlMaintenance[]> {
const settled: JournalSqlMaintenance[] = []
let pages = 0
await compareJournalLegacyChatsWhenIdle(
{
maintain: async (operation) => {
settled.push(operation)
return null
}
},
{
database: async () => ({
kind: 'legacy-compares',
compares: pages++ === 0 ? [{ sessionId: 'huge', revision: 'r1' }] : []
}),
compareLegacyChat: async () => {
const failure = failures.shift()
if (failure) {
throw failure
}
return { revision: 'r1', held: true }
}
},
async () => true,
() => stopping,
() => {}
)
return settled
}
const exited = () => new Error('journal SQL reader exited')
it('absorbs a chat whose compare took its reader down twice instead of retrying it every launch', async () => {
expect(await runWith([exited(), exited()])).toEqual([
{
kind: 'legacy-compare-settle',
sessionId: 'huge',
revision: 'r1',
error: 'journal SQL reader exited'
}
])
expect(await runWith([exited()])).toEqual([
{ kind: 'legacy-compare-settle', sessionId: 'huge', revision: 'r1', held: true }
])
expect(await runWith([exited()], true)).toEqual([])
})
it("leaves only the damaged chat's compare pending when this file's storage is damaged", async () => {
const corrupt = Object.assign(new Error('database disk image is malformed'), {
code: 'ERR_SQLITE_ERROR',
errcode: 11
})
const settled: JournalSqlMaintenance[] = []
let pages = 0
await compareJournalLegacyChatsWhenIdle(
{
maintain: async (operation) => {
settled.push(operation)
return null
}
},
{
database: async () => ({
kind: 'legacy-compares',
compares:
pages++ === 0
? [
{ sessionId: 'damaged', revision: 'r1' },
{ sessionId: 'healthy', revision: 'r1' }
]
: []
}),
compareLegacyChat: async (sessionId) => {
if (sessionId === 'damaged') {
throw corrupt
}
return { revision: 'r1', held: true }
}
},
async () => true,
() => false,
() => {}
)
expect(settled).toEqual([
{ kind: 'legacy-compare-settle', sessionId: 'healthy', revision: 'r1', held: true }
])
})
it('stops at a missing original instead of asking about every chat awaiting a compare', async () => {
const asked: string[] = []
let pages = 0
await compareJournalLegacyChatsWhenIdle(
{ maintain: async () => null },
{
database: async () => ({
kind: 'legacy-compares',
compares:
pages++ === 0
? [
{ sessionId: 'first', revision: 'r1' },
{ sessionId: 'second', revision: 'r1' }
]
: []
}),
compareLegacyChat: async (sessionId) => {
asked.push(sessionId)
return null
}
},
async () => true,
() => false,
() => {}
)
expect(asked).toEqual(['first'])
})
@@ -0,0 +1,73 @@
import type { JournalSqlOwner } from './journal-sql-owner'
import type { JournalSqlReaderPool } from './journal-sql-reader-pool'
import { isSqliteCorruption, isSqliteStorageFailure } from '../../sqlite/sqlite-read-failure'
/**
* Converted chats an older Orca may have continued, with no revision to compare, are compared one
* per idle step on a reader. Each settles durably, so an interrupted run resumes next launch.
*/
export async function compareJournalLegacyChatsWhenIdle(
writer: Pick<JournalSqlOwner, 'maintain'>,
readers: Pick<JournalSqlReaderPool, 'database' | 'compareLegacyChat'>,
boundary: () => Promise<boolean>,
stopped: () => boolean,
report: (error: unknown, sessionId: string) => void
): Promise<void> {
let after: string | null = null
while (await boundary()) {
const page = await readers.database({ kind: 'legacy-compares', after })
if (page.kind !== 'legacy-compares') {
throw new Error('journal reader returned no legacy compares')
}
if (page.compares.length === 0) {
return
}
for (const compare of page.compares) {
if (!(await boundary())) {
return
}
const result = await compareOnce(readers, compare.sessionId, stopped, report)
// No original, stopping, or this file's storage failing: no other chat would fare better.
if (result === null) {
return
}
if (result === 'pending') {
continue
}
if ('error' in result || result.revision === compare.revision) {
await writer.maintain({ kind: 'legacy-compare-settle', ...compare, ...result })
}
}
after = page.compares.at(-1)!.sessionId
}
}
/**
* This file's own failures, contention and stopping leave the compare for the next launch; damage
* here may be this chat's alone, so only it stays pending. A reader
* that exits gets one retry, since another request may have taken it down; a second exit is this
* chat's, absorbed with its error so it never reruns into the same crash.
*/
async function compareOnce(
readers: Pick<JournalSqlReaderPool, 'compareLegacyChat'>,
sessionId: string,
stopped: () => boolean,
report: (error: unknown, sessionId: string) => void
): Promise<Awaited<ReturnType<JournalSqlReaderPool['compareLegacyChat']>> | 'pending'> {
for (let attempt = 0; ; attempt++) {
try {
return await readers.compareLegacyChat(sessionId)
} catch (error) {
report(error, sessionId)
if (stopped() || isSqliteStorageFailure(error)) {
return null
}
if (isSqliteCorruption(error)) {
return 'pending'
}
if (attempt > 0) {
return { error: error instanceof Error ? error.message : String(error) }
}
}
}
}
@@ -27,6 +27,8 @@ import { discardForgottenJournalChatPage } from './journal-forgotten-chats'
import { installAgentSessionStoreRows } from '../../runtime/agent-session/records/agent-session-record-rows'
import { storedAgentSessionBackground } from '../../runtime/agent-session/records/agent-session-tab-table'
import type { JournalSqlStoredRecords } from './journal-sql-database-read'
import { settleJournalLegacyCompare } from './database/journal-legacy-compare'
import { recordJournalLegacyCopyFailure } from './database/journal-legacy-copies'
import {
fillJournalConnectorReplies,
sentJournalConnectorReplies
@@ -60,6 +62,8 @@ export type JournalSqlMaintenance =
| { kind: 'import-finish'; header: JournalDomainImportHeader }
| JournalDomainMaintenance
| { kind: 'event-tail-prune'; now: number }
| ({ kind: 'legacy-compare-settle' } & Parameters<typeof settleJournalLegacyCompare>[1])
| { kind: 'legacy-copy-failed'; sessionId: string; error: string }
// The reply outbox's rows outside any chat's lane: filled with outcomes, and sent.
| { kind: 'connector-replies-fill'; fills: readonly JournalConnectorReplyFill[]; now: number }
| { kind: 'connector-replies-sent'; keys: readonly JournalConnectorReplyKey[]; now: number }
@@ -129,6 +133,12 @@ export function applyJournalSqlMaintenance(
return completeJournalDomainCompaction(db, operation.now)
case 'event-tail-prune':
return pruneJournalSqlEventTail(db, operation.now)
case 'legacy-compare-settle':
settleJournalLegacyCompare(db, operation)
return null
case 'legacy-copy-failed':
recordJournalLegacyCopyFailure(db, operation.sessionId, operation.error)
return null
case 'stored-output-compaction':
return compactJournalStoredOutputPage(db)
}
@@ -11,8 +11,10 @@ import { readJournalDomainHistoryFrame } from './journal-domain-history-page'
import { journalDomainImportState } from './journal-domain-import-rows'
import {
journalLegacyChatRevision,
journalLegacyConversionSource,
openJournalLegacySource
} from './database/journal-legacy-source'
import { compareJournalLegacyChat } from './database/journal-legacy-compare'
import { installWorkerSendLedger } from '../../worker-threads/worker-send-ledger'
import { limitJournalReaderReply } from './journal-sql-reader-reply-limit'
@@ -98,22 +100,30 @@ port.on('message', ({ id, operation }: JournalSqlReaderRequest) => {
break
case 'conversion-prepare': {
// Each frozen-source read is its own short statement; no snapshot spans the restore.
if (!original && existsSync(workerData.originalFilename)) {
original = openJournalLegacySource(workerData.originalFilename, { timeout: 5000 })
}
const prepared = original
? (prepareJournalDomainImport(original, operation.sessionId) ??
prepareEmptyConversion(original, operation.sessionId))
const source = openOriginal()
const from = journalLegacyConversionSource(db, operation.sessionId)
const prepared = source
? (prepareJournalDomainImport(source, from) ?? prepareEmptyConversion(source, from))
: null
if (prepared) {
const { pages, ...header } = prepared
conversions.set(id, pages)
reply({ type: 'prepared', id, conversion: header })
// A copy takes the original chat's history; the original's revision stays the original's.
const conversion =
from === operation.sessionId
? header
: { ...header, sessionId: operation.sessionId, sourceRevision: null }
reply({ type: 'prepared', id, conversion })
} else {
reply({ type: 'prepared', id, conversion: null })
}
break
}
case 'legacy-compare': {
const result = compareJournalLegacyChat(openOriginal, db, operation.sessionId)
reply({ type: 'legacy-compare', id, result })
break
}
case 'conversion-page': {
const pages = conversions.get(operation.conversionId)
if (!pages) {
@@ -160,6 +170,13 @@ port.on('message', ({ id, operation }: JournalSqlReaderRequest) => {
}
})
function openOriginal(): Database.Database | undefined {
if (!original && existsSync(workerData.originalFilename)) {
original = openJournalLegacySource(workerData.originalFilename, { timeout: 5000 })
}
return original
}
/** Preserve empty saved epochs independently of legacy-source retirement. */
function prepareEmptyConversion(
source: Database.Database,
@@ -4,6 +4,7 @@ import { JOURNAL_SQL_READERS } from './journal-sql-reader-requests'
import type { JournalForkPreparationInput } from './journal-fork-preparation'
import { throwIfSignalAborted } from '../../../shared/concurrency/abort-signal-reason'
import type { JournalOutlineCache } from './journal-domain-outline'
import type { JournalLegacyChatComparison } from './journal-sql-reader-protocol'
import {
currentWorkerEntryLayout,
resolveWorkerThreadEntryPath
@@ -127,6 +128,15 @@ export class JournalSqlReaderPool {
return this.conversionSlot.prepareConversion(sessionId)
}
/** Null when the original journal is gone. */
async compareLegacyChat(sessionId: string): Promise<JournalLegacyChatComparison | null> {
const response = await this.slot().request({ kind: 'legacy-compare', sessionId })
if (response.type !== 'legacy-compare') {
throw new Error('journal reader returned no legacy compare')
}
return response.result
}
async importState(sessionId: string) {
const response = await this.slot().request({ kind: 'import-state', sessionId })
if (response.type !== 'import-state') {
@@ -16,6 +16,9 @@ import type {
} from './journal-domain-history-page'
import type { JournalDomainImportState } from './journal-domain-import-rows'
/** A chat that can't be compared answers with its error. */
export type JournalLegacyChatComparison = { revision: string; held: boolean } | { error: string }
export type JournalSqlReaderOperation =
| { kind: 'fork-prepare'; input: JournalForkPreparationInput }
| { kind: 'fork-next'; preparationId: number }
@@ -31,6 +34,7 @@ export type JournalSqlReaderOperation =
| { kind: 'import-state'; sessionId: string }
| { kind: 'conversion-page'; conversionId: number }
| { kind: 'conversion-release'; conversionId: number }
| { kind: 'legacy-compare'; sessionId: string }
export type JournalSqlReaderRequest = { id: number; operation: JournalSqlReaderOperation }
export type JournalSqlReaderReply =
@@ -44,6 +48,7 @@ export type JournalSqlReaderReply =
| { type: 'import-state'; id: number; state: JournalDomainImportState }
| { type: 'rows'; id: number; page: JournalDomainImportPage | null }
| { type: 'released'; id: number }
| { type: 'legacy-compare'; id: number; result: JournalLegacyChatComparison | null }
| { type: 'failed'; id?: number; error: JournalSqlFailure }
export type JournalSqlReaderResult = Exclude<JournalSqlReaderReply, { type: 'ready' | 'failed' }>
@@ -44,10 +44,19 @@ export function limitJournalReaderReply<T extends { type: string }>(
return {
type: 'failed',
id: reply.id,
error: journalSqlFailure(new Error(`journal read reply (${reply.type}) exceeds ${limit} bytes`))
error: journalSqlFailure(
new Error(`${REPLY_LIMIT_FAILURE} (${reply.type}) exceeds ${limit} bytes`)
)
}
}
const REPLY_LIMIT_FAILURE = 'journal read reply'
/** A chat whose rows can't fit one reply fails the same way at every retry. */
export function isJournalReaderReplyLimitFailure(error: unknown): boolean {
return error instanceof Error && error.message.startsWith(`${REPLY_LIMIT_FAILURE} (`)
}
function isRecordSetReply(reply: unknown): boolean {
if (typeof reply !== 'object' || reply === null || !('type' in reply) || !('result' in reply)) {
return false
@@ -109,6 +109,7 @@ export function describeJournalReaderOperation(
return { operation: operation.kind, chat: chatHash(operation.input.source.sessionId) }
case 'conversion-prepare':
case 'import-state':
case 'legacy-compare':
return { operation: operation.kind, chat: chatHash(operation.sessionId) }
case 'fork-next':
case 'fork-release':
@@ -9,8 +9,7 @@ import { join } from 'node:path'
import { afterEach, expect, it, vi } from 'vitest'
import { codexProviderHandle } from '../../../shared/agent-session/lifecycle/provider-handles/agent-session-provider-handle-encoding'
import { openStructuredAgentSessionSqlJournalDatabase } from '../../runtime/agent-session/installation/structured-agent-session-journal-open'
import { journalDatabasePath } from './database/journal-host-database'
import { journalDomainDatabasePath } from './database/journal-domain-database'
import { journalDatabasePath, journalDomainDatabasePath } from './database/journal-host-database'
import type { JournalSqlDatabase } from './database/journal-sql-database'
import { SqlAgentSessionJournal } from './journal-sql-store'
import type { JournalSqlStoreOperation } from './journal-sql-store'
@@ -6,9 +6,9 @@ import { mkdirSync } from 'node:fs'
import { dirname } from 'node:path'
import { parentPort, workerData } from 'node:worker_threads'
import type Database from '../../sqlite/sync-database'
import { journalDomainDatabasePath } from './database/journal-host-database'
import {
initializeJournalDomainDatabase,
journalDomainDatabasePath,
openJournalDomainDatabase
} from './database/journal-domain-database'
import { JournalCheckpointOwner } from './database/journal-checkpoint-owner'
@@ -9,6 +9,7 @@ import type { JournalRow } from './durable-rows/journal-row-schema'
import { resolveJournalWriterStoreMutation } from './journal-writer-store-mutation'
import { commitAgentSessionPublicationFence } from '../../runtime/agent-session/agent-session-publication-fence'
import { markJournalChatForgotten } from './journal-forgotten-chats'
import { absorbForgottenJournalLegacyChat } from './database/journal-legacy-compare'
import { forgetJournalConnectorReplies } from './connector-replies/journal-connector-replies'
export function applyJournalWriterStoreEffect(
@@ -34,6 +35,7 @@ export function applyJournalWriterStoreEffect(
writeAgentSessionStoreRows(db, resolved.writes)
if (input.kind === 'command' && input.command.kind === 'forget-idle' && resolved.value === true) {
markJournalChatForgotten(db, input.command.sessionId)
absorbForgottenJournalLegacyChat(db, input.command.sessionId)
// Its connector requests are owed an outcome still: the chat no longer has them.
forgetJournalConnectorReplies(db, input.command.sessionId, Date.now())
}
@@ -6,11 +6,11 @@ import { join } from 'node:path'
import { afterEach, beforeEach, expect, it, vi } from 'vitest'
import { JournalSqlDatabase } from '../../../native-chat/agent-session-journal/database/journal-sql-database'
import { openJournalDatabase } from '../../../native-chat/agent-session-journal/database/journal-database'
import { journalDatabasePath } from '../../../native-chat/agent-session-journal/database/journal-host-database'
import {
journalDomainDatabasePath,
openJournalDomainDatabase
} from '../../../native-chat/agent-session-journal/database/journal-domain-database'
journalDatabasePath,
journalDomainDatabasePath
} from '../../../native-chat/agent-session-journal/database/journal-host-database'
import { openJournalDomainDatabase } from '../../../native-chat/agent-session-journal/database/journal-domain-database'
import {
journalSqlReaderPath,
journalSqlWorkerPath
@@ -2,8 +2,10 @@ import { sessionCrewChanges } from '../../orchestration/crew/crew-changes'
// Launch admission opens the single-writer store before chat-host installation; both must reuse this instance.
import { existsSync } from 'node:fs'
import { journalDatabasePath } from '../../../native-chat/agent-session-journal/database/journal-host-database'
import { journalDomainDatabasePath } from '../../../native-chat/agent-session-journal/database/journal-domain-database'
import {
journalDatabasePath,
journalDomainDatabasePath
} from '../../../native-chat/agent-session-journal/database/journal-host-database'
import { journalDatabaseHoldsAgentSessions } from '../../../native-chat/agent-session-journal/database/journal-database'
import { AgentSessionRecordStore } from './agent-session-record-store'
import type { JournalSqlDatabase } from '../../../native-chat/agent-session-journal/database/journal-sql-database'
+23
View File
@@ -43,6 +43,29 @@ export function isTransientSqliteContention(error: unknown): boolean {
return CONTENTION_MESSAGE.test(errorText(error))
}
const SQLITE_NOMEM = 7
const SQLITE_READONLY = 8
const SQLITE_IOERR = 10
const SQLITE_FULL = 13
const STORAGE_ERRCODES = new Set([
SQLITE_BUSY,
SQLITE_LOCKED,
SQLITE_NOMEM,
SQLITE_READONLY,
SQLITE_IOERR,
SQLITE_FULL,
SQLITE_CANTOPEN
])
/**
* The database or its disk failed, not the row being written: every other row would fail too.
* Corruption is the caller's call: it may be one row's page or the whole file.
*/
export function isSqliteStorageFailure(error: unknown): boolean {
const errcode = primaryErrcode(error)
return (errcode !== null && STORAGE_ERRCODES.has(errcode)) || isTransientSqliteContention(error)
}
export type SqliteReadFailureKind =
/** Another connection holds the lock; the same read can succeed later. */
| 'contended'
@@ -27,7 +27,7 @@ import { wslHostsGateOpen } from '../host-lifecycle/wsl-host-gate'
import { isWslDriveUncConversionComplete } from './wsl-drive-unc-conversion-renderer'
import { getHistoryRoot, listWslHistoryRoots } from '../../pty/history/terminal-history-paths'
import { orchestrationDbFile } from '../../runtime/orchestration/db/orchestration-db-file'
import { journalDomainDatabasePath } from '../../native-chat/agent-session-journal/database/journal-domain-database'
import { journalDomainDatabasePath } from '../../native-chat/agent-session-journal/database/journal-host-database'
import { rewriteWslDriveUncState } from './wsl-drive-unc-state-rewrite'
import { WSL_DRIVE_UNC_STATE_DISPOSITIONS } from '../../../shared/wsl-move/wsl-drive-unc-state-dispositions'
@@ -1,9 +1,7 @@
import path from 'node:path'
import { randomUUID } from 'node:crypto'
import {
journalDomainDatabasePath,
openJournalDomainDatabase
} from '../../../../src/main/native-chat/agent-session-journal/database/journal-domain-database'
import { openJournalDomainDatabase } from '../../../../src/main/native-chat/agent-session-journal/database/journal-domain-database'
import { journalDomainDatabasePath } from '../../../../src/main/native-chat/agent-session-journal/database/journal-host-database'
import { OrchestrationDb } from '../../../../src/main/runtime/orchestration/db/orchestration-db'
import {
agentSessionRecordFixture,