mirror of
https://github.com/stablyai/orca.git
synced 2026-11-19 04:23:57 +00:00
fix(native-chat): terminal restore no longer waits for chats an older Orca saved (#43527)
* fix(native-chat): terminal restore no longer waits for chats an older Orca saved The open that terminal restore and SSH reconnect wait for now only reads and writes per-chat records. A converted chat an older Orca continued gets its copy record at once; the copy's history converts from the original like any unconverted chat, paged on a reader. A converted chat with no saved revision is compared in the background, one chat per idle step. A chat that fails is recorded with its error and absorbed, so it no longer reruns the pass at every launch; storage failures pause the pass for the next launch instead. Tracked in #42783. * fix(native-chat): keep copies an older build readied early, and pending work across pauses A copy's mapping now clears when its history converts here, so a copy another build readied first is found and kept again from the original. Copy-due markers clear per chat, so a paused pass loses none. A pending copy is found by its original's id, however many times the original moved. Forced copies use a set lookup. * fix(native-chat): bound and absorb the remaining recovery work A copy whose history fails to convert is recorded and left to an open instead of every idle pass. The background compare restores at most 20,000 original rows per chat; a longer chat keeps its turns as a copy. Corruption fails only its own chat. Compare markers are written with the pass, and a later absorbed revision clears an earlier failure. * fix(native-chat): never mark a copy failed for a quit or a lost reader Only a failure of the original's own content records a copy as failed, and never while maintenance is stopping. Corruption after the original's revision read pauses the pass instead of absorbing chats, since it may be this file's. * fix(native-chat): compare a whole chat per idle step and absorb one that crashes its reader Per the council's ruling, the row budget is removed: it forced duplicate copies of long chats whose conversion then did the same full restore. A compare that takes its reader down is recorded and absorbed instead of retried every launch; stopping and contention still leave it for the next launch. * test: give the forgotten-chats maintenance fake the legacy compare step * fix(agent-session-journal): forgotten chats absorb their pending compare; this file's damage leaves a compare pending A chat forgotten while its older-Orca compare or copy waited takes that revision, so the next open never brings it back as a copy. Compare reads this file before the original, and only the original's failures settle as the chat's answer; this file's damage, contention and stopping stay pending. A reader exit gets one retry before it is absorbed, and a compare error no longer skips that launch's idle conversion. * fix(agent-session-journal): a damaged original settles its compare instead of staying pending * fix(agent-session-journal): a message an older Orca queued makes its compare keep a copy * chore: count the compare's card read and the stranded-copy point read in the sqlite baseline * test(agent-session-journal): the copy keeps the older Orca's queued message beside the chat's own * perf(agent-session-journal): a copy due on an unchanged original no longer reads or replays every chat * chore: prune the blocking baselines main's worker-only rules cleared; count the reconcile's split-out files * perf(agent-session-journal): keep the older-Orca reconcile on workers and off whole-profile reads The chat-store path moves beside the journal's own, so the main thread no longer reaches the reconcile chain. A pass over copies due reads only those chats' claims and looks tabs up one at a time, and a compare with no original reads nothing from this file. * perf(agent-session-journal): recording the tab index keeps this file's tabs as a set * perf(agent-session-journal): a missing original ends the compare pass instead of asking about every chat * docs(agent-session-journal): say what a pass over copies due still scans * fix(agent-session-journal): damage to one chat here leaves only its compare pending * docs(agent-session-journal): a pass over copies due scans tabs once per due chat * fix(agent-session-journal): a copy too large for one reader reply is recorded, not retried every launch * chore: drop the tab lookup's cast; prune the sync-fs rows the worker-only move cleared * fix(agent-session-journal): an original that won't open ends the compare pass; a stranded rollback stops the reconcile Drops the statement-count test: the repo doesn't test work counts. * fix(agent-session-journal): a forgotten copy drops its mapping and failure keys * test(e2e): take the chat-store path from journal-host-database * fix(agent-session-journal): name the older-Orca compare in reader crash diagnostics
This commit is contained in:
@@ -826,22 +826,13 @@
|
||||
4 src/main/native-chat/agent-session-attachments/agent-session-attachment-claims.ts
|
||||
1 src/main/native-chat/agent-session-journal/context-epochs/journal-context-cursor.ts
|
||||
12 src/main/native-chat/agent-session-journal/database/journal-database.ts
|
||||
13 src/main/native-chat/agent-session-journal/database/journal-domain-database.ts # store-open tail fold (one-shot startup); same SQL already in main-blocking-sqlite-baseline
|
||||
5 src/main/native-chat/agent-session-journal/database/journal-domain-migrations.ts # tail table create at store open (one-shot startup); worker-only: +2 create the retired-epoch pending table in a transaction on the journal SQL worker
|
||||
8 src/main/native-chat/agent-session-journal/database/journal-domain-profile-import.ts
|
||||
3 src/main/native-chat/agent-session-journal/database/journal-host-database.ts
|
||||
40 src/main/native-chat/agent-session-journal/database/journal-legacy-reconcile.ts
|
||||
1 src/main/native-chat/agent-session-journal/database/journal-legacy-source-receipt.ts
|
||||
4 src/main/native-chat/agent-session-journal/database/journal-legacy-source.ts
|
||||
1 src/main/native-chat/agent-session-journal/durable-rows/journal-row-table.ts
|
||||
2 src/main/native-chat/agent-session-journal/history-import/adopted-transcript-source.ts
|
||||
1 src/main/native-chat/agent-session-journal/history-import/journal-hermes-import.ts
|
||||
5 src/main/native-chat/agent-session-journal/history-import/journal-legacy-import.ts
|
||||
2 src/main/native-chat/agent-session-journal/input-order/journal-input-conversion.ts
|
||||
1 src/main/native-chat/agent-session-journal/journal-domain-import-discard.ts
|
||||
10 src/main/native-chat/agent-session-journal/journal-domain-import-rows.ts
|
||||
1 src/main/native-chat/agent-session-journal/journal-domain-import-submissions.ts
|
||||
7 src/main/native-chat/agent-session-journal/journal-domain-import.ts
|
||||
1 src/main/native-chat/agent-session-journal/journal-domain-input-eligibility.ts
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-item-display.ts
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-maintenance.ts # worker-only: the retired prune settles its pending entry on the journal SQL worker
|
||||
@@ -849,7 +840,6 @@
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-queue-pauses.ts
|
||||
6 src/main/native-chat/agent-session-journal/journal-domain-queue-reads.ts
|
||||
1 src/main/native-chat/agent-session-journal/journal-domain-schema.ts
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-stop-notes.ts
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-submission-consume.ts
|
||||
1 src/main/native-chat/agent-session-journal/journal-forgotten-chats.ts
|
||||
11 src/main/native-chat/agent-session-journal/journal-sql-queue-read.ts
|
||||
@@ -860,12 +850,7 @@
|
||||
5 src/main/native-chat/agent-session-journal/journal-stored-output-rows.ts # streamed-text tail (#43716): same SQL already in main-blocking-sqlite-baseline; journal SQL path, not a timer or handler
|
||||
10 src/main/native-chat/agent-session-journal/journal-text-tail.ts # streamed-text tail (#43716): same SQL already in main-blocking-sqlite-baseline; journal SQL path, not a timer or handler
|
||||
2 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-format.ts
|
||||
2 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-table.ts
|
||||
1 src/main/native-chat/agent-session-journal/queued-messages/journal-queue-import.ts
|
||||
1 src/main/native-chat/agent-session-journal/replay/journal-historical-inputs.ts
|
||||
13 src/main/native-chat/agent-session-journal/replay/journal-open.ts
|
||||
4 src/main/native-chat/agent-session-journal/replay/journal-read-row.ts
|
||||
2 src/main/native-chat/agent-session-journal/replay/journal-retained-inputs.ts
|
||||
3 src/main/native-chat/agent-session-journal/replay/journal-stored-cursor.ts
|
||||
3 src/main/native-chat/agent-session-journal/writing/command-receipt-table.ts
|
||||
1 src/main/native-chat/agent-session-journal/writing/command-receipt-transaction.ts
|
||||
@@ -1212,8 +1197,6 @@
|
||||
1 src/main/runtime/agent-session/model-catalog/structured-agent-model-catalog-discovery.ts
|
||||
1 src/main/runtime/agent-session/ownership/agent-session-claim-identity.ts
|
||||
4 src/main/runtime/agent-session/ownership/agent-session-process-identity-probe.ts
|
||||
5 src/main/runtime/agent-session/records/agent-session-background-meta.ts
|
||||
5 src/main/runtime/agent-session/records/agent-session-record-rows.ts
|
||||
1 src/main/runtime/agent-session/records/agent-session-record-store-slot.ts
|
||||
8 src/main/runtime/agent-session/records/agent-session-runtime-end-record.ts
|
||||
2 src/main/runtime/agent-session/records/agent-session-store-transactions.ts
|
||||
|
||||
@@ -29,42 +29,28 @@
|
||||
19 src/main/native-chat/agent-session-attachments/agent-session-attachment-claims.ts
|
||||
5 src/main/native-chat/agent-session-journal/database/journal-checkpoint-owner.ts
|
||||
22 src/main/native-chat/agent-session-journal/database/journal-database.ts
|
||||
14 src/main/native-chat/agent-session-journal/database/journal-domain-database.ts
|
||||
8 src/main/native-chat/agent-session-journal/database/journal-domain-migrations.ts # worker-only: +1 is the reply outbox table migration, run on the journal SQL worker
|
||||
22 src/main/native-chat/agent-session-journal/database/journal-domain-profile-import.ts
|
||||
1 src/main/native-chat/agent-session-journal/database/journal-host-database.ts
|
||||
32 src/main/native-chat/agent-session-journal/database/journal-legacy-reconcile.ts
|
||||
18 src/main/native-chat/agent-session-journal/database/journal-legacy-source.ts
|
||||
5 src/main/native-chat/agent-session-journal/database/journal-retire-pending.ts # worker-only: retired-epoch pending table, triggers and settle, run on the journal SQL worker
|
||||
5 src/main/native-chat/agent-session-journal/database/journal-text-tail-schema.ts
|
||||
12 src/main/native-chat/agent-session-journal/durable-rows/journal-row-table.ts
|
||||
18 src/main/native-chat/agent-session-journal/history-import/journal-pi-active-branch.ts
|
||||
6 src/main/native-chat/agent-session-journal/journal-domain-import-discard.ts
|
||||
24 src/main/native-chat/agent-session-journal/journal-domain-import-rows.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-domain-import-submissions.ts
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-import.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-domain-item-display.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-domain-item-names.ts
|
||||
12 src/main/native-chat/agent-session-journal/journal-domain-maintenance.ts
|
||||
10 src/main/native-chat/agent-session-journal/journal-domain-queue-failures.ts
|
||||
6 src/main/native-chat/agent-session-journal/journal-domain-queue-pauses.ts
|
||||
8 src/main/native-chat/agent-session-journal/journal-domain-queue-reads.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-domain-queue-revision.ts
|
||||
10 src/main/native-chat/agent-session-journal/journal-domain-queue-rows.ts
|
||||
6 src/main/native-chat/agent-session-journal/journal-domain-stop-notes.ts
|
||||
2 src/main/native-chat/agent-session-journal/journal-domain-submission-consume.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-domain-submission-records.ts
|
||||
20 src/main/native-chat/agent-session-journal/journal-forgotten-chats.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-sql-accepted-inputs.ts
|
||||
4 src/main/native-chat/agent-session-journal/journal-stored-output-rows.ts
|
||||
24 src/main/native-chat/agent-session-journal/journal-text-tail.ts
|
||||
15 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-format.ts
|
||||
6 src/main/native-chat/agent-session-journal/outcomes/journal-outcome-table.ts
|
||||
2 src/main/native-chat/agent-session-journal/queued-messages/journal-queue-import.ts
|
||||
5 src/main/native-chat/agent-session-journal/queued-messages/queued-message-schema.ts
|
||||
10 src/main/native-chat/agent-session-journal/replay/journal-historical-inputs.ts
|
||||
4 src/main/native-chat/agent-session-journal/replay/journal-read-row.ts
|
||||
8 src/main/native-chat/agent-session-journal/replay/journal-retained-inputs.ts
|
||||
1 src/main/native-chat/agent-session-journal/writing/command-receipt-schema.ts
|
||||
8 src/main/native-chat/agent-session-journal/writing/command-receipt-table.ts
|
||||
2 src/main/native-chat/agent-session-journal/writing/journal-goal-attachment-inverse.ts
|
||||
@@ -95,9 +81,7 @@
|
||||
4 src/main/persistence/profile-state/legacy-json/profile-state-json-acceptance.ts
|
||||
2 src/main/persistence/profile-state/recovery/profile-state-section-repair.ts
|
||||
4 src/main/rate-limits/providers/opencode-go/opencode-go-api-key-source.ts
|
||||
4 src/main/runtime/agent-session/records/agent-session-background-meta.ts
|
||||
2 src/main/runtime/agent-session/records/agent-session-record-lineage.ts
|
||||
38 src/main/runtime/agent-session/records/agent-session-record-rows.ts
|
||||
23 src/main/runtime/orchestration/db/dispatch-context/presentation-completion-source.ts
|
||||
6 src/main/runtime/orchestration/db/dispatch-row-writer.ts
|
||||
1 src/main/runtime/orchestration/db/federation/federated-stub-home-run-backfill.ts
|
||||
|
||||
@@ -316,9 +316,7 @@
|
||||
1 src/main/muse/hook-config-json.ts
|
||||
4 src/main/muse/hook-service.ts
|
||||
2 src/main/native-chat/agent-session-attachments/agent-session-attachment-claims.ts
|
||||
8 src/main/native-chat/agent-session-journal/database/journal-domain-database.ts
|
||||
1 src/main/native-chat/agent-session-journal/database/journal-host-database.ts
|
||||
3 src/main/native-chat/agent-session-journal/database/journal-legacy-source-receipt.ts
|
||||
1 src/main/native-chat/transcript-discovery/host-readable-transcript-path.ts
|
||||
2 src/main/native-chat/transcript-filesystem/process/wsl-transcript-fs-process-spawn.ts
|
||||
2 src/main/node-runtime/managed-runtime-release-install.ts
|
||||
|
||||
@@ -21,10 +21,9 @@ import {
|
||||
createAgentSessionRecordTablesSql
|
||||
} from './journal-database-schema'
|
||||
import { openJournalDatabase } from './journal-database'
|
||||
import { journalDatabasePath } from './journal-host-database'
|
||||
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
|
||||
import {
|
||||
initializeJournalDomainDatabase,
|
||||
journalDomainDatabasePath,
|
||||
openJournalDomainDatabase
|
||||
} from './journal-domain-database'
|
||||
import { readMailWithdrawal } from '../writing/mail-withdrawal-receipts'
|
||||
|
||||
@@ -9,7 +9,7 @@ import {
|
||||
journalPragmaNumber,
|
||||
type OpenJournalDatabase
|
||||
} from './journal-database'
|
||||
import { journalDatabasePath } from './journal-host-database'
|
||||
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
|
||||
import { tryFoldJournalTextTails } from '../journal-text-tail'
|
||||
import {
|
||||
JOURNAL_DOMAIN_SCHEMA_VERSION,
|
||||
@@ -26,10 +26,6 @@ import {
|
||||
JOURNAL_LEGACY_SOURCE_RECEIPT
|
||||
} from './journal-legacy-source-receipt'
|
||||
|
||||
export function journalDomainDatabasePath(stateDirectory: string): string {
|
||||
return join(stateDirectory, 'agent-session-domain.db')
|
||||
}
|
||||
|
||||
/** Run once on the writer worker, before opening its steady-state writable connection. */
|
||||
export async function initializeJournalDomainDatabase(stateDirectory: string): Promise<string> {
|
||||
const destination = journalDomainDatabasePath(stateDirectory)
|
||||
|
||||
@@ -19,6 +19,10 @@ export function journalDatabasePath(stateDirectory: string): string {
|
||||
return join(stateDirectory, JOURNAL_DATABASE_FILE)
|
||||
}
|
||||
|
||||
export function journalDomainDatabasePath(stateDirectory: string): string {
|
||||
return join(stateDirectory, 'agent-session-domain.db')
|
||||
}
|
||||
|
||||
/** What journal collaborators need from a connection owner; the SQL writer supplies its own. */
|
||||
export type JournalTransactionalDatabase = Pick<
|
||||
JournalHostDatabase,
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
import type Database from '../../../sqlite/sync-database'
|
||||
import type { SqliteRow } from '../../../sqlite/sqlite-statement'
|
||||
import { AGENT_SESSION_TABS_RECORDED } from '../../../runtime/agent-session/records/agent-session-record-rows'
|
||||
import { readMeta } from './journal-legacy-reconcile-marks'
|
||||
|
||||
export type Claim = SqliteRow
|
||||
|
||||
/** A Set over a walk's rows, or point reads over this file's own. */
|
||||
type Lookup = {
|
||||
has(value: string): boolean
|
||||
add(value: string): unknown
|
||||
delete(value: string): unknown
|
||||
}
|
||||
|
||||
/** The original's tabs and claims, and this file's tabs, read once per walk. */
|
||||
export type Adoption = {
|
||||
savedTabs: Map<string, string>
|
||||
claims: Map<string, Claim[]>
|
||||
tabbed: Lookup
|
||||
takenTabs: Lookup
|
||||
nextPosition: number
|
||||
tabsRecorded: boolean
|
||||
/** Tabs this pass added, so a chat that fails can take back its own. */
|
||||
added: [sessionId: string, tab: string][]
|
||||
}
|
||||
|
||||
/**
|
||||
* A pass over chats due a copy reads only those chats' claims and looks this file's tabs up one at a
|
||||
* time; a copy never takes the original's saved tab. Neither table is indexed by chat, so that pass
|
||||
* still scans the claims once and this file's tabs once per due chat, keeping only their rows.
|
||||
*/
|
||||
export function readAdoption(
|
||||
source: Database.Database,
|
||||
db: Database.Database,
|
||||
only: readonly string[] | null
|
||||
): Adoption {
|
||||
const claims = new Map<string, Claim[]>()
|
||||
for (const claim of source
|
||||
.prepare(`SELECT upload_id,session_id,claimed_at FROM agent_session_attachment_claims
|
||||
WHERE ?1 IS NULL OR session_id IN (SELECT value FROM json_each(?1))`)
|
||||
.all(only && JSON.stringify(only))) {
|
||||
const id = String(claim.session_id)
|
||||
const list = claims.get(id) ?? []
|
||||
list.push(claim)
|
||||
claims.set(id, list)
|
||||
}
|
||||
const tail = db.prepare('SELECT COALESCE(MAX(position),-1) AS tail FROM agent_session_tabs').get()
|
||||
const tabsRecorded = readMeta(db, AGENT_SESSION_TABS_RECORDED) !== undefined
|
||||
// Recording the tab index adds one tab per record, so it takes this file's tabs as a set.
|
||||
const pointed = only !== null && tabsRecorded
|
||||
const tabs = pointed ? [] : db.prepare('SELECT tab_id,session_id FROM agent_session_tabs').all()
|
||||
return {
|
||||
savedTabs: new Map(
|
||||
only
|
||||
? []
|
||||
: source
|
||||
.prepare('SELECT tab_id,session_id FROM agent_session_tabs')
|
||||
.all()
|
||||
.map((row) => [String(row.session_id), String(row.tab_id)])
|
||||
),
|
||||
claims,
|
||||
tabbed: pointed
|
||||
? tabLookup(db, 'session_id')
|
||||
: new Set(tabs.map((row) => String(row.session_id))),
|
||||
takenTabs: pointed ? tabLookup(db, 'tab_id') : new Set(tabs.map((row) => String(row.tab_id))),
|
||||
nextPosition: Number(tail?.tail ?? -1) + 1,
|
||||
tabsRecorded,
|
||||
added: []
|
||||
}
|
||||
}
|
||||
|
||||
/** This file's own rows answer, so a tab added or rolled back needs no bookkeeping here. */
|
||||
function tabLookup(db: Database.Database, column: 'session_id' | 'tab_id'): Lookup {
|
||||
const query = db.prepare(`SELECT 1 FROM agent_session_tabs WHERE ${column}=?`)
|
||||
return {
|
||||
has: (value) => query.get(value) !== undefined,
|
||||
add: () => undefined,
|
||||
delete: () => undefined
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
import type Database from '../../../sqlite/sync-database'
|
||||
import { isSqliteCorruption, isSqliteStorageFailure } from '../../../sqlite/sqlite-read-failure'
|
||||
import { isUnsettledQueuedMessage } from '../queued-messages/queued-message-table'
|
||||
import { prepareJournalDomainImport, type JournalDomainImportPage } from '../journal-domain-import'
|
||||
import {
|
||||
journalLegacyChatRevision,
|
||||
journalLegacyFailureKey,
|
||||
recordJournalLegacyChatRevision,
|
||||
settleJournalLegacyCopy
|
||||
} from './journal-legacy-source'
|
||||
import {
|
||||
deleteMeta,
|
||||
JOURNAL_LEGACY_COMPARE,
|
||||
JOURNAL_LEGACY_COPY_DUE,
|
||||
prefixEnd,
|
||||
readMeta,
|
||||
recordJournalLegacyFailure,
|
||||
writeMeta
|
||||
} from './journal-legacy-reconcile-marks'
|
||||
|
||||
export type JournalLegacyCompare = { sessionId: string; revision: string }
|
||||
|
||||
/** Converted chats awaiting a background compare, in pages after `after`. */
|
||||
export function pendingJournalLegacyCompares(
|
||||
db: Database.Database,
|
||||
after: string | null
|
||||
): JournalLegacyCompare[] {
|
||||
return db
|
||||
.prepare(`SELECT key,value FROM agent_session_store_meta WHERE key>? AND key<?
|
||||
ORDER BY key LIMIT 16`)
|
||||
.all(`${JOURNAL_LEGACY_COMPARE}${after ?? ''}`, prefixEnd(JOURNAL_LEGACY_COMPARE))
|
||||
.map((row) => ({
|
||||
sessionId: String(row.key).slice(JOURNAL_LEGACY_COMPARE.length),
|
||||
revision: String(row.value)
|
||||
}))
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether this file already holds every item of the original chat. Runs on a reader, one chat per
|
||||
* idle step, and stops at the first item this chat never held. Its unit is the chat's restore,
|
||||
* the same one converting that chat takes; an interrupted compare restarts that chat. Only the
|
||||
* original's failures are the chat's answer; this file's own failures throw and stay pending.
|
||||
*/
|
||||
export function compareJournalLegacyChat(
|
||||
openSource: () => Database.Database | undefined,
|
||||
db: Database.Database,
|
||||
sessionId: string
|
||||
): { revision: string; held: boolean } | { error: string } | null {
|
||||
let source: Database.Database | undefined
|
||||
try {
|
||||
source = openSource()
|
||||
} catch (error) {
|
||||
// An original that won't open is no chat's answer; it ends the pass until it opens again.
|
||||
if (isSqliteCorruption(error)) {
|
||||
return null
|
||||
}
|
||||
return originalFailure(error)
|
||||
}
|
||||
// With no original there is nothing to compare, so this file isn't read.
|
||||
if (!source) {
|
||||
return null
|
||||
}
|
||||
const held = { items: heldItems(db, sessionId), cards: heldCards(db, sessionId) }
|
||||
try {
|
||||
const imported = prepareJournalDomainImport(source, sessionId)
|
||||
if (!imported) {
|
||||
return { revision: journalLegacyChatRevision(source, sessionId) ?? 'null', held: true }
|
||||
}
|
||||
return {
|
||||
revision: imported.sourceRevision ?? 'null',
|
||||
held: heldEveryItem(held, imported.pages)
|
||||
}
|
||||
} catch (error) {
|
||||
return originalFailure(error)
|
||||
}
|
||||
}
|
||||
|
||||
function originalFailure(error: unknown): { error: string } {
|
||||
if (isSqliteStorageFailure(error)) {
|
||||
throw error
|
||||
}
|
||||
return { error: error instanceof Error ? error.message : String(error) }
|
||||
}
|
||||
|
||||
/** Ignored when a later open marked a newer revision; a failed compare is absorbed like a chat. */
|
||||
export function settleJournalLegacyCompare(
|
||||
db: Database.Database,
|
||||
compare: JournalLegacyCompare & ({ held: boolean } | { error: string })
|
||||
): void {
|
||||
const key = `${JOURNAL_LEGACY_COMPARE}${compare.sessionId}`
|
||||
if (readMeta(db, key) !== compare.revision) {
|
||||
return
|
||||
}
|
||||
deleteMeta(db, key)
|
||||
if ('error' in compare) {
|
||||
recordJournalLegacyFailure(db, {
|
||||
sessionId: compare.sessionId,
|
||||
revision: compare.revision,
|
||||
error: new Error(compare.error)
|
||||
})
|
||||
} else if (compare.held) {
|
||||
recordJournalLegacyChatRevision(db, compare.sessionId, compare.revision)
|
||||
} else {
|
||||
writeMeta(db, `${JOURNAL_LEGACY_COPY_DUE}${compare.sessionId}`, compare.revision)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Removing an item here leaves its tombstone, so every item this chat converted stays held, at
|
||||
* its converted revision or a later one.
|
||||
*/
|
||||
function heldItems(db: Database.Database, id: string): Map<string, number> {
|
||||
return new Map(
|
||||
db
|
||||
.prepare(`SELECT item_id AS id, MAX(revision) AS revision FROM (
|
||||
SELECT item_id, revision FROM journal_items WHERE session_id=?1
|
||||
UNION ALL SELECT item_id, ?2 FROM journal_item_tombstones WHERE session_id=?1)
|
||||
GROUP BY item_id`)
|
||||
.all(id, Number.MAX_SAFE_INTEGER)
|
||||
.map((row) => [String(row.id), Number(row.revision)])
|
||||
)
|
||||
}
|
||||
|
||||
/** Every card this chat ever queued, whatever became of it. */
|
||||
function heldCards(db: Database.Database, id: string): Set<string> {
|
||||
return new Set(
|
||||
db
|
||||
.prepare('SELECT DISTINCT message_id AS id FROM journal_queue_cards WHERE session_id=?')
|
||||
.all(id)
|
||||
.map((row) => String(row.id))
|
||||
)
|
||||
}
|
||||
|
||||
/** A message the older Orca queued and never sent has no item, only its card. */
|
||||
function heldEveryItem(
|
||||
held: { items: Map<string, number>; cards: Set<string> },
|
||||
pages: Iterable<JournalDomainImportPage>
|
||||
): boolean {
|
||||
for (const page of pages) {
|
||||
if (
|
||||
page.kind === 'items' &&
|
||||
!page.rows.every(({ item }) => (held.items.get(item.itemId) ?? -1) >= item.revision)
|
||||
) {
|
||||
return false
|
||||
}
|
||||
if (
|
||||
page.kind === 'cards' &&
|
||||
page.rows.some((card) => isUnsettledQueuedMessage(card) && !held.cards.has(card.messageId))
|
||||
) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
/**
|
||||
* A forgotten chat takes the revision its compare or copy waited on, so an older Orca's unchanged
|
||||
* copy never brings it back; a forgotten copy drops its mapping, so no key outlives it.
|
||||
*/
|
||||
export function absorbForgottenJournalLegacyChat(db: Database.Database, sessionId: string): void {
|
||||
for (const prefix of [JOURNAL_LEGACY_COMPARE, JOURNAL_LEGACY_COPY_DUE]) {
|
||||
const revision = readMeta(db, `${prefix}${sessionId}`)
|
||||
if (revision !== undefined) {
|
||||
deleteMeta(db, `${prefix}${sessionId}`)
|
||||
recordJournalLegacyChatRevision(db, sessionId, revision)
|
||||
}
|
||||
}
|
||||
settleJournalLegacyCopy(db, sessionId)
|
||||
deleteMeta(db, journalLegacyFailureKey(sessionId))
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
import type Database from '../../../sqlite/sync-database'
|
||||
import { runJournalTransaction } from './journal-database'
|
||||
import {
|
||||
journalLegacyCopySourceKey,
|
||||
journalLegacyFailureKey,
|
||||
journalLegacyPendingCopyKey,
|
||||
recordJournalLegacyChatRevision,
|
||||
settleJournalLegacyCopy
|
||||
} from './journal-legacy-source'
|
||||
import {
|
||||
continuedCopyId,
|
||||
deleteMeta,
|
||||
holdsPendingCopy,
|
||||
JOURNAL_LEGACY_COPY_DUE,
|
||||
revisionHoldsHistory,
|
||||
writeMeta
|
||||
} from './journal-legacy-reconcile-marks'
|
||||
|
||||
/**
|
||||
* A background compare found turns this file lacks, or a copy was `stranded`: readied by another
|
||||
* build without the original's history.
|
||||
*/
|
||||
export type JournalLegacyForcedCopy = { stranded?: string }
|
||||
|
||||
/** Adds the original's record as this id, with its tab, catalog row and claims. */
|
||||
type Adopt = (as: string) => void
|
||||
|
||||
/**
|
||||
* The original's version of a converted chat it moved past becomes a second chat beside it. Only
|
||||
* its record is made here; its history converts from the original chat when opened or idle. With
|
||||
* no revision to compare (files copied before revisions were kept), a background compare decides.
|
||||
*/
|
||||
export function keepContinuedChat(
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
revision: string,
|
||||
shown: string | undefined,
|
||||
adopt: Adopt,
|
||||
pass: { baselines: [string, string][]; compares: [string, string][] }
|
||||
): void {
|
||||
if (shown === undefined && revisionHoldsHistory(revision)) {
|
||||
pass.compares.push([id, revision])
|
||||
return
|
||||
}
|
||||
if (holdsPendingCopy(db, id)) {
|
||||
pass.baselines.push([id, revision])
|
||||
return
|
||||
}
|
||||
runJournalTransaction(db, () => keepCopy(db, id, revision, adopt))
|
||||
}
|
||||
|
||||
/** Background conversion skips a copy whose history failed to convert; opening it still retries. */
|
||||
export function recordJournalLegacyCopyFailure(
|
||||
db: Database.Database,
|
||||
copyId: string,
|
||||
error: string
|
||||
): void {
|
||||
if (
|
||||
db
|
||||
.prepare('SELECT 1 FROM agent_session_store_meta WHERE key=?')
|
||||
.get(journalLegacyCopySourceKey(copyId))
|
||||
) {
|
||||
writeMeta(db, journalLegacyFailureKey(copyId), JSON.stringify({ revision: null, error }))
|
||||
}
|
||||
}
|
||||
|
||||
/** In the caller's transaction. */
|
||||
export function keepForcedCopy(
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
revision: string,
|
||||
forced: JournalLegacyForcedCopy,
|
||||
adopt: Adopt
|
||||
): void {
|
||||
deleteMeta(db, `${JOURNAL_LEGACY_COPY_DUE}${id}`)
|
||||
if (forced.stranded === undefined) {
|
||||
keepCopy(db, id, revision, adopt)
|
||||
return
|
||||
}
|
||||
settleJournalLegacyCopy(db, forced.stranded)
|
||||
keepCopy(db, id, revision, adopt, continuedCopyId(id, forced.stranded))
|
||||
}
|
||||
|
||||
function keepCopy(
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
revision: string,
|
||||
adopt: Adopt,
|
||||
copy = continuedCopyId(id, revision)
|
||||
): void {
|
||||
if (!revisionHoldsHistory(revision)) {
|
||||
recordJournalLegacyChatRevision(db, id, revision)
|
||||
return
|
||||
}
|
||||
adopt(copy)
|
||||
writeMeta(db, journalLegacyCopySourceKey(copy), id)
|
||||
writeMeta(db, journalLegacyPendingCopyKey(id), copy)
|
||||
recordJournalLegacyChatRevision(db, id, revision)
|
||||
}
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import type Database from '../../../sqlite/sync-database'
|
||||
import {
|
||||
journalLegacyCopySourceKey,
|
||||
journalLegacyFailureKey,
|
||||
journalLegacyPendingCopyKey,
|
||||
recordJournalLegacyChatRevision
|
||||
} from './journal-legacy-source'
|
||||
|
||||
/** A converted chat with no revision to compare whose original moved; compared in the background. */
|
||||
export const JOURNAL_LEGACY_COMPARE = 'journal.legacy-compare:'
|
||||
/** Compared and found to hold turns this file lacks; the next open keeps them as a copy. */
|
||||
export const JOURNAL_LEGACY_COPY_DUE = 'journal.legacy-copy-due:'
|
||||
|
||||
export type JournalLegacyFailure = {
|
||||
sessionId: string
|
||||
revision: string | undefined
|
||||
error: unknown
|
||||
}
|
||||
|
||||
/** Absorbed with its error: retried only when the older Orca writes this chat again. */
|
||||
export function recordJournalLegacyFailure(
|
||||
db: Database.Database,
|
||||
failure: JournalLegacyFailure
|
||||
): void {
|
||||
console.warn('[agent-session-journal] keeping an older Orca chat failed', failure.error)
|
||||
if (failure.revision !== undefined) {
|
||||
recordJournalLegacyChatRevision(db, failure.sessionId, failure.revision)
|
||||
}
|
||||
writeMeta(
|
||||
db,
|
||||
journalLegacyFailureKey(failure.sessionId),
|
||||
JSON.stringify({ revision: failure.revision ?? null, error: failureMessage(failure.error) })
|
||||
)
|
||||
deleteMeta(db, `${JOURNAL_LEGACY_COPY_DUE}${failure.sessionId}`)
|
||||
}
|
||||
|
||||
export function failureMessage(error: unknown): string {
|
||||
return (error instanceof Error ? error.message : String(error)).slice(0, 500)
|
||||
}
|
||||
|
||||
export function readMeta(db: Database.Database, key: string): string | undefined {
|
||||
const row = db.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)
|
||||
return row === undefined ? undefined : String(row.value)
|
||||
}
|
||||
|
||||
export function metaKeys(db: Database.Database, prefix: string): string[] {
|
||||
return db
|
||||
.prepare('SELECT key FROM agent_session_store_meta WHERE key>? AND key<? ORDER BY key')
|
||||
.all(prefix, prefixEnd(prefix))
|
||||
.map((row) => String(row.key).slice(prefix.length))
|
||||
}
|
||||
|
||||
/** Keys here are ASCII and every prefix ends in ':', so ';' bounds its range. */
|
||||
export function prefixEnd(prefix: string): string {
|
||||
return `${prefix.slice(0, -1)};`
|
||||
}
|
||||
|
||||
export function deleteMeta(db: Database.Database, key: string): void {
|
||||
db.prepare('DELETE FROM agent_session_store_meta WHERE key=?').run(key)
|
||||
}
|
||||
|
||||
export function writeMeta(db: Database.Database, key: string, value: string): void {
|
||||
db.prepare('INSERT OR REPLACE INTO agent_session_store_meta (key, value) VALUES (?, ?)').run(
|
||||
key,
|
||||
value
|
||||
)
|
||||
}
|
||||
|
||||
/** Derived from the revision, so a pass interrupted after making it finds the same chat. */
|
||||
export function continuedCopyId(id: string, revision: string): string {
|
||||
return createHash('sha256').update(`${id}\0${revision}`).digest('hex').slice(0, 32)
|
||||
}
|
||||
|
||||
/** A copy of this original chat still waiting to convert reads the current original. */
|
||||
export function holdsPendingCopy(db: Database.Database, id: string): boolean {
|
||||
const copy = readMeta(db, journalLegacyPendingCopyKey(id))
|
||||
return (
|
||||
copy !== undefined &&
|
||||
db
|
||||
.prepare(`SELECT 1 FROM agent_session_records WHERE session_id=?1 AND NOT EXISTS (
|
||||
SELECT 1 FROM journal_chat_state WHERE session_id=?1 AND epoch IS NOT NULL)`)
|
||||
.get(copy) !== undefined
|
||||
)
|
||||
}
|
||||
|
||||
const COPY_SOURCE = journalLegacyCopySourceKey('')
|
||||
|
||||
/**
|
||||
* Copies another build readied before this one converted them: an older build that can't read
|
||||
* the mapping founds them empty. Conversion here clears the mapping, so only those remain.
|
||||
*/
|
||||
export function strandedCopies(db: Database.Database): [original: string, copy: string][] {
|
||||
const mapped = db
|
||||
.prepare('SELECT key, value FROM agent_session_store_meta WHERE key>? AND key<?')
|
||||
.all(COPY_SOURCE, prefixEnd(COPY_SOURCE))
|
||||
if (mapped.length === 0) {
|
||||
return []
|
||||
}
|
||||
const founded = db.prepare(
|
||||
'SELECT 1 FROM journal_chat_state WHERE session_id=? AND epoch IS NOT NULL'
|
||||
)
|
||||
return mapped
|
||||
.map((row): [string, string] => [String(row.value), String(row.key).slice(COPY_SOURCE.length)])
|
||||
.filter(([, copy]) => founded.get(copy) !== undefined)
|
||||
}
|
||||
|
||||
/** An original chat whose current epoch saved no row has nothing to keep. */
|
||||
export function revisionHoldsHistory(revision: string): boolean {
|
||||
const parsed: unknown = JSON.parse(revision)
|
||||
return Array.isArray(parsed) && parsed[1] !== 0
|
||||
}
|
||||
+197
-16
@@ -7,10 +7,9 @@ import type Database from '../../../sqlite/sync-database'
|
||||
import { openJournalDatabase, runJournalTransaction } from './journal-database'
|
||||
import {
|
||||
initializeJournalDomainDatabase,
|
||||
journalDomainDatabasePath,
|
||||
openJournalDomainDatabase
|
||||
} from './journal-domain-database'
|
||||
import { journalDatabasePath } from './journal-host-database'
|
||||
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
|
||||
import { reconcileJournalLegacySource } from './journal-legacy-reconcile'
|
||||
import { insertJournalRow, publishJournalSessionEpoch } from '../durable-rows/journal-row-table'
|
||||
import { journalSqlRetryRows } from '../journal-sql-retry-rows.test-fixture'
|
||||
@@ -19,6 +18,8 @@ import { JournalSqlConversions } from '../journal-sql-conversions'
|
||||
import { journalSqlReaderPath } from '../journal-sql-worker-bundle.test-fixture'
|
||||
import { applyJournalSqlMaintenance } from '../journal-sql-maintenance'
|
||||
import type { JournalRow } from '../durable-rows/journal-row-schema'
|
||||
import { AGENT_SESSION_JOURNAL_QUEUE_SCHEMA_VERSION } from '../../../../shared/agent-session/journal/agent-session-journal-types'
|
||||
import { readJournalSqlDatabase } from '../journal-sql-database-read'
|
||||
|
||||
const cleanup: (() => Promise<void> | void)[] = []
|
||||
afterEach(async () => {
|
||||
@@ -73,6 +74,16 @@ async function upgrade(root: string) {
|
||||
return {
|
||||
db,
|
||||
convert: (sessionId: string) => conversions.ensure(sessionId),
|
||||
/** One background pass over converted chats awaiting a compare, as maintenance runs it. */
|
||||
compare: async () => {
|
||||
const page = await pool.database({ kind: 'legacy-compares', after: null })
|
||||
for (const compare of page.kind === 'legacy-compares' ? page.compares : []) {
|
||||
const result = await pool.compareLegacyChat(compare.sessionId)
|
||||
runJournalTransaction(db, () =>
|
||||
applyJournalSqlMaintenance(db, { kind: 'legacy-compare-settle', ...compare, ...result! })
|
||||
)
|
||||
}
|
||||
},
|
||||
close: async () => {
|
||||
await conversions.close()
|
||||
await pool.close()
|
||||
@@ -126,17 +137,6 @@ it('keeps every chat and turn an older Orca saved after a downgrade, without cha
|
||||
.update(readFileSync(journalDatabasePath(root)))
|
||||
.digest('hex')
|
||||
|
||||
// Upgrade again; one chat fails to join the first time and joins on the next launch.
|
||||
const blocked = openJournalDomainDatabase(journalDomainDatabasePath(root)).db
|
||||
blocked.exec(`CREATE TRIGGER blocked BEFORE INSERT ON agent_session_records
|
||||
WHEN NEW.session_id='started-older' BEGIN SELECT RAISE(ABORT,'disk full'); END`)
|
||||
blocked.close()
|
||||
const failing = openJournalDomainDatabase(await initializeJournalDomainDatabase(root)).db
|
||||
expect(
|
||||
failing.prepare("SELECT 1 FROM agent_session_records WHERE session_id='started-older'").get()
|
||||
).toBeUndefined()
|
||||
failing.exec('DROP TRIGGER blocked')
|
||||
failing.close()
|
||||
const second = await upgrade(root)
|
||||
cleanup.push(second.close)
|
||||
const { db } = second
|
||||
@@ -146,6 +146,9 @@ it('keeps every chat and turn an older Orca saved after a downgrade, without cha
|
||||
WHERE session_id NOT IN ('converted','unopened','started-older','closed')`)
|
||||
.all()
|
||||
.map((row) => String(row.session_id))
|
||||
// Terminal restore waits for the open, so the copy's history converts later, from the original.
|
||||
expect(items(db, copy)).toEqual([])
|
||||
expect(await second.convert(copy)).toBe('epoch')
|
||||
expect(items(db, copy)).toEqual([...converted, 'older-turn'])
|
||||
expect(
|
||||
JSON.parse(
|
||||
@@ -215,6 +218,18 @@ it('fills in revisions for chats converted before they were kept, then still kee
|
||||
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
|
||||
})
|
||||
original.pragma('wal_checkpoint(TRUNCATE)')
|
||||
await (await upgrade(root)).close()
|
||||
|
||||
// Written twice more before the copy converted: that copy reads the current original, so no second.
|
||||
for (const [index, itemId] of ['older-turn-2', 'older-turn-3'].entries()) {
|
||||
runJournalTransaction(original, () => {
|
||||
insertJournalRow(original, 'converted', turn(itemId, tip + 2 + index))
|
||||
})
|
||||
original.pragma('wal_checkpoint(TRUNCATE)')
|
||||
if (index === 0) {
|
||||
await (await upgrade(root)).close()
|
||||
}
|
||||
}
|
||||
const later = await upgrade(root)
|
||||
cleanup.push(later.close)
|
||||
const copies = later.db
|
||||
@@ -222,7 +237,10 @@ it('fills in revisions for chats converted before they were kept, then still kee
|
||||
.all()
|
||||
.map((row) => String(row.session_id))
|
||||
expect(copies).toHaveLength(1)
|
||||
expect(items(later.db, copies[0]!)).toContain('older-turn')
|
||||
expect(await later.convert(copies[0]!)).toBe('epoch')
|
||||
expect(items(later.db, copies[0]!)).toEqual(
|
||||
expect.arrayContaining(['older-turn', 'older-turn-2', 'older-turn-3'])
|
||||
)
|
||||
expect(
|
||||
later.db
|
||||
.prepare("SELECT value FROM agent_session_store_meta WHERE key='session_tabs_recorded'")
|
||||
@@ -302,9 +320,16 @@ it('keeps only what an older Orca wrote after a file from before revisions were
|
||||
// The older Orca continues one converted chat and starts another.
|
||||
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
|
||||
olderOrcaChat(original, 'started-older', [rows[0], turn('started-turn', 2)])
|
||||
const known = ['closed', 'converted', 'idle', 'open', 'started-older']
|
||||
const compared = await upgrade(root)
|
||||
// With no revision to compare, the open reads no history; a background pass compares it.
|
||||
expect(
|
||||
compared.db.prepare('SELECT COUNT(*) AS records FROM agent_session_records').get()
|
||||
).toEqual({ records: known.length })
|
||||
await compared.compare()
|
||||
await compared.close()
|
||||
const later = await upgrade(root)
|
||||
cleanup.push(later.close)
|
||||
const known = ['closed', 'converted', 'idle', 'open', 'started-older']
|
||||
const [copy] = later.db
|
||||
.prepare(
|
||||
`SELECT session_id FROM agent_session_records WHERE session_id NOT IN (${known
|
||||
@@ -313,8 +338,9 @@ it('keeps only what an older Orca wrote after a file from before revisions were
|
||||
)
|
||||
.all(...known)
|
||||
.map((row) => String(row.session_id))
|
||||
expect(await later.convert(copy!)).toBe('epoch')
|
||||
expect(items(later.db, copy!)).toContain('older-turn')
|
||||
expect(tabbedChats(later.db)).toEqual(['open', copy, 'started-older'])
|
||||
expect(tabbedChats(later.db)).toEqual(['open', 'started-older', copy])
|
||||
expect(later.db.prepare('SELECT COUNT(*) AS records FROM agent_session_records').get()).toEqual({
|
||||
records: known.length + 1
|
||||
})
|
||||
@@ -331,6 +357,161 @@ it('keeps only what an older Orca wrote after a file from before revisions were
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps a message an older Orca queued in a chat from before revisions were kept', async () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
|
||||
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
|
||||
const { db: original } = openJournalDatabase(journalDatabasePath(root))
|
||||
cleanup.push(() => original.close())
|
||||
olderOrcaChat(original, 'queued', rows)
|
||||
const first = await upgrade(root)
|
||||
expect(await first.convert('queued')).toBe('epoch')
|
||||
asOlderChatStore(first.db)
|
||||
await first.close()
|
||||
|
||||
insertJournalRow(original, 'queued', {
|
||||
v: AGENT_SESSION_JOURNAL_QUEUE_SCHEMA_VERSION,
|
||||
epoch: 'epoch',
|
||||
seq: tip + 1,
|
||||
ts: 100,
|
||||
fence: 1,
|
||||
kind: 'queue',
|
||||
messageId: 'older-queued',
|
||||
change: {
|
||||
op: 'queued',
|
||||
body: { kind: 'message', role: 'user', blocks: [{ type: 'text', text: 'later' }] },
|
||||
position: 1,
|
||||
queuedAt: null
|
||||
}
|
||||
})
|
||||
const compared = await upgrade(root)
|
||||
await compared.compare()
|
||||
await compared.close()
|
||||
const later = await upgrade(root)
|
||||
cleanup.push(later.close)
|
||||
const copy = later.db
|
||||
.prepare("SELECT session_id FROM agent_session_records WHERE session_id!='queued'")
|
||||
.get()?.session_id
|
||||
expect(await later.convert(String(copy))).toBe('epoch')
|
||||
expect(
|
||||
later.db
|
||||
.prepare('SELECT message_id FROM journal_queue_cards WHERE session_id=?')
|
||||
.all(String(copy))
|
||||
).toContainEqual({ message_id: 'older-queued' })
|
||||
})
|
||||
|
||||
it('records a chat it cannot keep and does not retry it until the older Orca writes it again', async () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
|
||||
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
|
||||
const { db: original } = openJournalDatabase(journalDatabasePath(root))
|
||||
cleanup.push(() => original.close())
|
||||
olderOrcaChat(original, 'kept', rows)
|
||||
await (await upgrade(root)).close()
|
||||
olderOrcaChat(original, 'refused', [rows[0], turn('refused-turn', 2)])
|
||||
original.pragma('wal_checkpoint(TRUNCATE)')
|
||||
|
||||
const blocked = openJournalDomainDatabase(journalDomainDatabasePath(root)).db
|
||||
blocked.exec(`CREATE TRIGGER refused BEFORE INSERT ON agent_session_records
|
||||
WHEN NEW.session_id='refused' BEGIN SELECT RAISE(ABORT,'record refused'); END`)
|
||||
blocked.close()
|
||||
const failing = openJournalDomainDatabase(await initializeJournalDomainDatabase(root)).db
|
||||
const recorded = (key: string) =>
|
||||
failing.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)?.value
|
||||
expect(JSON.parse(String(recorded('journal.legacy-failed:refused')))).toMatchObject({
|
||||
error: expect.stringContaining('record refused')
|
||||
})
|
||||
expect(recorded('journal.legacy-revisions-kept.v1')).toBe('1')
|
||||
failing.exec('DROP TRIGGER refused')
|
||||
failing.close()
|
||||
|
||||
const refusedRecord = (db: Database.Database) =>
|
||||
db.prepare("SELECT 1 FROM agent_session_records WHERE session_id='refused'").get()
|
||||
const absorbed = await upgrade(root)
|
||||
expect(refusedRecord(absorbed.db)).toBeUndefined()
|
||||
await absorbed.close()
|
||||
|
||||
insertJournalRow(original, 'refused', turn('refused-turn-2', 3))
|
||||
original.pragma('wal_checkpoint(TRUNCATE)')
|
||||
const retried = await upgrade(root)
|
||||
cleanup.push(retried.close)
|
||||
expect(refusedRecord(retried.db)).toEqual({ 1: 1 })
|
||||
})
|
||||
|
||||
it('keeps a fresh copy when another build readied a copy before its history converted here', async () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
|
||||
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
|
||||
const { db: original } = openJournalDatabase(journalDatabasePath(root))
|
||||
cleanup.push(() => original.close())
|
||||
olderOrcaChat(original, 'converted', rows)
|
||||
const first = await upgrade(root)
|
||||
expect(await first.convert('converted')).toBe('epoch')
|
||||
await first.close()
|
||||
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
|
||||
original.pragma('wal_checkpoint(TRUNCATE)')
|
||||
|
||||
const copied = await upgrade(root)
|
||||
const copies = (db: Database.Database) =>
|
||||
db
|
||||
.prepare("SELECT session_id FROM agent_session_records WHERE session_id!='converted'")
|
||||
.all()
|
||||
.map((row) => String(row.session_id))
|
||||
const [stranded] = copies(copied.db)
|
||||
// An older build that can't read the copy's source founds it empty, as a new chat.
|
||||
copied.db
|
||||
.prepare(
|
||||
"INSERT INTO journal_chat_state(workspace_id,session_id,epoch) VALUES('workspace',?,'founded') ON CONFLICT(session_id) DO UPDATE SET epoch='founded'"
|
||||
)
|
||||
.run(stranded!)
|
||||
await copied.close()
|
||||
|
||||
const again = await upgrade(root)
|
||||
cleanup.push(again.close)
|
||||
const kept = copies(again.db).find((id) => id !== stranded)
|
||||
expect(copies(again.db)).toHaveLength(2)
|
||||
expect(await again.convert(kept!)).toBe('epoch')
|
||||
expect(items(again.db, kept!)).toContain('older-turn')
|
||||
})
|
||||
|
||||
it('leaves a copy whose history failed to convert to an open, not to every idle pass', async () => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'orca-legacy-reconcile-'))
|
||||
cleanup.push(() => rmSync(root, { recursive: true, force: true }))
|
||||
const { db: original } = openJournalDatabase(journalDatabasePath(root))
|
||||
cleanup.push(() => original.close())
|
||||
olderOrcaChat(original, 'converted', rows)
|
||||
const first = await upgrade(root)
|
||||
expect(await first.convert('converted')).toBe('epoch')
|
||||
await first.close()
|
||||
insertJournalRow(original, 'converted', turn('older-turn', tip + 1))
|
||||
original.pragma('wal_checkpoint(TRUNCATE)')
|
||||
const later = await upgrade(root)
|
||||
cleanup.push(later.close)
|
||||
const { db } = later
|
||||
const [copy] = db
|
||||
.prepare("SELECT session_id FROM agent_session_records WHERE session_id!='converted'")
|
||||
.all()
|
||||
.map((row) => String(row.session_id))
|
||||
const idle = () => {
|
||||
const page = readJournalSqlDatabase(db, { kind: 'sessions', after: null })
|
||||
return page.kind === 'sessions' ? page.sessionIds : []
|
||||
}
|
||||
expect(idle()).toContain(copy)
|
||||
|
||||
for (const sessionId of [copy!, 'converted']) {
|
||||
runJournalTransaction(db, () =>
|
||||
applyJournalSqlMaintenance(db, { kind: 'legacy-copy-failed', sessionId, error: 'damaged' })
|
||||
)
|
||||
}
|
||||
expect(idle()).not.toContain(copy)
|
||||
expect(idle()).toContain('converted')
|
||||
|
||||
expect(await later.convert(copy!)).toBe('epoch')
|
||||
expect(items(db, copy!)).toContain('older-turn')
|
||||
expect(
|
||||
db
|
||||
.prepare('SELECT 1 FROM agent_session_store_meta WHERE key=?')
|
||||
.get(`journal.legacy-failed:${copy}`)
|
||||
).toBeUndefined()
|
||||
})
|
||||
|
||||
// Perf contract: an unchanged original only needs baselines, so it skips adoption at any fleet size.
|
||||
it.each([10, 100])(
|
||||
'fills in %i missing baselines without reading tabs, claims or chat state when the original is unchanged',
|
||||
|
||||
@@ -1,21 +1,32 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import type Database from '../../../sqlite/sync-database'
|
||||
import type { SqliteRow } from '../../../sqlite/sqlite-statement'
|
||||
import { structuredAgentSessionTabId } from '../../../../shared/agent-session/view/structured-agent-session-projection'
|
||||
import { prepareJournalDomainImport, type JournalDomainImportPage } from '../journal-domain-import'
|
||||
import {
|
||||
finishJournalDomainImport,
|
||||
insertJournalDomainImportPage
|
||||
} from '../journal-domain-import-rows'
|
||||
import { runJournalTransaction } from './journal-database'
|
||||
import { isSqliteCorruption, isSqliteStorageFailure } from '../../../sqlite/sqlite-read-failure'
|
||||
import { AGENT_SESSION_TABS_RECORDED } from '../../../runtime/agent-session/records/agent-session-record-rows'
|
||||
import { readAdoption, type Adoption, type Claim } from './journal-legacy-adoption'
|
||||
import { journalDatabasePath } from './journal-host-database'
|
||||
import {
|
||||
journalLegacyChatRevision,
|
||||
journalLegacyChatRevisionKey,
|
||||
openJournalLegacySource,
|
||||
recordJournalLegacyChatRevision
|
||||
recordJournalLegacyChatRevision,
|
||||
settleJournalLegacyCopy
|
||||
} from './journal-legacy-source'
|
||||
import {
|
||||
keepContinuedChat,
|
||||
keepForcedCopy,
|
||||
type JournalLegacyForcedCopy
|
||||
} from './journal-legacy-copies'
|
||||
import {
|
||||
JOURNAL_LEGACY_COMPARE,
|
||||
JOURNAL_LEGACY_COPY_DUE,
|
||||
metaKeys,
|
||||
readMeta,
|
||||
recordJournalLegacyFailure,
|
||||
strandedCopies,
|
||||
writeMeta,
|
||||
type JournalLegacyFailure
|
||||
} from './journal-legacy-reconcile-marks'
|
||||
import {
|
||||
captureJournalLegacySourceReceipt,
|
||||
JOURNAL_LEGACY_SOURCE_RECEIPT,
|
||||
@@ -24,12 +35,13 @@ import {
|
||||
|
||||
/** Set after a pass absorbed every original chat, so each chat here carries its revision. */
|
||||
export const JOURNAL_LEGACY_REVISIONS_KEPT = 'journal.legacy-revisions-kept.v1'
|
||||
|
||||
/**
|
||||
* An older Orca keeps writing the original journal after a downgrade. When the original changed,
|
||||
* each original chat's revision is compared with the one this file shows: a chat it started, or
|
||||
* one not converted here yet, joins as itself; a converted chat it continued is kept as a second
|
||||
* chat beside it, so no turn it saved goes missing. Nothing here deletes a chat or its turns.
|
||||
* Terminal restore waits for this pass, so it reads no history: a copy's history converts later
|
||||
* like any unconverted chat's, and a chat needing a history compare is compared in the background.
|
||||
*/
|
||||
export function reconcileJournalLegacySource(stateDirectory: string, db: Database.Database): void {
|
||||
const sourcePath = journalDatabasePath(stateDirectory)
|
||||
@@ -42,19 +54,29 @@ export function reconcileJournalLegacySource(stateDirectory: string, db: Databas
|
||||
const saved = readMeta(db, JOURNAL_LEGACY_SOURCE_RECEIPT)
|
||||
const unchanged = sameJournalLegacySource(saved, captured)
|
||||
const kept = readMeta(db, JOURNAL_LEGACY_REVISIONS_KEPT) !== undefined
|
||||
if (unchanged && kept) {
|
||||
const forced = new Map<string, JournalLegacyForcedCopy>(
|
||||
metaKeys(db, JOURNAL_LEGACY_COPY_DUE).map((id) => [id, {}])
|
||||
)
|
||||
for (const [id, copy] of strandedCopies(db)) {
|
||||
forced.set(id, { stranded: copy })
|
||||
}
|
||||
if (unchanged && kept && forced.size === 0) {
|
||||
return
|
||||
}
|
||||
const receipt = JSON.stringify(captured)
|
||||
const pass: Pass = { kept, baselines: [] }
|
||||
const pass: Pass = { kept, forced, baselines: [], compares: [], failures: [] }
|
||||
const source = openJournalLegacySource(sourcePath)
|
||||
let complete = true
|
||||
try {
|
||||
if (unchanged) {
|
||||
// The original wrote nothing since the copy, so each chat is as shown here.
|
||||
pass.baselines = missingBaselines(source, db)
|
||||
// The original wrote nothing since the copy, so each chat is as shown here; once every
|
||||
// chat carries its revision, only chats due a copy have work.
|
||||
if (!kept) {
|
||||
pass.baselines = missingBaselines(source, db).filter(([id]) => !forced.has(id))
|
||||
}
|
||||
complete = forced.size === 0 || absorbChats(source, db, pass, false)
|
||||
} else {
|
||||
complete = absorbChats(source, db, pass)
|
||||
complete = absorbChats(source, db, pass, true)
|
||||
}
|
||||
} finally {
|
||||
source.close()
|
||||
@@ -71,7 +93,17 @@ export function reconcileJournalLegacySource(stateDirectory: string, db: Databas
|
||||
for (const [id, revision] of pass.baselines) {
|
||||
recordJournalLegacyChatRevision(db, id, revision)
|
||||
}
|
||||
// A failed chat leaves the original unabsorbed, so the next launch retries it.
|
||||
for (const [id, revision] of pass.compares) {
|
||||
writeMeta(db, `${JOURNAL_LEGACY_COMPARE}${id}`, revision)
|
||||
}
|
||||
for (const failure of pass.failures) {
|
||||
recordJournalLegacyFailure(db, failure)
|
||||
const { stranded } = pass.forced.get(failure.sessionId) ?? {}
|
||||
if (stranded !== undefined) {
|
||||
settleJournalLegacyCopy(db, stranded)
|
||||
}
|
||||
}
|
||||
// A storage failure leaves the original unabsorbed; the next launch retries it.
|
||||
if (complete) {
|
||||
writeMeta(db, JOURNAL_LEGACY_SOURCE_RECEIPT, receipt)
|
||||
writeMeta(db, JOURNAL_LEGACY_REVISIONS_KEPT, '1')
|
||||
@@ -83,19 +115,12 @@ export function reconcileJournalLegacySource(stateDirectory: string, db: Databas
|
||||
type Pass = {
|
||||
/** Every chat in the original at the copy was given a revision here. */
|
||||
kept: boolean
|
||||
/** Chats that get a copy whatever their revision shows. */
|
||||
forced: ReadonlyMap<string, JournalLegacyForcedCopy>
|
||||
baselines: [string, string][]
|
||||
}
|
||||
|
||||
type Claim = SqliteRow
|
||||
|
||||
/** The original's tabs and claims, and this file's tabs, read once per pass. */
|
||||
type Adoption = {
|
||||
savedTabs: Map<string, string>
|
||||
claims: Map<string, Claim[]>
|
||||
tabbed: Set<string>
|
||||
takenTabs: Set<string>
|
||||
nextPosition: number
|
||||
tabsRecorded: boolean
|
||||
/** Converted chats with no revision to compare, for the background compare. */
|
||||
compares: [string, string][]
|
||||
failures: JournalLegacyFailure[]
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -120,64 +145,69 @@ function originalChatIds(source: Database.Database, order: 'saved' | 'any'): str
|
||||
.map((row) => String(row.session_id))
|
||||
}
|
||||
|
||||
/** True when every chat was absorbed; a failed chat is logged and retried next launch. */
|
||||
function absorbChats(source: Database.Database, db: Database.Database, pass: Pass): boolean {
|
||||
let adoption = readAdoption(source, db)
|
||||
/**
|
||||
* False only when a storage failure stopped the pass. A chat that fails otherwise is recorded with its
|
||||
* error and taken as absorbed, so one bad chat can't rerun the pass at every launch. Without a walk,
|
||||
* only chats due a copy are visited.
|
||||
*/
|
||||
function absorbChats(
|
||||
source: Database.Database,
|
||||
db: Database.Database,
|
||||
pass: Pass,
|
||||
walk: boolean
|
||||
): boolean {
|
||||
const adoption = readAdoption(source, db, walk ? null : [...pass.forced.keys()])
|
||||
runJournalTransaction(db, () => {
|
||||
for (const [sessionId, claims] of adoption.claims) {
|
||||
insertClaims(db, sessionId, claims)
|
||||
}
|
||||
})
|
||||
let complete = true
|
||||
for (const id of originalChatIds(source, 'saved')) {
|
||||
const chats = walk ? originalChatIds(source, 'saved') : [...pass.forced.keys()]
|
||||
for (const id of chats) {
|
||||
let revision: string | undefined
|
||||
const added = adoption.added.length
|
||||
const { nextPosition, tabsRecorded } = adoption
|
||||
try {
|
||||
absorbChat(source, db, id, adoption, pass)
|
||||
// A chat with no history still has a value, so one deleted here stays deleted.
|
||||
revision = journalLegacyChatRevision(source, id) ?? 'null'
|
||||
absorbChat(source, db, id, revision, adoption, pass)
|
||||
} catch (error) {
|
||||
complete = false
|
||||
// The failed chat's transaction rolled back; reread what this file now holds.
|
||||
adoption = readAdoption(source, db)
|
||||
console.warn('[agent-session-journal] keeping an older Orca chat failed', error)
|
||||
// Corruption past the original's own revision read may be this file's: never absorb it. A
|
||||
// rollback that failed leaves this connection mid-transaction for every later chat.
|
||||
if (
|
||||
isSqliteStorageFailure(error) ||
|
||||
(revision !== undefined && isSqliteCorruption(error)) ||
|
||||
db.isTransaction
|
||||
) {
|
||||
console.warn('[agent-session-journal] keeping chats an older Orca saved paused', error)
|
||||
return false
|
||||
}
|
||||
pass.failures.push({ sessionId: id, revision, error })
|
||||
// The failed chat's transaction rolled back, and with it any tab it added.
|
||||
for (const [sessionId, tab] of adoption.added.splice(added)) {
|
||||
adoption.tabbed.delete(sessionId)
|
||||
adoption.takenTabs.delete(tab)
|
||||
}
|
||||
Object.assign(adoption, { nextPosition, tabsRecorded })
|
||||
}
|
||||
}
|
||||
return complete
|
||||
}
|
||||
|
||||
function readAdoption(source: Database.Database, db: Database.Database): Adoption {
|
||||
const claims = new Map<string, Claim[]>()
|
||||
for (const claim of source
|
||||
.prepare('SELECT upload_id,session_id,claimed_at FROM agent_session_attachment_claims')
|
||||
.all()) {
|
||||
const id = String(claim.session_id)
|
||||
const list = claims.get(id) ?? []
|
||||
list.push(claim)
|
||||
claims.set(id, list)
|
||||
}
|
||||
const tabs = db.prepare('SELECT tab_id,session_id FROM agent_session_tabs').all()
|
||||
const tail = db.prepare('SELECT COALESCE(MAX(position),-1) AS tail FROM agent_session_tabs').get()
|
||||
return {
|
||||
savedTabs: new Map(
|
||||
source
|
||||
.prepare('SELECT tab_id,session_id FROM agent_session_tabs')
|
||||
.all()
|
||||
.map((row) => [String(row.session_id), String(row.tab_id)])
|
||||
),
|
||||
claims,
|
||||
tabbed: new Set(tabs.map((row) => String(row.session_id))),
|
||||
takenTabs: new Set(tabs.map((row) => String(row.tab_id))),
|
||||
nextPosition: Number(tail?.tail ?? -1) + 1,
|
||||
tabsRecorded: readMeta(db, AGENT_SESSION_TABS_RECORDED) !== undefined
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
function absorbChat(
|
||||
source: Database.Database,
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
revision: string,
|
||||
adoption: Adoption,
|
||||
pass: Pass
|
||||
): void {
|
||||
// A chat with no history still has a value, so one deleted here stays deleted.
|
||||
const revision = journalLegacyChatRevision(source, id) ?? 'null'
|
||||
const adopt = (copy: string) => adoptChat(source, db, id, copy, adoption)
|
||||
const forced = pass.forced.get(id)
|
||||
if (forced) {
|
||||
runJournalTransaction(db, () => keepForcedCopy(db, id, revision, forced, adopt))
|
||||
return
|
||||
}
|
||||
const shown = readMeta(db, journalLegacyChatRevisionKey(id))
|
||||
if (revision === shown) {
|
||||
return
|
||||
@@ -197,7 +227,7 @@ function absorbChat(
|
||||
return
|
||||
}
|
||||
if (converted) {
|
||||
keepContinuedChat(source, db, id, revision, shown !== undefined, adoption, pass)
|
||||
keepContinuedChat(db, id, revision, shown, adopt, pass)
|
||||
return
|
||||
}
|
||||
// Not converted here yet: its conversion reads the current original, so it joins as itself.
|
||||
@@ -238,10 +268,11 @@ function adoptChat(
|
||||
const session = source
|
||||
.prepare('SELECT workspace_id FROM journal_sessions WHERE session_id=?')
|
||||
.get(id)
|
||||
if (as === id && session) {
|
||||
// A copy is cataloged unconverted too; its history converts from the original chat.
|
||||
if (session) {
|
||||
db.prepare(
|
||||
'INSERT INTO journal_chat_state(session_id,workspace_id) VALUES(?,?) ON CONFLICT DO NOTHING'
|
||||
).run(id, session.workspace_id)
|
||||
).run(as, session.workspace_id)
|
||||
}
|
||||
if (as !== id) {
|
||||
insertClaims(db, as, adoption.claims.get(id) ?? [])
|
||||
@@ -274,73 +305,7 @@ function addTab(db: Database.Database, adoption: Adoption, sessionId: string, ta
|
||||
adoption.nextPosition += 1
|
||||
adoption.tabbed.add(sessionId)
|
||||
adoption.takenTabs.add(tab)
|
||||
}
|
||||
|
||||
/**
|
||||
* The original's version of a converted chat it moved past becomes a second chat beside it. With
|
||||
* no revision to compare (files copied before revisions were kept), the original moved past this
|
||||
* chat only if its restored history has an item this chat never held.
|
||||
*/
|
||||
function keepContinuedChat(
|
||||
source: Database.Database,
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
revision: string,
|
||||
compared: boolean,
|
||||
adoption: Adoption,
|
||||
pass: Pass
|
||||
): void {
|
||||
const imported = prepareJournalDomainImport(source, id)
|
||||
if (!imported) {
|
||||
pass.baselines.push([id, revision])
|
||||
return
|
||||
}
|
||||
const { pages: read, ...header } = imported
|
||||
let pages: Iterable<JournalDomainImportPage> = read
|
||||
if (!compared) {
|
||||
const all = [...read]
|
||||
if (heldEveryItem(db, id, all)) {
|
||||
pass.baselines.push([id, revision])
|
||||
return
|
||||
}
|
||||
pages = all
|
||||
}
|
||||
// Derived from the revision, so an interrupted import resumes into the same chat.
|
||||
const copy = createHash('sha256').update(`${id}\0${revision}`).digest('hex').slice(0, 32)
|
||||
const target = { sessionId: copy, epoch: header.epoch }
|
||||
for (const page of pages) {
|
||||
runJournalTransaction(db, () => insertJournalDomainImportPage(db, target, page))
|
||||
}
|
||||
runJournalTransaction(db, () => {
|
||||
finishJournalDomainImport(db, { ...header, sessionId: copy, sourceRevision: null })
|
||||
adoptChat(source, db, id, copy, adoption)
|
||||
recordJournalLegacyChatRevision(db, id, revision)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Removing an item here leaves its tombstone, so every item this chat converted stays held, at
|
||||
* its converted revision or a later one.
|
||||
*/
|
||||
function heldEveryItem(
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
pages: readonly JournalDomainImportPage[]
|
||||
): boolean {
|
||||
const held = new Map(
|
||||
db
|
||||
.prepare(`SELECT item_id AS id, MAX(revision) AS revision FROM (
|
||||
SELECT item_id, revision FROM journal_items WHERE session_id=?1
|
||||
UNION ALL SELECT item_id, ?2 FROM journal_item_tombstones WHERE session_id=?1)
|
||||
GROUP BY item_id`)
|
||||
.all(id, Number.MAX_SAFE_INTEGER)
|
||||
.map((row) => [String(row.id), Number(row.revision)])
|
||||
)
|
||||
return pages.every(
|
||||
(page) =>
|
||||
page.kind !== 'items' ||
|
||||
page.rows.every(({ item }) => (held.get(item.itemId) ?? -1) >= item.revision)
|
||||
)
|
||||
adoption.added.push([sessionId, tab])
|
||||
}
|
||||
|
||||
/** Uploads the older Orca attached stay claimed, so the sweeper never removes them. */
|
||||
@@ -351,15 +316,3 @@ function insertClaims(db: Database.Database, sessionId: string, claims: readonly
|
||||
insert.run(claim.upload_id!, sessionId, claim.claimed_at!)
|
||||
}
|
||||
}
|
||||
|
||||
function readMeta(db: Database.Database, key: string): string | undefined {
|
||||
const row = db.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)
|
||||
return row === undefined ? undefined : String(row.value)
|
||||
}
|
||||
|
||||
function writeMeta(db: Database.Database, key: string, value: string): void {
|
||||
db.prepare('INSERT OR REPLACE INTO agent_session_store_meta (key, value) VALUES (?, ?)').run(
|
||||
key,
|
||||
value
|
||||
)
|
||||
}
|
||||
|
||||
@@ -6,10 +6,9 @@ import { afterEach, expect, it } from 'vitest'
|
||||
import { openJournalDatabase, runJournalTransaction } from './journal-database'
|
||||
import {
|
||||
initializeJournalDomainDatabase,
|
||||
journalDomainDatabasePath,
|
||||
openJournalDomainDatabase
|
||||
} from './journal-domain-database'
|
||||
import { journalDatabasePath } from './journal-host-database'
|
||||
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
|
||||
import { insertJournalRow, publishJournalSessionEpoch } from '../durable-rows/journal-row-table'
|
||||
import { journalSqlRetryRows } from '../journal-sql-retry-rows.test-fixture'
|
||||
import { JournalSqlReaderPool } from '../journal-sql-reader-pool'
|
||||
|
||||
@@ -90,6 +90,7 @@ export function journalLegacyChatRevisionKey(sessionId: string): string {
|
||||
return `journal.legacy-revision:${sessionId}`
|
||||
}
|
||||
|
||||
/** Absorbing a revision clears any failure an earlier one recorded. */
|
||||
export function recordJournalLegacyChatRevision(
|
||||
db: Database.Database,
|
||||
sessionId: string,
|
||||
@@ -99,4 +100,47 @@ export function recordJournalLegacyChatRevision(
|
||||
journalLegacyChatRevisionKey(sessionId),
|
||||
revision
|
||||
)
|
||||
db.prepare('DELETE FROM agent_session_store_meta WHERE key=?').run(
|
||||
journalLegacyFailureKey(sessionId)
|
||||
)
|
||||
}
|
||||
|
||||
/** A chat, or a copy's history, that could not be kept, with its error; not retried by itself. */
|
||||
export function journalLegacyFailureKey(sessionId: string): string {
|
||||
return `journal.legacy-failed:${sessionId}`
|
||||
}
|
||||
|
||||
/** A copy kept for an original chat an older Orca continued converts its history from that chat. */
|
||||
export function journalLegacyCopySourceKey(copyId: string): string {
|
||||
return `journal.legacy-copy-source:${copyId}`
|
||||
}
|
||||
|
||||
/** The original chat a copy converts from, or the chat itself. */
|
||||
export function journalLegacyConversionSource(db: Database.Database, sessionId: string): string {
|
||||
const row = db
|
||||
.prepare('SELECT value FROM agent_session_store_meta WHERE key=?')
|
||||
.get(journalLegacyCopySourceKey(sessionId))
|
||||
return row === undefined ? sessionId : String(row.value)
|
||||
}
|
||||
|
||||
/** The copy kept for an original chat that hasn't converted here yet. */
|
||||
export function journalLegacyPendingCopyKey(sessionId: string): string {
|
||||
return `journal.legacy-pending-copy:${sessionId}`
|
||||
}
|
||||
|
||||
/** Once a copy is ready here its history no longer reads from the original. */
|
||||
export function settleJournalLegacyCopy(db: Database.Database, copyId: string): void {
|
||||
const key = journalLegacyCopySourceKey(copyId)
|
||||
const original = db.prepare('SELECT value FROM agent_session_store_meta WHERE key=?').get(key)
|
||||
if (original === undefined) {
|
||||
return
|
||||
}
|
||||
db.prepare('DELETE FROM agent_session_store_meta WHERE key IN (?,?)').run(
|
||||
key,
|
||||
journalLegacyFailureKey(copyId)
|
||||
)
|
||||
db.prepare('DELETE FROM agent_session_store_meta WHERE key=? AND value=?').run(
|
||||
journalLegacyPendingCopyKey(String(original.value)),
|
||||
copyId
|
||||
)
|
||||
}
|
||||
|
||||
@@ -15,8 +15,7 @@ import type { JournalWriterEffect, JournalWriterEffectResult } from '../journal-
|
||||
import type { CommandReceiptScope } from '../writing/command-receipt-schema'
|
||||
import { AgentSessionJournalError } from '../writing/journal-write-guards'
|
||||
import { assertJournalChatReadable } from '../replay/journal-open-failure'
|
||||
import { journalDatabasePath } from './journal-host-database'
|
||||
import { journalDomainDatabasePath } from './journal-domain-database'
|
||||
import { journalDatabasePath, journalDomainDatabasePath } from './journal-host-database'
|
||||
import type { JournalSqlStoredRecords } from '../journal-sql-database-read'
|
||||
|
||||
/** The runtime owns workers, never a writable connection or a retained chat projection. */
|
||||
|
||||
@@ -9,7 +9,10 @@ import type Database from '../../sqlite/sync-database'
|
||||
import { insertJournalDomainQueueRow, journalDomainQueueRow } from './journal-domain-queue-rows'
|
||||
import type { JournalDomainImportHeader, JournalDomainImportPage } from './journal-domain-import'
|
||||
import { writeJournalDomainItemNames } from './journal-domain-item-names'
|
||||
import { recordJournalLegacyChatRevision } from './database/journal-legacy-source'
|
||||
import {
|
||||
recordJournalLegacyChatRevision,
|
||||
settleJournalLegacyCopy
|
||||
} from './database/journal-legacy-source'
|
||||
import { hydrateJournalStoredItem, journalStoredItemRow } from './journal-stored-output-rows'
|
||||
|
||||
export type JournalDomainImportState =
|
||||
@@ -214,6 +217,7 @@ export function finishJournalDomainImport(
|
||||
if (header.sourceRevision) {
|
||||
recordJournalLegacyChatRevision(db, header.sessionId, header.sourceRevision)
|
||||
}
|
||||
settleJournalLegacyCopy(db, header.sessionId)
|
||||
return state.epoch
|
||||
}
|
||||
|
||||
|
||||
@@ -305,6 +305,7 @@ it('finishes a retired pass during activity while an already-started conversion
|
||||
}
|
||||
},
|
||||
{
|
||||
compareLegacyChats: async () => {},
|
||||
ensure: async (_sessionId, foreground) => {
|
||||
if (foreground) {
|
||||
throw new Error('background conversion was promoted')
|
||||
|
||||
@@ -122,7 +122,7 @@ it('deletes an idle chat with its tab, then every journal row in background page
|
||||
inTransaction(db, () => applyJournalSqlMaintenance(db, operation))
|
||||
},
|
||||
{ database: async (operation) => readJournalSqlDatabase(db, operation) },
|
||||
{ ensure: async () => null },
|
||||
{ ensure: async () => null, compareLegacyChats: async () => {} },
|
||||
new JournalSqlActivity(),
|
||||
(error) => {
|
||||
throw error
|
||||
@@ -149,6 +149,43 @@ it('deletes an idle chat with its tab, then every journal row in background page
|
||||
expect(count(db, "journal_chat_state WHERE session_id='kept-chat'")).toBe(1)
|
||||
})
|
||||
|
||||
it('takes the revision a chat awaiting its older-Orca compare waited on, so no copy brings it back', () => {
|
||||
const db = journal()
|
||||
db.prepare(
|
||||
`DELETE FROM agent_session_store_meta WHERE key='journal.legacy-revision:gone-chat'`
|
||||
).run()
|
||||
db.prepare(
|
||||
`INSERT INTO agent_session_store_meta VALUES('journal.legacy-compare:gone-chat','["legacy",4]')`
|
||||
).run()
|
||||
expect(forget(db)).toBe(true)
|
||||
expect(count(db, "agent_session_store_meta WHERE key LIKE 'journal.legacy-compare:%'")).toBe(0)
|
||||
expect(
|
||||
db
|
||||
.prepare(
|
||||
"SELECT value FROM agent_session_store_meta WHERE key='journal.legacy-revision:gone-chat'"
|
||||
)
|
||||
.get()?.value
|
||||
).toBe('["legacy",4]')
|
||||
})
|
||||
|
||||
it("drops a forgotten copy's mapping and failure, so none of its keys outlive it", () => {
|
||||
const db = journal()
|
||||
for (const [key, value] of [
|
||||
['journal.legacy-copy-source:gone-chat', 'original-chat'],
|
||||
['journal.legacy-failed:gone-chat', '{"revision":null,"error":"damaged"}'],
|
||||
['journal.legacy-pending-copy:original-chat', 'gone-chat']
|
||||
]) {
|
||||
db.prepare('INSERT INTO agent_session_store_meta VALUES(?,?)').run(key, value)
|
||||
}
|
||||
expect(forget(db)).toBe(true)
|
||||
expect(
|
||||
count(
|
||||
db,
|
||||
"agent_session_store_meta WHERE key LIKE 'journal.legacy-copy-source:%' OR key LIKE 'journal.legacy-failed:%' OR key LIKE 'journal.legacy-pending-copy:%'"
|
||||
)
|
||||
).toBe(0)
|
||||
})
|
||||
|
||||
const refusals: {
|
||||
name: string
|
||||
gone?: Partial<AgentSessionRecord>
|
||||
|
||||
@@ -28,7 +28,7 @@ export class JournalSqlBackgroundMaintenance {
|
||||
constructor(
|
||||
private readonly writer: Pick<JournalSqlOwner, 'maintain'>,
|
||||
private readonly readers: Pick<JournalSqlReaderPool, 'database'>,
|
||||
private readonly conversions: Pick<JournalSqlConversions, 'ensure'>,
|
||||
private readonly conversions: Pick<JournalSqlConversions, 'ensure' | 'compareLegacyChats'>,
|
||||
private readonly activity: JournalSqlActivity,
|
||||
private readonly onError: (error: unknown, sessionId?: string) => void,
|
||||
private readonly onTick?: () => void
|
||||
@@ -114,6 +114,7 @@ export class JournalSqlBackgroundMaintenance {
|
||||
if (this.discover) {
|
||||
this.discover = false
|
||||
try {
|
||||
await this.conversions.compareLegacyChats(() => this.boundary())
|
||||
await this.convertSavedChats()
|
||||
} catch (error) {
|
||||
this.report(error)
|
||||
|
||||
@@ -1,9 +1,20 @@
|
||||
import { setImmediate } from 'node:timers/promises'
|
||||
import type { JournalSqlOwner } from './journal-sql-owner'
|
||||
import type { JournalSqlReaderPool } from './journal-sql-reader-pool'
|
||||
import { compareJournalLegacyChatsWhenIdle } from './journal-sql-legacy-compares'
|
||||
import { isJournalSqlSourceContentFailure } from './journal-sql-failure'
|
||||
import { isJournalReaderReplyLimitFailure } from './journal-sql-reader-reply-limit'
|
||||
|
||||
type Conversion = { promotion: AbortController; result: Promise<string | null> }
|
||||
|
||||
function warnLegacy(step: string, sessionId: string, error: unknown): void {
|
||||
console.warn(
|
||||
`[agent-session-journal] ${step} a chat an older Orca saved failed`,
|
||||
sessionId,
|
||||
error
|
||||
)
|
||||
}
|
||||
|
||||
/** One lock per chat, shared by background work and opens; promotion wakes a paused conversion. */
|
||||
export class JournalSqlConversions {
|
||||
private readonly pending = new Map<string, Conversion>()
|
||||
@@ -12,10 +23,26 @@ export class JournalSqlConversions {
|
||||
|
||||
constructor(
|
||||
private readonly writer: Pick<JournalSqlOwner, 'maintain'>,
|
||||
private readonly readers: Pick<JournalSqlReaderPool, 'importState' | 'prepareConversion'>,
|
||||
private readonly readers: Pick<
|
||||
JournalSqlReaderPool,
|
||||
'importState' | 'prepareConversion' | 'database' | 'compareLegacyChat'
|
||||
>,
|
||||
private readonly waitForIdle: (promotion: AbortSignal) => Promise<void>
|
||||
) {}
|
||||
|
||||
/** Converted chats an older Orca may have continued, compared before idle conversion. */
|
||||
compareLegacyChats(boundary: () => Promise<boolean>): Promise<void> {
|
||||
return compareJournalLegacyChatsWhenIdle(
|
||||
this.writer,
|
||||
this.readers,
|
||||
boundary,
|
||||
() => this.closed,
|
||||
(error, sessionId) => warnLegacy('comparing', sessionId, error)
|
||||
).catch((error: unknown) =>
|
||||
console.warn('[agent-session-journal] comparing chats an older Orca saved stopped', error)
|
||||
)
|
||||
}
|
||||
|
||||
async ensure(sessionId: string, foreground = true): Promise<string | null> {
|
||||
if (this.closed) {
|
||||
throw new Error('journal conversions are closed')
|
||||
@@ -42,6 +69,10 @@ export class JournalSqlConversions {
|
||||
}
|
||||
return epoch
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
this.noteCopyFailure(sessionId, error)
|
||||
throw error
|
||||
})
|
||||
.finally(() => {
|
||||
this.pending.delete(sessionId)
|
||||
})
|
||||
@@ -49,6 +80,20 @@ export class JournalSqlConversions {
|
||||
return result
|
||||
}
|
||||
|
||||
/** A copy whose original content can't convert or fit a reply is left to an open, never to a quit's rejection. */
|
||||
private noteCopyFailure(sessionId: string, error: unknown): void {
|
||||
if (
|
||||
this.closed ||
|
||||
!(isJournalSqlSourceContentFailure(error) || isJournalReaderReplyLimitFailure(error))
|
||||
) {
|
||||
return
|
||||
}
|
||||
const message = (error instanceof Error ? error.message : String(error)).slice(0, 500)
|
||||
void this.writer
|
||||
.maintain({ kind: 'legacy-copy-failed', sessionId, error: message })
|
||||
.catch((failure: unknown) => warnLegacy('recording', sessionId, failure))
|
||||
}
|
||||
|
||||
/** A ready physical epoch survives context changes for the lifetime of the database owner. */
|
||||
rememberReady(sessionId: string, epoch: string): void {
|
||||
this.ready.set(sessionId, epoch)
|
||||
|
||||
@@ -12,6 +12,14 @@ import {
|
||||
} from '../../runtime/agent-session/records/agent-session-tab-table'
|
||||
import { readCommandReceipt } from './writing/command-receipt-table'
|
||||
import type { CommandReceiptScope } from './writing/command-receipt-schema'
|
||||
import {
|
||||
journalLegacyCopySourceKey,
|
||||
journalLegacyFailureKey
|
||||
} from './database/journal-legacy-source'
|
||||
import {
|
||||
pendingJournalLegacyCompares,
|
||||
type JournalLegacyCompare
|
||||
} from './database/journal-legacy-compare'
|
||||
import { readForgottenJournalChats } from './journal-forgotten-chats'
|
||||
import { readJournalConnectorReplies } from './connector-replies/journal-connector-replies'
|
||||
import type { JournalConnectorReplyRow } from '../../../shared/agent-session/journal/journal-connector-reply'
|
||||
@@ -21,6 +29,12 @@ export type JournalSqlStoredRecords = Omit<AgentSessionStoreState, 'sessionTabs'
|
||||
backgroundSessions: StoredAgentSessionBackground | null
|
||||
}
|
||||
|
||||
/** Background conversion leaves a copy whose history failed to convert to an open. */
|
||||
const UNFAILED_COPY = `NOT EXISTS (SELECT 1 FROM agent_session_store_meta f
|
||||
WHERE f.key='${journalLegacyFailureKey('')}'||c.session_id
|
||||
AND EXISTS (SELECT 1 FROM agent_session_store_meta m
|
||||
WHERE m.key='${journalLegacyCopySourceKey('')}'||c.session_id))`
|
||||
|
||||
/** A pending chat; epoch is null once its head is gone, generation settles exactly this entry. */
|
||||
export type JournalRetiredHead = { sessionId: string; epoch: string | null; generation: number }
|
||||
|
||||
@@ -33,6 +47,7 @@ export type JournalSqlDatabaseRead =
|
||||
| { kind: 'retired-heads'; after: string | null; through: string | null }
|
||||
| { kind: 'attachment-claims'; after: number }
|
||||
| { kind: 'attachment-sweeps'; after: string | null }
|
||||
| { kind: 'legacy-compares'; after: string | null }
|
||||
| { kind: 'forgotten-chats'; after: string | null }
|
||||
| { kind: 'connector-replies'; sessionId?: string }
|
||||
|
||||
@@ -42,6 +57,7 @@ export type JournalSqlDatabaseReadResult =
|
||||
| { kind: 'receipt'; receipt: ReturnType<typeof readCommandReceipt> }
|
||||
| { kind: 'epoch'; epoch: string | null }
|
||||
| { kind: 'sessions'; sessionIds: string[] }
|
||||
| { kind: 'legacy-compares'; compares: JournalLegacyCompare[] }
|
||||
| { kind: 'retired-heads'; heads: JournalRetiredHead[]; through: string | null }
|
||||
| {
|
||||
kind: 'attachment-claims'
|
||||
@@ -106,15 +122,22 @@ function read(
|
||||
.get(operation.sessionId)
|
||||
?.epoch?.toString() ?? null
|
||||
}
|
||||
case 'legacy-compares':
|
||||
return {
|
||||
kind: 'legacy-compares',
|
||||
compares: pendingJournalLegacyCompares(db, operation.after)
|
||||
}
|
||||
case 'sessions': {
|
||||
const rows =
|
||||
operation.after === null
|
||||
? db
|
||||
.prepare('SELECT session_id FROM journal_chat_state ORDER BY session_id LIMIT 32')
|
||||
.prepare(
|
||||
`SELECT session_id FROM journal_chat_state c WHERE ${UNFAILED_COPY} ORDER BY session_id LIMIT 32`
|
||||
)
|
||||
.all()
|
||||
: db
|
||||
.prepare(`SELECT session_id FROM journal_chat_state
|
||||
WHERE session_id>? ORDER BY session_id LIMIT 32`)
|
||||
.prepare(`SELECT session_id FROM journal_chat_state c
|
||||
WHERE session_id>? AND ${UNFAILED_COPY} ORDER BY session_id LIMIT 32`)
|
||||
.all(operation.after)
|
||||
return { kind: 'sessions', sessionIds: rows.map((row) => String(row.session_id)) }
|
||||
}
|
||||
|
||||
@@ -101,8 +101,10 @@ it('expires replay during continuous activity even while conversion and sweeps w
|
||||
}
|
||||
}
|
||||
},
|
||||
{ database: async (operation) => readJournalSqlDatabase(db, operation) },
|
||||
{ ensure: async () => null },
|
||||
{
|
||||
database: async (operation) => readJournalSqlDatabase(db, operation)
|
||||
},
|
||||
{ ensure: async () => null, compareLegacyChats: async () => {} },
|
||||
activity,
|
||||
(error) => {
|
||||
throw error
|
||||
|
||||
@@ -44,6 +44,11 @@ function journalSqlSourceContent(error: unknown): JournalSourceContent | undefin
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** The original journal's own content failed, not the transport or this process's lifecycle. */
|
||||
export function isJournalSqlSourceContentFailure(error: unknown): boolean {
|
||||
return journalSqlSourceContent(error) !== undefined
|
||||
}
|
||||
|
||||
export function journalSqlSourceFailure(error: unknown): JournalSqlFailure {
|
||||
const failure = journalSqlFailure(error)
|
||||
if (isSqliteCorruption(error)) {
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { expect, it } from 'vitest'
|
||||
import { compareJournalLegacyChatsWhenIdle } from './journal-sql-legacy-compares'
|
||||
import type { JournalSqlMaintenance } from './journal-sql-maintenance'
|
||||
|
||||
async function runWith(failures: Error[], stopping = false): Promise<JournalSqlMaintenance[]> {
|
||||
const settled: JournalSqlMaintenance[] = []
|
||||
let pages = 0
|
||||
await compareJournalLegacyChatsWhenIdle(
|
||||
{
|
||||
maintain: async (operation) => {
|
||||
settled.push(operation)
|
||||
return null
|
||||
}
|
||||
},
|
||||
{
|
||||
database: async () => ({
|
||||
kind: 'legacy-compares',
|
||||
compares: pages++ === 0 ? [{ sessionId: 'huge', revision: 'r1' }] : []
|
||||
}),
|
||||
compareLegacyChat: async () => {
|
||||
const failure = failures.shift()
|
||||
if (failure) {
|
||||
throw failure
|
||||
}
|
||||
return { revision: 'r1', held: true }
|
||||
}
|
||||
},
|
||||
async () => true,
|
||||
() => stopping,
|
||||
() => {}
|
||||
)
|
||||
return settled
|
||||
}
|
||||
|
||||
const exited = () => new Error('journal SQL reader exited')
|
||||
|
||||
it('absorbs a chat whose compare took its reader down twice instead of retrying it every launch', async () => {
|
||||
expect(await runWith([exited(), exited()])).toEqual([
|
||||
{
|
||||
kind: 'legacy-compare-settle',
|
||||
sessionId: 'huge',
|
||||
revision: 'r1',
|
||||
error: 'journal SQL reader exited'
|
||||
}
|
||||
])
|
||||
expect(await runWith([exited()])).toEqual([
|
||||
{ kind: 'legacy-compare-settle', sessionId: 'huge', revision: 'r1', held: true }
|
||||
])
|
||||
expect(await runWith([exited()], true)).toEqual([])
|
||||
})
|
||||
|
||||
it("leaves only the damaged chat's compare pending when this file's storage is damaged", async () => {
|
||||
const corrupt = Object.assign(new Error('database disk image is malformed'), {
|
||||
code: 'ERR_SQLITE_ERROR',
|
||||
errcode: 11
|
||||
})
|
||||
const settled: JournalSqlMaintenance[] = []
|
||||
let pages = 0
|
||||
await compareJournalLegacyChatsWhenIdle(
|
||||
{
|
||||
maintain: async (operation) => {
|
||||
settled.push(operation)
|
||||
return null
|
||||
}
|
||||
},
|
||||
{
|
||||
database: async () => ({
|
||||
kind: 'legacy-compares',
|
||||
compares:
|
||||
pages++ === 0
|
||||
? [
|
||||
{ sessionId: 'damaged', revision: 'r1' },
|
||||
{ sessionId: 'healthy', revision: 'r1' }
|
||||
]
|
||||
: []
|
||||
}),
|
||||
compareLegacyChat: async (sessionId) => {
|
||||
if (sessionId === 'damaged') {
|
||||
throw corrupt
|
||||
}
|
||||
return { revision: 'r1', held: true }
|
||||
}
|
||||
},
|
||||
async () => true,
|
||||
() => false,
|
||||
() => {}
|
||||
)
|
||||
expect(settled).toEqual([
|
||||
{ kind: 'legacy-compare-settle', sessionId: 'healthy', revision: 'r1', held: true }
|
||||
])
|
||||
})
|
||||
|
||||
it('stops at a missing original instead of asking about every chat awaiting a compare', async () => {
|
||||
const asked: string[] = []
|
||||
let pages = 0
|
||||
await compareJournalLegacyChatsWhenIdle(
|
||||
{ maintain: async () => null },
|
||||
{
|
||||
database: async () => ({
|
||||
kind: 'legacy-compares',
|
||||
compares:
|
||||
pages++ === 0
|
||||
? [
|
||||
{ sessionId: 'first', revision: 'r1' },
|
||||
{ sessionId: 'second', revision: 'r1' }
|
||||
]
|
||||
: []
|
||||
}),
|
||||
compareLegacyChat: async (sessionId) => {
|
||||
asked.push(sessionId)
|
||||
return null
|
||||
}
|
||||
},
|
||||
async () => true,
|
||||
() => false,
|
||||
() => {}
|
||||
)
|
||||
expect(asked).toEqual(['first'])
|
||||
})
|
||||
@@ -0,0 +1,73 @@
|
||||
import type { JournalSqlOwner } from './journal-sql-owner'
|
||||
import type { JournalSqlReaderPool } from './journal-sql-reader-pool'
|
||||
import { isSqliteCorruption, isSqliteStorageFailure } from '../../sqlite/sqlite-read-failure'
|
||||
|
||||
/**
|
||||
* Converted chats an older Orca may have continued, with no revision to compare, are compared one
|
||||
* per idle step on a reader. Each settles durably, so an interrupted run resumes next launch.
|
||||
*/
|
||||
export async function compareJournalLegacyChatsWhenIdle(
|
||||
writer: Pick<JournalSqlOwner, 'maintain'>,
|
||||
readers: Pick<JournalSqlReaderPool, 'database' | 'compareLegacyChat'>,
|
||||
boundary: () => Promise<boolean>,
|
||||
stopped: () => boolean,
|
||||
report: (error: unknown, sessionId: string) => void
|
||||
): Promise<void> {
|
||||
let after: string | null = null
|
||||
while (await boundary()) {
|
||||
const page = await readers.database({ kind: 'legacy-compares', after })
|
||||
if (page.kind !== 'legacy-compares') {
|
||||
throw new Error('journal reader returned no legacy compares')
|
||||
}
|
||||
if (page.compares.length === 0) {
|
||||
return
|
||||
}
|
||||
for (const compare of page.compares) {
|
||||
if (!(await boundary())) {
|
||||
return
|
||||
}
|
||||
const result = await compareOnce(readers, compare.sessionId, stopped, report)
|
||||
// No original, stopping, or this file's storage failing: no other chat would fare better.
|
||||
if (result === null) {
|
||||
return
|
||||
}
|
||||
if (result === 'pending') {
|
||||
continue
|
||||
}
|
||||
if ('error' in result || result.revision === compare.revision) {
|
||||
await writer.maintain({ kind: 'legacy-compare-settle', ...compare, ...result })
|
||||
}
|
||||
}
|
||||
after = page.compares.at(-1)!.sessionId
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* This file's own failures, contention and stopping leave the compare for the next launch; damage
|
||||
* here may be this chat's alone, so only it stays pending. A reader
|
||||
* that exits gets one retry, since another request may have taken it down; a second exit is this
|
||||
* chat's, absorbed with its error so it never reruns into the same crash.
|
||||
*/
|
||||
async function compareOnce(
|
||||
readers: Pick<JournalSqlReaderPool, 'compareLegacyChat'>,
|
||||
sessionId: string,
|
||||
stopped: () => boolean,
|
||||
report: (error: unknown, sessionId: string) => void
|
||||
): Promise<Awaited<ReturnType<JournalSqlReaderPool['compareLegacyChat']>> | 'pending'> {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try {
|
||||
return await readers.compareLegacyChat(sessionId)
|
||||
} catch (error) {
|
||||
report(error, sessionId)
|
||||
if (stopped() || isSqliteStorageFailure(error)) {
|
||||
return null
|
||||
}
|
||||
if (isSqliteCorruption(error)) {
|
||||
return 'pending'
|
||||
}
|
||||
if (attempt > 0) {
|
||||
return { error: error instanceof Error ? error.message : String(error) }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -27,6 +27,8 @@ import { discardForgottenJournalChatPage } from './journal-forgotten-chats'
|
||||
import { installAgentSessionStoreRows } from '../../runtime/agent-session/records/agent-session-record-rows'
|
||||
import { storedAgentSessionBackground } from '../../runtime/agent-session/records/agent-session-tab-table'
|
||||
import type { JournalSqlStoredRecords } from './journal-sql-database-read'
|
||||
import { settleJournalLegacyCompare } from './database/journal-legacy-compare'
|
||||
import { recordJournalLegacyCopyFailure } from './database/journal-legacy-copies'
|
||||
import {
|
||||
fillJournalConnectorReplies,
|
||||
sentJournalConnectorReplies
|
||||
@@ -60,6 +62,8 @@ export type JournalSqlMaintenance =
|
||||
| { kind: 'import-finish'; header: JournalDomainImportHeader }
|
||||
| JournalDomainMaintenance
|
||||
| { kind: 'event-tail-prune'; now: number }
|
||||
| ({ kind: 'legacy-compare-settle' } & Parameters<typeof settleJournalLegacyCompare>[1])
|
||||
| { kind: 'legacy-copy-failed'; sessionId: string; error: string }
|
||||
// The reply outbox's rows outside any chat's lane: filled with outcomes, and sent.
|
||||
| { kind: 'connector-replies-fill'; fills: readonly JournalConnectorReplyFill[]; now: number }
|
||||
| { kind: 'connector-replies-sent'; keys: readonly JournalConnectorReplyKey[]; now: number }
|
||||
@@ -129,6 +133,12 @@ export function applyJournalSqlMaintenance(
|
||||
return completeJournalDomainCompaction(db, operation.now)
|
||||
case 'event-tail-prune':
|
||||
return pruneJournalSqlEventTail(db, operation.now)
|
||||
case 'legacy-compare-settle':
|
||||
settleJournalLegacyCompare(db, operation)
|
||||
return null
|
||||
case 'legacy-copy-failed':
|
||||
recordJournalLegacyCopyFailure(db, operation.sessionId, operation.error)
|
||||
return null
|
||||
case 'stored-output-compaction':
|
||||
return compactJournalStoredOutputPage(db)
|
||||
}
|
||||
|
||||
@@ -11,8 +11,10 @@ import { readJournalDomainHistoryFrame } from './journal-domain-history-page'
|
||||
import { journalDomainImportState } from './journal-domain-import-rows'
|
||||
import {
|
||||
journalLegacyChatRevision,
|
||||
journalLegacyConversionSource,
|
||||
openJournalLegacySource
|
||||
} from './database/journal-legacy-source'
|
||||
import { compareJournalLegacyChat } from './database/journal-legacy-compare'
|
||||
import { installWorkerSendLedger } from '../../worker-threads/worker-send-ledger'
|
||||
import { limitJournalReaderReply } from './journal-sql-reader-reply-limit'
|
||||
|
||||
@@ -98,22 +100,30 @@ port.on('message', ({ id, operation }: JournalSqlReaderRequest) => {
|
||||
break
|
||||
case 'conversion-prepare': {
|
||||
// Each frozen-source read is its own short statement; no snapshot spans the restore.
|
||||
if (!original && existsSync(workerData.originalFilename)) {
|
||||
original = openJournalLegacySource(workerData.originalFilename, { timeout: 5000 })
|
||||
}
|
||||
const prepared = original
|
||||
? (prepareJournalDomainImport(original, operation.sessionId) ??
|
||||
prepareEmptyConversion(original, operation.sessionId))
|
||||
const source = openOriginal()
|
||||
const from = journalLegacyConversionSource(db, operation.sessionId)
|
||||
const prepared = source
|
||||
? (prepareJournalDomainImport(source, from) ?? prepareEmptyConversion(source, from))
|
||||
: null
|
||||
if (prepared) {
|
||||
const { pages, ...header } = prepared
|
||||
conversions.set(id, pages)
|
||||
reply({ type: 'prepared', id, conversion: header })
|
||||
// A copy takes the original chat's history; the original's revision stays the original's.
|
||||
const conversion =
|
||||
from === operation.sessionId
|
||||
? header
|
||||
: { ...header, sessionId: operation.sessionId, sourceRevision: null }
|
||||
reply({ type: 'prepared', id, conversion })
|
||||
} else {
|
||||
reply({ type: 'prepared', id, conversion: null })
|
||||
}
|
||||
break
|
||||
}
|
||||
case 'legacy-compare': {
|
||||
const result = compareJournalLegacyChat(openOriginal, db, operation.sessionId)
|
||||
reply({ type: 'legacy-compare', id, result })
|
||||
break
|
||||
}
|
||||
case 'conversion-page': {
|
||||
const pages = conversions.get(operation.conversionId)
|
||||
if (!pages) {
|
||||
@@ -160,6 +170,13 @@ port.on('message', ({ id, operation }: JournalSqlReaderRequest) => {
|
||||
}
|
||||
})
|
||||
|
||||
function openOriginal(): Database.Database | undefined {
|
||||
if (!original && existsSync(workerData.originalFilename)) {
|
||||
original = openJournalLegacySource(workerData.originalFilename, { timeout: 5000 })
|
||||
}
|
||||
return original
|
||||
}
|
||||
|
||||
/** Preserve empty saved epochs independently of legacy-source retirement. */
|
||||
function prepareEmptyConversion(
|
||||
source: Database.Database,
|
||||
|
||||
@@ -4,6 +4,7 @@ import { JOURNAL_SQL_READERS } from './journal-sql-reader-requests'
|
||||
import type { JournalForkPreparationInput } from './journal-fork-preparation'
|
||||
import { throwIfSignalAborted } from '../../../shared/concurrency/abort-signal-reason'
|
||||
import type { JournalOutlineCache } from './journal-domain-outline'
|
||||
import type { JournalLegacyChatComparison } from './journal-sql-reader-protocol'
|
||||
import {
|
||||
currentWorkerEntryLayout,
|
||||
resolveWorkerThreadEntryPath
|
||||
@@ -127,6 +128,15 @@ export class JournalSqlReaderPool {
|
||||
return this.conversionSlot.prepareConversion(sessionId)
|
||||
}
|
||||
|
||||
/** Null when the original journal is gone. */
|
||||
async compareLegacyChat(sessionId: string): Promise<JournalLegacyChatComparison | null> {
|
||||
const response = await this.slot().request({ kind: 'legacy-compare', sessionId })
|
||||
if (response.type !== 'legacy-compare') {
|
||||
throw new Error('journal reader returned no legacy compare')
|
||||
}
|
||||
return response.result
|
||||
}
|
||||
|
||||
async importState(sessionId: string) {
|
||||
const response = await this.slot().request({ kind: 'import-state', sessionId })
|
||||
if (response.type !== 'import-state') {
|
||||
|
||||
@@ -16,6 +16,9 @@ import type {
|
||||
} from './journal-domain-history-page'
|
||||
import type { JournalDomainImportState } from './journal-domain-import-rows'
|
||||
|
||||
/** A chat that can't be compared answers with its error. */
|
||||
export type JournalLegacyChatComparison = { revision: string; held: boolean } | { error: string }
|
||||
|
||||
export type JournalSqlReaderOperation =
|
||||
| { kind: 'fork-prepare'; input: JournalForkPreparationInput }
|
||||
| { kind: 'fork-next'; preparationId: number }
|
||||
@@ -31,6 +34,7 @@ export type JournalSqlReaderOperation =
|
||||
| { kind: 'import-state'; sessionId: string }
|
||||
| { kind: 'conversion-page'; conversionId: number }
|
||||
| { kind: 'conversion-release'; conversionId: number }
|
||||
| { kind: 'legacy-compare'; sessionId: string }
|
||||
|
||||
export type JournalSqlReaderRequest = { id: number; operation: JournalSqlReaderOperation }
|
||||
export type JournalSqlReaderReply =
|
||||
@@ -44,6 +48,7 @@ export type JournalSqlReaderReply =
|
||||
| { type: 'import-state'; id: number; state: JournalDomainImportState }
|
||||
| { type: 'rows'; id: number; page: JournalDomainImportPage | null }
|
||||
| { type: 'released'; id: number }
|
||||
| { type: 'legacy-compare'; id: number; result: JournalLegacyChatComparison | null }
|
||||
| { type: 'failed'; id?: number; error: JournalSqlFailure }
|
||||
|
||||
export type JournalSqlReaderResult = Exclude<JournalSqlReaderReply, { type: 'ready' | 'failed' }>
|
||||
|
||||
@@ -44,10 +44,19 @@ export function limitJournalReaderReply<T extends { type: string }>(
|
||||
return {
|
||||
type: 'failed',
|
||||
id: reply.id,
|
||||
error: journalSqlFailure(new Error(`journal read reply (${reply.type}) exceeds ${limit} bytes`))
|
||||
error: journalSqlFailure(
|
||||
new Error(`${REPLY_LIMIT_FAILURE} (${reply.type}) exceeds ${limit} bytes`)
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
const REPLY_LIMIT_FAILURE = 'journal read reply'
|
||||
|
||||
/** A chat whose rows can't fit one reply fails the same way at every retry. */
|
||||
export function isJournalReaderReplyLimitFailure(error: unknown): boolean {
|
||||
return error instanceof Error && error.message.startsWith(`${REPLY_LIMIT_FAILURE} (`)
|
||||
}
|
||||
|
||||
function isRecordSetReply(reply: unknown): boolean {
|
||||
if (typeof reply !== 'object' || reply === null || !('type' in reply) || !('result' in reply)) {
|
||||
return false
|
||||
|
||||
@@ -109,6 +109,7 @@ export function describeJournalReaderOperation(
|
||||
return { operation: operation.kind, chat: chatHash(operation.input.source.sessionId) }
|
||||
case 'conversion-prepare':
|
||||
case 'import-state':
|
||||
case 'legacy-compare':
|
||||
return { operation: operation.kind, chat: chatHash(operation.sessionId) }
|
||||
case 'fork-next':
|
||||
case 'fork-release':
|
||||
|
||||
@@ -9,8 +9,7 @@ import { join } from 'node:path'
|
||||
import { afterEach, expect, it, vi } from 'vitest'
|
||||
import { codexProviderHandle } from '../../../shared/agent-session/lifecycle/provider-handles/agent-session-provider-handle-encoding'
|
||||
import { openStructuredAgentSessionSqlJournalDatabase } from '../../runtime/agent-session/installation/structured-agent-session-journal-open'
|
||||
import { journalDatabasePath } from './database/journal-host-database'
|
||||
import { journalDomainDatabasePath } from './database/journal-domain-database'
|
||||
import { journalDatabasePath, journalDomainDatabasePath } from './database/journal-host-database'
|
||||
import type { JournalSqlDatabase } from './database/journal-sql-database'
|
||||
import { SqlAgentSessionJournal } from './journal-sql-store'
|
||||
import type { JournalSqlStoreOperation } from './journal-sql-store'
|
||||
|
||||
@@ -6,9 +6,9 @@ import { mkdirSync } from 'node:fs'
|
||||
import { dirname } from 'node:path'
|
||||
import { parentPort, workerData } from 'node:worker_threads'
|
||||
import type Database from '../../sqlite/sync-database'
|
||||
import { journalDomainDatabasePath } from './database/journal-host-database'
|
||||
import {
|
||||
initializeJournalDomainDatabase,
|
||||
journalDomainDatabasePath,
|
||||
openJournalDomainDatabase
|
||||
} from './database/journal-domain-database'
|
||||
import { JournalCheckpointOwner } from './database/journal-checkpoint-owner'
|
||||
|
||||
@@ -9,6 +9,7 @@ import type { JournalRow } from './durable-rows/journal-row-schema'
|
||||
import { resolveJournalWriterStoreMutation } from './journal-writer-store-mutation'
|
||||
import { commitAgentSessionPublicationFence } from '../../runtime/agent-session/agent-session-publication-fence'
|
||||
import { markJournalChatForgotten } from './journal-forgotten-chats'
|
||||
import { absorbForgottenJournalLegacyChat } from './database/journal-legacy-compare'
|
||||
import { forgetJournalConnectorReplies } from './connector-replies/journal-connector-replies'
|
||||
|
||||
export function applyJournalWriterStoreEffect(
|
||||
@@ -34,6 +35,7 @@ export function applyJournalWriterStoreEffect(
|
||||
writeAgentSessionStoreRows(db, resolved.writes)
|
||||
if (input.kind === 'command' && input.command.kind === 'forget-idle' && resolved.value === true) {
|
||||
markJournalChatForgotten(db, input.command.sessionId)
|
||||
absorbForgottenJournalLegacyChat(db, input.command.sessionId)
|
||||
// Its connector requests are owed an outcome still: the chat no longer has them.
|
||||
forgetJournalConnectorReplies(db, input.command.sessionId, Date.now())
|
||||
}
|
||||
|
||||
@@ -6,11 +6,11 @@ import { join } from 'node:path'
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest'
|
||||
import { JournalSqlDatabase } from '../../../native-chat/agent-session-journal/database/journal-sql-database'
|
||||
import { openJournalDatabase } from '../../../native-chat/agent-session-journal/database/journal-database'
|
||||
import { journalDatabasePath } from '../../../native-chat/agent-session-journal/database/journal-host-database'
|
||||
import {
|
||||
journalDomainDatabasePath,
|
||||
openJournalDomainDatabase
|
||||
} from '../../../native-chat/agent-session-journal/database/journal-domain-database'
|
||||
journalDatabasePath,
|
||||
journalDomainDatabasePath
|
||||
} from '../../../native-chat/agent-session-journal/database/journal-host-database'
|
||||
import { openJournalDomainDatabase } from '../../../native-chat/agent-session-journal/database/journal-domain-database'
|
||||
import {
|
||||
journalSqlReaderPath,
|
||||
journalSqlWorkerPath
|
||||
|
||||
@@ -2,8 +2,10 @@ import { sessionCrewChanges } from '../../orchestration/crew/crew-changes'
|
||||
// Launch admission opens the single-writer store before chat-host installation; both must reuse this instance.
|
||||
|
||||
import { existsSync } from 'node:fs'
|
||||
import { journalDatabasePath } from '../../../native-chat/agent-session-journal/database/journal-host-database'
|
||||
import { journalDomainDatabasePath } from '../../../native-chat/agent-session-journal/database/journal-domain-database'
|
||||
import {
|
||||
journalDatabasePath,
|
||||
journalDomainDatabasePath
|
||||
} from '../../../native-chat/agent-session-journal/database/journal-host-database'
|
||||
import { journalDatabaseHoldsAgentSessions } from '../../../native-chat/agent-session-journal/database/journal-database'
|
||||
import { AgentSessionRecordStore } from './agent-session-record-store'
|
||||
import type { JournalSqlDatabase } from '../../../native-chat/agent-session-journal/database/journal-sql-database'
|
||||
|
||||
@@ -43,6 +43,29 @@ export function isTransientSqliteContention(error: unknown): boolean {
|
||||
return CONTENTION_MESSAGE.test(errorText(error))
|
||||
}
|
||||
|
||||
const SQLITE_NOMEM = 7
|
||||
const SQLITE_READONLY = 8
|
||||
const SQLITE_IOERR = 10
|
||||
const SQLITE_FULL = 13
|
||||
const STORAGE_ERRCODES = new Set([
|
||||
SQLITE_BUSY,
|
||||
SQLITE_LOCKED,
|
||||
SQLITE_NOMEM,
|
||||
SQLITE_READONLY,
|
||||
SQLITE_IOERR,
|
||||
SQLITE_FULL,
|
||||
SQLITE_CANTOPEN
|
||||
])
|
||||
|
||||
/**
|
||||
* The database or its disk failed, not the row being written: every other row would fail too.
|
||||
* Corruption is the caller's call: it may be one row's page or the whole file.
|
||||
*/
|
||||
export function isSqliteStorageFailure(error: unknown): boolean {
|
||||
const errcode = primaryErrcode(error)
|
||||
return (errcode !== null && STORAGE_ERRCODES.has(errcode)) || isTransientSqliteContention(error)
|
||||
}
|
||||
|
||||
export type SqliteReadFailureKind =
|
||||
/** Another connection holds the lock; the same read can succeed later. */
|
||||
| 'contended'
|
||||
|
||||
@@ -27,7 +27,7 @@ import { wslHostsGateOpen } from '../host-lifecycle/wsl-host-gate'
|
||||
import { isWslDriveUncConversionComplete } from './wsl-drive-unc-conversion-renderer'
|
||||
import { getHistoryRoot, listWslHistoryRoots } from '../../pty/history/terminal-history-paths'
|
||||
import { orchestrationDbFile } from '../../runtime/orchestration/db/orchestration-db-file'
|
||||
import { journalDomainDatabasePath } from '../../native-chat/agent-session-journal/database/journal-domain-database'
|
||||
import { journalDomainDatabasePath } from '../../native-chat/agent-session-journal/database/journal-host-database'
|
||||
import { rewriteWslDriveUncState } from './wsl-drive-unc-state-rewrite'
|
||||
import { WSL_DRIVE_UNC_STATE_DISPOSITIONS } from '../../../shared/wsl-move/wsl-drive-unc-state-dispositions'
|
||||
|
||||
|
||||
@@ -1,9 +1,7 @@
|
||||
import path from 'node:path'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import {
|
||||
journalDomainDatabasePath,
|
||||
openJournalDomainDatabase
|
||||
} from '../../../../src/main/native-chat/agent-session-journal/database/journal-domain-database'
|
||||
import { openJournalDomainDatabase } from '../../../../src/main/native-chat/agent-session-journal/database/journal-domain-database'
|
||||
import { journalDomainDatabasePath } from '../../../../src/main/native-chat/agent-session-journal/database/journal-host-database'
|
||||
import { OrchestrationDb } from '../../../../src/main/runtime/orchestration/db/orchestration-db'
|
||||
import {
|
||||
agentSessionRecordFixture,
|
||||
|
||||
Reference in New Issue
Block a user