From 65631e449af36bba24b6a2d6c331a3c184e7a3e2 Mon Sep 17 00:00:00 2001 From: Jinwoo Hong <73622457+Jinwoo-H@users.noreply.github.com> Date: Wed, 9 Sep 2026 03:36:41 -0400 Subject: [PATCH] refactor(ai-vault): split the session scanner into a transcript reader and consumers (#19666) * refactor(ai-vault): split the session scanner into a transcript reader and consumers The scanner's only output was the Session History summary; a second reader of the same transcripts (a search index) had nowhere to plug in without hooking the parse itself. Extract a reader that owns each file read, keeps the resumable cursor and publishes every decoded message to registered consumers. The session list stays a fold inside the parser and is the only consumer here. Parsers take an optional message sink instead of a scope. Also: read Cursor chats///meta.json for cwd, title and timestamps (Cursor transcripts carry only role and message); share the lazily spawned worker-thread host between the OpenCode SQLite reader and the port-scan probe; probe OpenCode's schema before querying; keep the newest-N discovery set with a bounded insert instead of sort+slice. Session list output is byte-identical to main across all 18 providers cold and append-resumed; the one Cursor session gains cwd/timestamps from meta.json. * fix(ai-vault): serialize per-path parses and report unpublished reads Overlapping parses of one transcript share the cached resume point's message channel, so the second beginRead dropped the first read's consumers and the first finishRead handed them the wrong outcome. Two callers really do overlap: a forced refresh restarts a scan while the aborted scan's parse is still in flight, and the title reader parses outside any scan. Restore the per-path lane around the whole lookup-read-store sequence. OpenCode's SQLite sessions are decoded on a worker thread the channel cannot reach, so their reads published no messages while reporting a complete span. Finish those reads as incomplete instead, so a consumer never records a cursor for a stream it did not receive. * fix(ai-vault): degrade a refused cursor chats read instead of dropping sessions A refused WSL read of Cursor's chats tree rethrew, and the per-file catch in discovery then recorded an issue and skipped the transcript. Before the meta.json join Cursor had no content dependency, so a stalled distro could not hide a Cursor session at all. Degrade to no metadata for the scan and report the chats root once. The parse cache stays honest without the throw: discovery stats no meta.json on a refused scan, so the entry's recorded size omits it and the next healthy scan re-reads the transcript. The per-scan index scope covered discovery only, so every Cursor finalize re-read the chats root to validate the module cache. Move the scope to scanAiVaultSessions, which spans discovery and parse. Also drop the unused signal parameters the sink threading added to the Devin and Hermes content parsers, by giving each file parser a private record parser instead. * fix(ai-vault): do not cache a cursor parse whose meta.json read was refused Discovery stats meta.json into the candidate's cache key, so when only the meta.json read is refused the un-enriched session was stored under a key that looks unchanged and reuseCachedSession never re-ran the enrich hook. The session stayed without cwd until Cursor rewrote the file. The enrich hook now reports 'refused', the resumable state exposes isCacheable, and the parse cache drops the entry instead of storing it, so the next healthy scan re-parses. The index-read branch is unaffected: it never stats meta.json, so its key is honest already. * fix(ai-vault): separate the transcript's size from its cache key sizeBytes folds a content dependency's size in, so it is a cache key rather than a file length. The reader compared a transcript byte offset against it and reported it as a whole-file read offset, which for Cline handed consumers an offset past the end of the file it read. Carry the dependency's own size on FileWithMtime and subtract it in the reader. A refused sibling stat rethrew, so discovery recorded an issue and skipped the transcript, the same drop removed for the readdir and read paths. Degrade to no dependency, note the tree once, and mark the key untrustworthy. An untrustworthy key no longer costs the resume cursor: the entry is stored under an mtime no stat can produce, so unchanged is false while the resume point survives and the next scan resumes instead of re-reading the whole transcript. * test(ai-vault): pin the untrustworthy-key mechanism, not just its effect Both refusal tests asserted that a later healthy scan re-enriches, which a plain store would also satisfy once the resume cursor was preserved. Assert the cache entry directly: its mtime is the unmatchable sentinel and its resume point survives. The sentinel is exported so the tests name the contract instead of repeating -1. * refactor(ai-vault): track a session's sidecar file apart from its transcript Folding Cursor's meta.json stat into the transcript's mtime/size made one key mean two things, and every round of review found another consequence: a byte offset could not be compared against it, a refused sibling read took the transcript down with it, and an un-enriched parse cached under it looked current forever. Main already had the answer for a file the transcript key cannot see: Codex titles are refreshed at reuse time over the cached session, not folded into the key. Discovery now records the sibling as its own observation, unknown when it could not be read. A cache hit needs both the transcript key and the sidecar to match. When only the sidecar moved, Cursor re-merges it over the stored un-enriched fold result and never re-reads the transcript; Cline, which reads its sibling as part of the parse, re-parses. Merging over the fold result rather than the accumulator makes enrichment pure, so a meta.json rewritten with a new cwd replaces the old one instead of losing to it. That was unreachable while the merge used ??= on a session it had already enriched. Cline and the remote scanner move to the same field, so the fold is gone from both discovery paths. * fix(ai-vault): tell an absent sidecar from an unreadable one Three places collapsed the two. sidecarUnchanged returned true for any observed 'none' without reading the entry, so a sidecar that was deleted, or one that was unreadable last scan, both read as cache hits. Native discovery mapped every non-WSL stat failure to 'none', so an EACCES on meta.json left a session enriched from a file nobody can see, with no scan issue. Remote discovery could not tell a missing sibling from a failed stat, because statRemoteSessionFile returns null for both. 'none' is now a claim: absent-now is a hit only when it was absent before or the agent never had a sidecar, and only ENOENT/ENOTDIR reads as absent. statRemoteSessionFile grows an opt-in rethrow so its caller can distinguish the two failures it already reports. Also rewrites three comments in the cursor chat-meta reader that still described the deleted fold. --- src/main/ai-vault/remote-session-file-stat.ts | 14 +- .../ai-vault/remote-session-parse-cache.ts | 24 +- .../remote-session-scanner-discovery.ts | 50 +- ...remote-session-sidecar-observation.test.ts | 84 +++ .../ai-vault/session-newest-files.test.ts | 31 + src/main/ai-vault/session-newest-files.ts | 50 ++ .../session-parse-cache-persistence.test.ts | 38 ++ .../session-parse-cache-persistence.ts | 23 +- .../ai-vault/session-parse-cache-store.ts | 111 ++++ src/main/ai-vault/session-parse-file-lane.ts | 28 + .../ai-vault/session-scanner-accumulator.ts | 56 +- .../ai-vault/session-scanner-agent-parser.ts | 61 +- .../ai-vault/session-scanner-agent-sources.ts | 9 +- .../session-scanner-antigravity-parser.ts | 14 +- .../ai-vault/session-scanner-candidates.ts | 46 ++ .../ai-vault/session-scanner-cline-parser.ts | 16 +- .../ai-vault/session-scanner-codex-parser.ts | 48 +- .../session-scanner-codex-session-meta.ts | 22 + .../session-scanner-copilot-parser.ts | 21 +- .../session-scanner-cursor-chat-meta.test.ts | 552 ++++++++++++++++++ .../session-scanner-cursor-chat-meta.ts | 294 ++++++++++ .../ai-vault/session-scanner-cursor-parser.ts | 21 +- .../ai-vault/session-scanner-devin-parser.ts | 23 +- .../ai-vault/session-scanner-discovery.ts | 163 ++++-- .../ai-vault/session-scanner-droid-parser.ts | 21 +- .../session-scanner-gemini-parsers.ts | 34 +- .../ai-vault/session-scanner-graph-parsers.ts | 20 +- .../ai-vault/session-scanner-grok-parser.ts | 6 +- .../ai-vault/session-scanner-hermes-parser.ts | 24 +- .../ai-vault/session-scanner-kimi-parser.ts | 6 +- .../session-scanner-opencode-parser.ts | 6 +- .../session-scanner-opencode-sqlite-schema.ts | 29 + ...n-scanner-opencode-sqlite-worker-client.ts | 118 +--- .../session-scanner-opencode-sqlite.ts | 30 +- ...session-scanner-parse-cache-agents.test.ts | 55 ++ .../ai-vault/session-scanner-parse-cache.ts | 352 ++++------- .../session-scanner-primary-parsers.ts | 24 +- .../session-scanner-sidecar-enrichment.ts | 79 +++ .../session-scanner-source-discovery.ts | 2 + src/main/ai-vault/session-scanner-types.ts | 11 +- src/main/ai-vault/session-scanner.ts | 188 +++--- .../ai-vault/session-sidecar-stat.test.ts | 74 +++ src/main/ai-vault/session-sidecar-stat.ts | 49 ++ .../ai-vault/session-title-file-reader.ts | 3 + .../ai-vault/session-transcript-channel.ts | 92 +++ .../session-transcript-consumers.test.ts | 330 +++++++++++ .../ai-vault/session-transcript-consumers.ts | 85 +++ ...session-transcript-message-content.test.ts | 61 ++ .../session-transcript-message-content.ts | 138 +++++ .../ai-vault/session-transcript-reader.ts | 150 +++++ src/main/lazy-worker-thread-host.ts | 102 ++++ src/main/ports/port-scan-command-client.ts | 121 +--- 52 files changed, 3265 insertions(+), 744 deletions(-) create mode 100644 src/main/ai-vault/remote-session-sidecar-observation.test.ts create mode 100644 src/main/ai-vault/session-newest-files.test.ts create mode 100644 src/main/ai-vault/session-newest-files.ts create mode 100644 src/main/ai-vault/session-parse-cache-store.ts create mode 100644 src/main/ai-vault/session-parse-file-lane.ts create mode 100644 src/main/ai-vault/session-scanner-candidates.ts create mode 100644 src/main/ai-vault/session-scanner-codex-session-meta.ts create mode 100644 src/main/ai-vault/session-scanner-cursor-chat-meta.test.ts create mode 100644 src/main/ai-vault/session-scanner-cursor-chat-meta.ts create mode 100644 src/main/ai-vault/session-scanner-opencode-sqlite-schema.ts create mode 100644 src/main/ai-vault/session-scanner-sidecar-enrichment.ts create mode 100644 src/main/ai-vault/session-sidecar-stat.test.ts create mode 100644 src/main/ai-vault/session-sidecar-stat.ts create mode 100644 src/main/ai-vault/session-transcript-channel.ts create mode 100644 src/main/ai-vault/session-transcript-consumers.test.ts create mode 100644 src/main/ai-vault/session-transcript-consumers.ts create mode 100644 src/main/ai-vault/session-transcript-message-content.test.ts create mode 100644 src/main/ai-vault/session-transcript-message-content.ts create mode 100644 src/main/ai-vault/session-transcript-reader.ts create mode 100644 src/main/lazy-worker-thread-host.ts diff --git a/src/main/ai-vault/remote-session-file-stat.ts b/src/main/ai-vault/remote-session-file-stat.ts index 57e6d2c8ed6..1453af3b1b8 100644 --- a/src/main/ai-vault/remote-session-file-stat.ts +++ b/src/main/ai-vault/remote-session-file-stat.ts @@ -13,7 +13,13 @@ export async function statRemoteSessionFile( agent: AiVaultAgent, executionHostId: ExecutionHostId, issues: AiVaultScanIssue[], - options?: { missingIsExpected?: boolean; signal?: AbortSignal } + options?: { + missingIsExpected?: boolean + signal?: AbortSignal + // Lets a caller tell a missing path from a failed stat, which both report + // as null; the issue is recorded either way before this rethrows. + rethrowFailures?: boolean + } ): Promise { try { throwIfAiVaultScanCancelled(options?.signal) @@ -31,7 +37,8 @@ export async function statRemoteSessionFile( } } catch (error) { throwIfAiVaultScanCancelled(options?.signal) - if (!options?.missingIsExpected || !isMissingRemoteSessionPathError(error)) { + const missing = isMissingRemoteSessionPathError(error) + if (!options?.missingIsExpected || !missing) { recordSessionScanIssue(issues, { executionHostId, agent, @@ -39,6 +46,9 @@ export async function statRemoteSessionFile( message: errorMessage(error) }) } + if (options?.rethrowFailures && !missing) { + throw error + } return null } } diff --git a/src/main/ai-vault/remote-session-parse-cache.ts b/src/main/ai-vault/remote-session-parse-cache.ts index 7d8b0ebe2c7..c892c4961ef 100644 --- a/src/main/ai-vault/remote-session-parse-cache.ts +++ b/src/main/ai-vault/remote-session-parse-cache.ts @@ -1,5 +1,6 @@ import type { AiVaultSession } from '../../shared/ai-vault-types' import type { RemoteScannerContext, RemoteSessionCandidate } from './remote-session-scanner-types' +import { sidecarUnchanged, type SessionSidecarObservation } from './session-sidecar-stat' // Matches the local scanner's cap. The relay sidecar is forked with // --max-old-space-size=384, and a retained session row is a title, a preview @@ -11,6 +12,7 @@ type RemoteSessionParseCacheEntry = { mtimeMs: number sizeBytes: number | null hostKey: string + sidecar?: SessionSidecarObservation session: AiVaultSession | null } @@ -55,13 +57,15 @@ function storeEntry(path: string, entry: RemoteSessionParseCacheEntry): void { * (#13753). The local scanner has had `parseAgentSessionFileCached` for exactly * this reason; this is its remote counterpart. * - * `(mtimeMs, sizeBytes)` is a sound validity key here because discovery already - * folds a source's `contentDependencyPath` stat into both fields - * (remote-session-scanner-discovery.ts), so a metadata-only transcript whose - * companion file changed still looks changed. Sources whose parse reads a file - * discovery does not stat — Codex looks its title up in `session_index.jsonl` — - * are not covered by that key and pass `refreshReusedSession` to re-derive the - * uncovered part without touching the transcript. + * `(mtimeMs, sizeBytes)` covers the transcript, and the sidecar observation + * discovery records beside it (remote-session-scanner-discovery.ts) covers a + * source's companion file, so a metadata-only transcript whose companion + * changed still looks changed. Remote Cline is the only such source; remote + * Cursor streams transcript content with no sibling to read. Sources whose + * parse reads a file discovery does not stat — Codex looks its title up in + * `session_index.jsonl` — are not covered by either and pass + * `refreshReusedSession` to re-derive the uncovered part without touching the + * transcript. * * Only a completed parse is stored. A read that threw stays uncached so a * transient filesystem failure cannot pin a wrong answer for the corpus's life. @@ -80,7 +84,10 @@ export async function parseRemoteSessionFileCached(args: { entry !== undefined && entry.hostKey === args.hostKey && entry.mtimeMs === file.mtimeMs && - (entry.sizeBytes === null || file.sizeBytes === undefined || entry.sizeBytes === file.sizeBytes) + (entry.sizeBytes === null || + file.sizeBytes === undefined || + entry.sizeBytes === file.sizeBytes) && + sidecarUnchanged(entry.sidecar, file.sidecar) if (unchanged) { if (args.stats) { args.stats.reused++ @@ -101,6 +108,7 @@ export async function parseRemoteSessionFileCached(args: { mtimeMs: file.mtimeMs, sizeBytes: file.sizeBytes ?? null, hostKey: args.hostKey, + sidecar: file.sidecar, session }) return session diff --git a/src/main/ai-vault/remote-session-scanner-discovery.ts b/src/main/ai-vault/remote-session-scanner-discovery.ts index 159bf049da6..1fb25213f32 100644 --- a/src/main/ai-vault/remote-session-scanner-discovery.ts +++ b/src/main/ai-vault/remote-session-scanner-discovery.ts @@ -4,6 +4,7 @@ import type { ExecutionHostId } from '../../shared/execution-host' import { joinRemotePath } from '../ssh/ssh-remote-platform' import { isMissingRemoteSessionPathError, statRemoteSessionFile } from './remote-session-file-stat' import type { FileWithMtime } from './session-scanner-types' +import type { SessionSidecarObservation } from './session-sidecar-stat' import { errorMessage } from './session-scanner-values' import { mapRemoteScanBatches } from './remote-session-scan-batching' import { throwIfAiVaultScanCancelled } from './ai-vault-scan-cancellation' @@ -61,23 +62,38 @@ async function statRemoteCandidateFile( if (!file || !source.contentDependencyPath) { return file } - const dependency = await statRemoteSessionFile( - context.provider, - source.contentDependencyPath(path), - source.agent, - context.executionHostId, - issues, - { missingIsExpected: true, signal: context.signal } - ) - if (!dependency) { - return file - } - const mtimeMs = Math.max(file.mtimeMs, dependency.mtimeMs) - return { - ...file, - mtimeMs, - modifiedAt: new Date(mtimeMs).toISOString(), - sizeBytes: (file.sizeBytes ?? 0) + (dependency.sizeBytes ?? 0) + const sidecarPath = source.contentDependencyPath(path) + // Recorded beside the transcript's own stat, never folded into it: one key + // cannot mean both "the transcript grew" and "the sibling changed". + return { ...file, sidecar: await observeRemoteSidecar(source, context, sidecarPath, issues) } +} + +/** + * A stat that failed for any reason other than a missing path is `'unknown'`, + * not `'none'`: serving the cached session over an unreadable sibling would + * publish metadata nobody can currently see. `statRemoteSessionFile` already + * recorded the issue for the failure. + */ +async function observeRemoteSidecar( + source: RemoteSessionSource, + context: RemoteScannerContext, + sidecarPath: string, + issues: AiVaultScanIssue[] +): Promise { + try { + const sidecar = await statRemoteSessionFile( + context.provider, + sidecarPath, + source.agent, + context.executionHostId, + issues, + { missingIsExpected: true, signal: context.signal, rethrowFailures: true } + ) + return sidecar + ? { path: sidecarPath, mtimeMs: sidecar.mtimeMs, sizeBytes: sidecar.sizeBytes ?? 0 } + : 'none' + } catch { + return 'unknown' } } diff --git a/src/main/ai-vault/remote-session-sidecar-observation.test.ts b/src/main/ai-vault/remote-session-sidecar-observation.test.ts new file mode 100644 index 00000000000..92ff41775ec --- /dev/null +++ b/src/main/ai-vault/remote-session-sidecar-observation.test.ts @@ -0,0 +1,84 @@ +import { describe, expect, it } from 'vitest' +import { getRemoteHostPlatform } from '../ssh/ssh-remote-platform' +import { scanRemoteAiVaultSessions } from './remote-session-scanner' +import { MemoryRemoteProvider } from './remote-session-scanner-test-fixtures' +import { resetRemoteSessionParseCacheForTests } from './remote-session-parse-cache' + +describe('remote sidecar observations', () => { + const sessionId = '1786466194549_sidecar' + const sessionDir = `/home/ada/.cline/data/sessions/${sessionId}` + const messagesPath = `${sessionDir}/${sessionId}.messages.json` + + function addCline(provider: MemoryRemoteProvider, firstPrompt: string): void { + provider.addFile( + `${sessionDir}/${sessionId}.json`, + JSON.stringify({ + version: 1, + session_id: sessionId, + started_at: '2026-08-11T16:36:34.551Z', + cwd: '/home/ada/repo' + }), + 10 + ) + provider.addFile( + messagesPath, + JSON.stringify({ + version: 1, + updated_at: '2026-08-11T16:38:00.000Z', + sessionId, + messages: [{ role: 'user', content: [{ type: 'text', text: firstPrompt }] }] + }), + 11 + ) + } + + const scan = (provider: MemoryRemoteProvider): ReturnType => + scanRemoteAiVaultSessions({ + provider, + executionHostId: 'ssh:dev-box', + remoteHome: '/home/ada', + hostPlatform: getRemoteHostPlatform('linux-x64') + }) + + it('re-parses when a messages-file stat fails rather than serving the cached row', async () => { + resetRemoteSessionParseCacheForTests() + const provider = new MemoryRemoteProvider() + addCline(provider, 'first prompt') + expect((await scan(provider)).sessions[0]).toMatchObject({ title: 'first prompt' }) + + // The sidecar changed underneath, and its stat now fails for a reason that + // is not "missing": nothing about it may be assumed, so the row is re-read. + addCline(provider, 'second prompt') + provider.failStat( + messagesPath, + Object.assign(new Error('permission denied'), { code: 'EACCES' }) + ) + + const refused = await scan(provider) + + expect(refused.sessions[0]).toMatchObject({ title: 'second prompt' }) + expect(refused.issues.map((issue) => issue.path)).toContain(messagesPath) + }) + + it('treats a genuinely missing messages file as no sidecar, not as unknown', async () => { + resetRemoteSessionParseCacheForTests() + const provider = new MemoryRemoteProvider() + provider.addFile( + `${sessionDir}/${sessionId}.json`, + JSON.stringify({ + version: 1, + session_id: sessionId, + started_at: '2026-08-11T16:36:34.551Z', + cwd: '/home/ada/repo' + }), + 10 + ) + + const first = await scan(provider) + const second = await scan(provider) + + expect(first.issues).toEqual([]) + expect(second.issues).toEqual([]) + expect(second.sessions[0]?.sessionId).toBe(sessionId) + }) +}) diff --git a/src/main/ai-vault/session-newest-files.test.ts b/src/main/ai-vault/session-newest-files.test.ts new file mode 100644 index 00000000000..092def09a16 --- /dev/null +++ b/src/main/ai-vault/session-newest-files.test.ts @@ -0,0 +1,31 @@ +import { expect, it } from 'vitest' +import { SessionNewestFiles } from './session-newest-files' +import type { FileWithMtime } from './session-scanner-types' + +function file(i: number): FileWithMtime { + const mtimeMs = (i * 7919) % 997 + return { path: String(i), mtimeMs, modifiedAt: new Date(mtimeMs).toISOString() } +} + +it('retains at most 12 of 100,000 candidates with stable newest-first ties', () => { + const all = Array.from({ length: 100_000 }, (_, i) => file(i)) + const retained = new SessionNewestFiles(12) + let peak = 0 + for (const candidate of all) { + retained.add(candidate) + peak = Math.max(peak, retained.size) + } + expect(peak).toBe(12) + expect(retained.newest()).toEqual(all.sort((a, b) => b.mtimeMs - a.mtimeMs).slice(0, 12)) +}) + +it('supports full backfill and empty requests', () => { + const all = new SessionNewestFiles(Infinity) + const none = new SessionNewestFiles(0) + for (let i = 0; i < 100; i++) { + all.add(file(i)) + none.add(file(i)) + } + expect(all.newest()).toHaveLength(100) + expect(none.newest()).toEqual([]) +}) diff --git a/src/main/ai-vault/session-newest-files.ts b/src/main/ai-vault/session-newest-files.ts new file mode 100644 index 00000000000..c315c26dfad --- /dev/null +++ b/src/main/ai-vault/session-newest-files.ts @@ -0,0 +1,50 @@ +import type { FileWithMtime } from './session-scanner-types' + +/** Retain only the requested newest files, preserving traversal order on ties. */ +export class SessionNewestFiles { + private readonly files: FileWithMtime[] = [] + + private readonly limit: number + + constructor(limit: number) { + this.limit = Math.max(0, Math.trunc(limit) || 0) + } + + add(file: FileWithMtime): void { + // The backfill enumerates with no limit; skip the insert search entirely. + if (!Number.isFinite(this.limit)) { + this.files.push(file) + return + } + if (this.limit <= 0) { + return + } + const last = this.files.at(-1) + if (this.files.length >= this.limit && last && file.mtimeMs <= last.mtimeMs) { + return + } + let low = 0 + let high = this.files.length + while (low < high) { + const middle = (low + high) >>> 1 + if (this.files[middle].mtimeMs >= file.mtimeMs) { + low = middle + 1 + } else { + high = middle + } + } + this.files.splice(low, 0, file) + if (this.files.length > this.limit) { + this.files.pop() + } + } + + get size(): number { + return this.files.length + } + + /** The unbounded path appends in traversal order, so the sort is not redundant. */ + newest(): FileWithMtime[] { + return [...this.files].sort((a, b) => b.mtimeMs - a.mtimeMs) + } +} diff --git a/src/main/ai-vault/session-parse-cache-persistence.test.ts b/src/main/ai-vault/session-parse-cache-persistence.test.ts index 27aac09ce22..ee617a2352d 100644 --- a/src/main/ai-vault/session-parse-cache-persistence.test.ts +++ b/src/main/ai-vault/session-parse-cache-persistence.test.ts @@ -29,9 +29,12 @@ import { parseAgentSessionFileCached, resetSessionParseCacheForTests, seedSessionParseCache, + snapshotSessionParseCacheForPersistence, type PersistedSessionParseCacheEntry, type SessionParseStats } from './session-scanner-parse-cache' +import { getSessionParseCacheEntry } from './session-parse-cache-store' +import type { SessionSidecarObservation } from './session-sidecar-stat' import { isolatedScanRoots } from './session-scanner-test-fixtures' import { parseClaudeSessionFile } from './session-scanner-primary-parsers' import type { FileWithMtime, SessionFileCandidate } from './session-scanner-types' @@ -526,3 +529,38 @@ describe('session parse cache persistence', () => { debugSpy.mockRestore() }) }) + +describe('sidecar observations survive the round trip', () => { + const OBSERVATIONS: [string, SessionSidecarObservation | undefined][] = [ + ['an object', { path: '/chats/a/meta.json', mtimeMs: 42, sizeBytes: 7 }], + ['none', 'none'], + ['unknown', 'unknown'], + ['absent', undefined] + ] + + it.each(OBSERVATIONS)('restores %s exactly', async (_label, sidecar) => { + const root = await makeTempDir() + const cacheFile = join(root, 'session-parse-cache.json') + const path = await writeTranscript(root) + initSessionParseCachePersistence({ filePath: cacheFile, appVersion: APP_VERSION }) + await ensureSessionParseCacheLoaded() + + const stats = createSessionParseStats() + await parseAgentSessionFileCached(await claudeCandidate(path), process.platform, stats) + const seeded = snapshotSessionParseCacheForPersistence().map( + ([entryPath, entry]): [string, PersistedSessionParseCacheEntry] => [ + entryPath, + sidecar === undefined ? entry : { ...entry, sidecar } + ] + ) + resetSessionParseCacheForTests() + seedSessionParseCache(seeded) + scheduleSessionParseCachePersist(stats) + await flushSessionParseCachePersistForTests() + + simulateRestart(cacheFile) + await ensureSessionParseCacheLoaded() + + expect(getSessionParseCacheEntry(path)?.sidecar).toEqual(sidecar) + }) +}) diff --git a/src/main/ai-vault/session-parse-cache-persistence.ts b/src/main/ai-vault/session-parse-cache-persistence.ts index 0a0cd44095e..a394ea2ad91 100644 --- a/src/main/ai-vault/session-parse-cache-persistence.ts +++ b/src/main/ai-vault/session-parse-cache-persistence.ts @@ -11,6 +11,7 @@ import { type PersistedSessionParseCacheEntry, type SessionParseStats } from './session-scanner-parse-cache' +import type { SessionSidecarObservation } from './session-sidecar-stat' // Bump when the persisted entry layout or cached session semantics change; a // mismatched file is discarded whole. @@ -181,17 +182,37 @@ function parsePersistedEntry(item: unknown): [string, PersistedSessionParseCache if (entry.session !== null && typeof entry.session !== 'object') { return null } + const sidecar = parsePersistedSidecar(entry.sidecar) return [ path, { mtimeMs: entry.mtimeMs, sizeBytes: entry.sizeBytes, platform: entry.platform as NodeJS.Platform, - session: entry.session as PersistedSessionParseCacheEntry['session'] + session: entry.session as PersistedSessionParseCacheEntry['session'], + ...(sidecar === undefined ? {} : { sidecar }) } ] } +// Why: added after SCHEMA_VERSION 2 shipped, so a file an older build wrote has +// no such field. Absent (or unreadable) means unknown, which costs one re-parse +// of the rows that have a sibling and nothing at all for the rest. +function parsePersistedSidecar(value: unknown): SessionSidecarObservation | undefined { + if (value === 'none' || value === 'unknown') { + return value + } + if (typeof value !== 'object' || value === null) { + return undefined + } + const record = value as Record + return typeof record.path === 'string' && + typeof record.mtimeMs === 'number' && + typeof record.sizeBytes === 'number' + ? { path: record.path, mtimeMs: record.mtimeMs, sizeBytes: record.sizeBytes } + : undefined +} + async function persistSnapshot(current: SessionParseCachePersistenceOptions): Promise { const directory = dirname(current.filePath) const tempPath = join(directory, `session-parse-cache-${process.pid}-${Date.now()}.tmp`) diff --git a/src/main/ai-vault/session-parse-cache-store.ts b/src/main/ai-vault/session-parse-cache-store.ts new file mode 100644 index 00000000000..1f1d624c85d --- /dev/null +++ b/src/main/ai-vault/session-parse-cache-store.ts @@ -0,0 +1,111 @@ +import type { AiVaultSession } from '../../shared/ai-vault-types' +import type { ResumableSessionParseState } from './session-scanner-types' +import type { SessionSidecarObservation } from './session-sidecar-stat' +import type { TranscriptMessageChannel } from './session-transcript-channel' + +// Sized past the default recency cap (1000) plus the in-scope cap (2000) so a +// full steady-state result set stays resident between forced rescans. +const MAX_CACHE_ENTRIES = 4096 + +export type SessionParseResumePoint = { + state: ResumableSessionParseState + // Byte offset just past the last complete ('\n'-terminated) line consumed; + // a trailing unterminated line is deliberately left before this point. + byteOffset: number + // Bound to the cached state, which keeps the reference its parsers were built + // with; a resumed read re-points this channel instead of replacing it. + channel: TranscriptMessageChannel +} + +export type SessionParseCacheEntry = { + mtimeMs: number + sizeBytes: number | null + platform: NodeJS.Platform + session: AiVaultSession | null + // What the sibling file looked like when `session` was built. Tracked apart + // from the transcript's key so each can go stale on its own. + sidecar?: SessionSidecarObservation + // The session the transcript alone produced, before any sibling was merged + // onto it. In-memory only: without it a sibling change costs one re-parse. + foldSession?: AiVaultSession | null + resume: SessionParseResumePoint | null +} + +const cache = new Map() + +export function resetSessionParseCacheForTests(): void { + cache.clear() +} + +// Drops one entry after its file is deleted. Cleanliness, not correctness: +// discovery walks disk first, so a trashed file is never rediscovered anyway. +export function invalidateSessionParseCacheEntry(path: string): void { + cache.delete(path) +} + +// Persisted subset of a cache entry: the non-serializable `resume` parser +// state is dropped, and `foldSession` with it, so a restart pays one re-parse +// for a session whose sibling moved rather than storing every row twice +// (see session-parse-cache-persistence.ts). +export type PersistedSessionParseCacheEntry = Omit + +export function snapshotSessionParseCacheForPersistence(): [ + string, + PersistedSessionParseCacheEntry +][] { + return [...cache].map(([path, entry]): [string, PersistedSessionParseCacheEntry] => [ + path, + { + mtimeMs: entry.mtimeMs, + sizeBytes: entry.sizeBytes, + platform: entry.platform, + session: entry.session, + ...(entry.sidecar === undefined ? {} : { sidecar: entry.sidecar }) + } + ]) +} + +// Seeded entries carry `resume: null`: after a restart an unchanged file is a +// cache hit; a file that changed while the app was closed pays one full +// (not incremental) re-parse. +export function seedSessionParseCache( + entries: Iterable<[string, PersistedSessionParseCacheEntry]> +): void { + const list = [...entries] + // Snapshot order is oldest→newest (LRU); an over-cap list keeps the newest + // tail rather than seeding the oldest entries and dropping the tail. + for (const [path, entry] of list.slice(Math.max(0, list.length - MAX_CACHE_ENTRIES))) { + if (cache.size >= MAX_CACHE_ENTRIES) { + return + } + // In-process entries are always fresher than persisted ones; never clobber. + if (cache.has(path)) { + continue + } + cache.set(path, { + mtimeMs: entry.mtimeMs, + sizeBytes: entry.sizeBytes, + platform: entry.platform, + session: entry.session, + // Absent in files an older build wrote; `sidecarUnchanged` reads that as + // unknown, so such a row re-enriches on its first scan. + sidecar: entry.sidecar, + resume: null + }) + } +} + +export function getSessionParseCacheEntry(path: string): SessionParseCacheEntry | undefined { + return cache.get(path) +} + +export function storeSessionParseCacheEntry(path: string, entry: SessionParseCacheEntry): void { + cache.delete(path) + cache.set(path, entry) + if (cache.size > MAX_CACHE_ENTRIES) { + const oldest = cache.keys().next() + if (!oldest.done) { + cache.delete(oldest.value) + } + } +} diff --git a/src/main/ai-vault/session-parse-file-lane.ts b/src/main/ai-vault/session-parse-file-lane.ts new file mode 100644 index 00000000000..34722300834 --- /dev/null +++ b/src/main/ai-vault/session-parse-file-lane.ts @@ -0,0 +1,28 @@ +const pending = new Map>() + +/** + * Serializes parses of one transcript path. + * + * Two callers really do overlap on the same file: a forced refresh aborts the + * running scan while its in-flight parse keeps going as the replacement scan + * starts it again, and `session-title-file-reader.ts` parses on its own, with + * no scan involved. Overlapping reads share the cached resume point's message + * channel, so the second `beginRead` would drop the first read's consumers and + * the first `finishRead` would hand them the wrong outcome; the later store + * could also move the cursor backwards. + */ +export async function inSessionParseFileLane(path: string, parse: () => Promise): Promise { + const previous = pending.get(path) + const run = (async () => { + await previous?.catch(() => undefined) + return parse() + })() + pending.set(path, run) + try { + return await run + } finally { + if (pending.get(path) === run) { + pending.delete(path) + } + } +} diff --git a/src/main/ai-vault/session-scanner-accumulator.ts b/src/main/ai-vault/session-scanner-accumulator.ts index caf08f4259d..88e09627eb7 100644 --- a/src/main/ai-vault/session-scanner-accumulator.ts +++ b/src/main/ai-vault/session-scanner-accumulator.ts @@ -23,6 +23,12 @@ import { normalizePreviewText, timestampMs } from './session-scanner-values' +import { NO_TRANSCRIPT_MESSAGES, type TranscriptMessageSink } from './session-transcript-consumers' +import { + boundedText, + transcriptMessageRole, + transcriptMessagesFromContent +} from './session-transcript-message-content' const SESSION_PREVIEW_MESSAGE_LIMIT = 5 @@ -30,9 +36,12 @@ export function createAccumulator(args: { agent: AiVaultAgent file: FileWithMtime sessionId: string + // Where every decoded message goes; absent for one-shot parses with no reader. + messages?: TranscriptMessageSink }): SessionAccumulator { return { agent: args.agent, + messages: args.messages ?? NO_TRANSCRIPT_MESSAGES, sessionId: args.sessionId, title: null, fallbackTitle: null, @@ -75,6 +84,9 @@ export function accumulatorFoldResumeState( }, // Finalize a snapshot: the live accumulator (and its preview array) keeps // accumulating appended lines after this session object is handed out. + // A sibling file's metadata is merged onto this result by the parse cache, + // never into the fold, so re-merging it later starts from what the + // transcript alone said (see session-scanner-sidecar-enrichment.ts). finalize: (platform, options) => finalizeSession(cloneSessionAccumulator(accumulator), platform, options) } @@ -96,7 +108,7 @@ export function finalizeSession( const title = accumulator.title || accumulator.fallbackTitle || - `${aiVaultAgentLabel(accumulator.agent)} ${sessionId.slice(0, 8)}` + generatedSessionTitle(accumulator.agent, sessionId) const executionHostId = options.executionHostId ?? LOCAL_EXECUTION_HOST_ID @@ -137,6 +149,15 @@ export function finalizeSession( } } +/** + * The title a session gets when neither the transcript nor the agent named it. + * Exported so a later merge can tell "the fold found no title" from a real one + * without re-deriving the string (session-scanner-sidecar-enrichment.ts). + */ +export function generatedSessionTitle(agent: AiVaultAgent, sessionId: string): string { + return `${aiVaultAgentLabel(agent)} ${sessionId.slice(0, 8)}` +} + export function updateTimeline(accumulator: SessionAccumulator, timestamp: unknown): void { const parsed = timestampMs(timestamp) if (!Number.isFinite(parsed)) { @@ -161,8 +182,13 @@ export function addPreviewMessage( // Why: Claude meta/injected turns still preview, but must not seed the // copyable first-prompt row. seedFirstUserPrompt?: boolean + // Set false by callers that already published this record's messages. + publishMessage?: boolean } ): void { + if (args.publishMessage !== false && accumulator.messages.active) { + publishTranscriptMessage(accumulator, args.role, args.text, args.timestamp) + } // Seeded before the preview-empty return so the copy body never depends on // preview-only normalization rules. seedFullFirstUserPrompt( @@ -199,15 +225,41 @@ export function addPreviewContent( () => extractFullFirstUserPromptText(content), options?.seedFirstUserPrompt ) + // Published from the content value, not the preview string: a consumer needs + // the whole turn, including the tool blocks the 220-char preview drops. + if (accumulator.messages.active) { + for (const message of transcriptMessagesFromContent(role, content, timestampIso(timestamp))) { + accumulator.messages.push(message) + } + } addPreviewMessage(accumulator, { role, text: extractPreviewContentText(content), timestamp, // Content path already seeded above when capture is enabled. - seedFirstUserPrompt: false + seedFirstUserPrompt: false, + publishMessage: false }) } +/** One already-flattened turn; the content path publishes per block instead. */ +function publishTranscriptMessage( + accumulator: SessionAccumulator, + role: AiVaultSessionPreviewMessage['role'], + text: string | null, + timestamp: unknown +): void { + const messageRole = transcriptMessageRole(role) + const messageText = text === null ? null : boundedText(text) + if (messageRole && messageText) { + accumulator.messages.push({ + role: messageRole, + text: messageText, + timestamp: timestampIso(timestamp) + }) + } +} + /** * Seed the copyable first prompt from the first real user turn. `fullText` is a * thunk so list scans (capture mode `none`) never pay the extraction cost. diff --git a/src/main/ai-vault/session-scanner-agent-parser.ts b/src/main/ai-vault/session-scanner-agent-parser.ts index 83f23943241..7b0d05f3683 100644 --- a/src/main/ai-vault/session-scanner-agent-parser.ts +++ b/src/main/ai-vault/session-scanner-agent-parser.ts @@ -6,7 +6,10 @@ import { parseClineSessionFile } from './session-scanner-cline-parser' import { parseGrokSessionFile } from './session-scanner-grok-parser' import { parseMessageGraphSessionFile, parseRovoSessionFile } from './session-scanner-graph-parsers' import { parseKimiSessionFile } from './session-scanner-kimi-parser' -import { splitOpenCodeSqliteCandidate } from './session-scanner-opencode-sqlite-paths' +import { + looksLikeOpenCodeSqliteCandidate, + splitOpenCodeSqliteCandidate +} from './session-scanner-opencode-sqlite-paths' import { parseOpenCodeSqliteSessionViaWorker } from './session-scanner-opencode-sqlite-worker-spawn' import { parseClaudeSessionFile } from './session-scanner-primary-parsers' import { parseGeminiSessionFile } from './session-scanner-gemini-parsers' @@ -16,6 +19,16 @@ import { parseCursorSessionFile } from './session-scanner-cursor-parser' import { parseHermesSessionFile } from './session-scanner-hermes-parser' import { parseOpenCodeSessionFile } from './session-scanner-opencode-parser' import type { SessionFileCandidate } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' + +/** + * False when a parser decodes its messages somewhere the channel cannot reach. + * OpenCode's SQLite sessions are read on a worker thread, so their messages + * never come back over the sink and the read must not be reported as complete. + */ +export function parserPublishesMessages(candidate: SessionFileCandidate): boolean { + return candidate.agent !== 'opencode' || !looksLikeOpenCodeSqliteCandidate(candidate.file.path) +} /** * Parse a single agent session file into an `AiVaultSession`. Routes to the @@ -24,25 +37,33 @@ import type { SessionFileCandidate } from './session-scanner-types' * `parseOpenCodeSqliteSession` instead of the legacy JSON parser. * @param candidate - The session file candidate to parse. * @param platform - The platform to use for resume command generation. + * @param messages - Where the parser publishes every decoded message. * @returns The parsed `AiVaultSession`, or `null` if parsing fails. */ export async function parseAgentSessionFile( candidate: SessionFileCandidate, - platform: NodeJS.Platform + platform: NodeJS.Platform, + messages?: TranscriptMessageSink ): Promise { switch (candidate.agent) { case 'claude': - return parseClaudeSessionFile(candidate.file, platform) + return parseClaudeSessionFile(candidate.file, platform, messages) case 'codex': - return parseCodexSessionFile(candidate.file, platform, candidate.codexHome) + return parseCodexSessionFile( + candidate.file, + platform, + candidate.codexHome, + undefined, + messages + ) case 'gemini': - return parseGeminiSessionFile(candidate.file, platform) + return parseGeminiSessionFile(candidate.file, platform, messages) case 'antigravity': - return parseAntigravitySessionFile(candidate.file, platform) + return parseAntigravitySessionFile(candidate.file, platform, messages) case 'copilot': - return parseCopilotSessionFile(candidate.file, platform) + return parseCopilotSessionFile(candidate.file, platform, messages) case 'cursor': - return parseCursorSessionFile(candidate.file, platform) + return parseCursorSessionFile(candidate.file, platform, messages) case 'opencode': { // Why: OpenCode 1.17.x sessions are read from SQLite via a synthetic // # candidate path. Legacy file-based sessions use @@ -55,29 +76,29 @@ export async function parseAgentSessionFile( platform }) } - return parseOpenCodeSessionFile(candidate.file, platform) + return parseOpenCodeSessionFile(candidate.file, platform, messages) } case 'grok': - return parseGrokSessionFile(candidate.file, platform) + return parseGrokSessionFile(candidate.file, platform, messages) case 'hermes': - return parseHermesSessionFile(candidate.file, platform) + return parseHermesSessionFile(candidate.file, platform, messages) case 'rovo': - return parseRovoSessionFile(candidate.file, platform) + return parseRovoSessionFile(candidate.file, platform, messages) case 'openclaw': - return parseMessageGraphSessionFile('openclaw', candidate.file, platform) + return parseMessageGraphSessionFile('openclaw', candidate.file, platform, messages) case 'pi': - return parseMessageGraphSessionFile('pi', candidate.file, platform) + return parseMessageGraphSessionFile('pi', candidate.file, platform, messages) case 'omp': - return parseMessageGraphSessionFile('omp', candidate.file, platform) + return parseMessageGraphSessionFile('omp', candidate.file, platform, messages) case 'prime-agent': - return parseMessageGraphSessionFile('prime-agent', candidate.file, platform) + return parseMessageGraphSessionFile('prime-agent', candidate.file, platform, messages) case 'droid': - return parseDroidSessionFile(candidate.file, platform) + return parseDroidSessionFile(candidate.file, platform, messages) case 'cline': - return parseClineSessionFile(candidate.file, platform) + return parseClineSessionFile(candidate.file, platform, messages) case 'devin': - return parseDevinSessionFile(candidate.file, platform) + return parseDevinSessionFile(candidate.file, platform, messages) case 'kimi': - return parseKimiSessionFile(candidate.file, platform) + return parseKimiSessionFile(candidate.file, platform, messages) } } diff --git a/src/main/ai-vault/session-scanner-agent-sources.ts b/src/main/ai-vault/session-scanner-agent-sources.ts index 8b2e66ea5b0..957d8d680a6 100644 --- a/src/main/ai-vault/session-scanner-agent-sources.ts +++ b/src/main/ai-vault/session-scanner-agent-sources.ts @@ -8,6 +8,7 @@ import { clineMessagesPathForMetadata, isClineSessionMetadataPath } from './session-scanner-cline-parser' +import { cursorChatMetaPath } from './session-scanner-cursor-chat-meta' import { resolveKimiSessionsDir } from './session-scanner-kimi-paths' import { OMP_SESSION_ARTIFACT_DIR_PATTERN } from './session-scanner-omp-subagent-transcripts' import { claudeProjectsRootDirs, OMP_SESSIONS_DIR, sessionRootDirs } from './session-scanner-roots' @@ -61,8 +62,9 @@ export type AiVaultAgentSource = { rootDirs: (options: AiVaultScanOptions, wslHomeDirs: readonly string[]) => string[] extensions: readonly string[] filePredicate?: (filePath: string) => boolean - // A sibling whose stat participates in candidate freshness and recency. - contentDependencyPath?: (filePath: string) => string + // A sibling whose stat participates in candidate freshness and recency; async + // for agents that have to look the sibling up rather than derive its path. + contentDependencyPath?: (filePath: string) => string | undefined | Promise // Return false to skip a directory; depth 0 is a child of the root. directoryPredicate?: (name: string, depth: number) => boolean // Roots that are alternates for one install rather than distinct locations, @@ -126,7 +128,8 @@ export const AI_VAULT_AGENT_SOURCES: AiVaultAgentSourceTable = { 'projects' ]), extensions: ['.jsonl'], - filePredicate: (filePath) => pathSegments(filePath).includes('agent-transcripts') + filePredicate: (filePath) => pathSegments(filePath).includes('agent-transcripts'), + contentDependencyPath: cursorChatMetaPath }, grok: { rootDirs: (options, wslHomeDirs) => diff --git a/src/main/ai-vault/session-scanner-antigravity-parser.ts b/src/main/ai-vault/session-scanner-antigravity-parser.ts index 1896138749c..126b56ce19a 100644 --- a/src/main/ai-vault/session-scanner-antigravity-parser.ts +++ b/src/main/ai-vault/session-scanner-antigravity-parser.ts @@ -15,6 +15,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { extractString, normalizeTitleText, parseJsonObject } from './session-scanner-values' type ParserSessionOptions = { @@ -24,12 +25,13 @@ type ParserSessionOptions = { export async function parseAntigravitySessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const input = openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan') const lines = createInterface({ input, crlfDelay: Infinity }) try { - return await parseAntigravitySessionLines({ file, lines, platform }) + return await parseAntigravitySessionLines({ file, lines, platform, messages }) } finally { // readline.close() leaves the underlying stream open; destroy it so a // mid-parse throw cannot leak the gated transcript handle. @@ -54,13 +56,14 @@ export async function parseAntigravitySessionContent( } export function createAntigravitySessionResumeState( - file: FileWithMtime + file: FileWithMtime, + messages?: TranscriptMessageSink ): ResumableSessionParseState { const sessionId = antigravityConversationIdFromTranscriptPath(file.path) ?? '' // Why: the transcript has no cwd/model fields. Workspace enrichment is a // separate, conservative history join; protobuf/SQLite blobs are unstable. return accumulatorFoldResumeState( - createAccumulator({ agent: 'antigravity', file, sessionId }), + createAccumulator({ agent: 'antigravity', file, sessionId, messages }), consumeAntigravityRecordLine ) } @@ -70,8 +73,9 @@ async function parseAntigravitySessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ParserSessionOptions + messages?: TranscriptMessageSink }): Promise { - const state = createAntigravitySessionResumeState(args.file) + const state = createAntigravitySessionResumeState(args.file, args.messages) for await (const line of args.lines) { state.consumeLine(line) } diff --git a/src/main/ai-vault/session-scanner-candidates.ts b/src/main/ai-vault/session-scanner-candidates.ts new file mode 100644 index 00000000000..80f146367e5 --- /dev/null +++ b/src/main/ai-vault/session-scanner-candidates.ts @@ -0,0 +1,46 @@ +import { readCodexRolloutSessionMetaId } from '../codex/codex-rollout-session-meta' +import { codexRolloutHardlinkIdentity, dedupeCodexRolloutAliases } from './codex-session-root-dedup' +import { antigravityHistoryPathForBrainDir } from './session-scanner-antigravity-paths' +import { codexHomeForSessionsDir } from './session-scanner-codex-paths' +import { DEFAULT_CODEX_HOME_DIR } from './session-scanner-source-discovery' +import type { + AiVaultScanOptions, + SessionFileCandidate, + SessionFileDiscovery +} from './session-scanner-types' + +/** Newest-first parse candidates for a discovery set, with Codex hardlink aliases collapsed. */ +export async function sessionCandidatesFromDiscoveries( + discoveries: SessionFileDiscovery[], + options: AiVaultScanOptions +): Promise { + return dedupeCodexRolloutAliases( + discoveries + .flatMap((discovery) => + discovery.files.map((file): SessionFileCandidate => ({ + agent: discovery.agent, + file, + codexHome: + discovery.agent === 'codex' + ? codexHomeForSessionsDir( + discovery.rootDir, + options.defaultCodexHomeDir ?? DEFAULT_CODEX_HOME_DIR + ) + : null, + antigravityHistoryPath: + discovery.agent === 'antigravity' + ? antigravityHistoryPathForBrainDir(discovery.rootDir) + : undefined + })) + ) + .sort((left, right) => right.file.mtimeMs - left.file.mtimeMs), + { + isCodex: (candidate) => candidate.agent === 'codex', + getFilePath: (candidate) => candidate.file.path, + getCodexHome: (candidate) => candidate.codexHome, + getHardlinkIdentity: (candidate) => codexRolloutHardlinkIdentity(candidate.file) + }, + (filePath) => readCodexRolloutSessionMetaId(filePath, options.signal, 'scan'), + options.signal + ) +} diff --git a/src/main/ai-vault/session-scanner-cline-parser.ts b/src/main/ai-vault/session-scanner-cline-parser.ts index 21a21664205..462108cfed9 100644 --- a/src/main/ai-vault/session-scanner-cline-parser.ts +++ b/src/main/ai-vault/session-scanner-cline-parser.ts @@ -9,6 +9,7 @@ import { updateTimeline } from './session-scanner-accumulator' import type { FileWithMtime } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { arrayValue, asRecord, @@ -35,7 +36,8 @@ export function clineMessagesPathForMetadata(filePath: string): string { export async function parseClineSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messageSink?: TranscriptMessageSink ): Promise { const metadataContent = await wslGatedReadFile(file.path, 'utf-8', 'scan') let messagesContent: string | null = null @@ -52,7 +54,7 @@ export async function parseClineSessionFile( throw error } } - return parseClineSessionContent(file, metadataContent, messagesContent, platform) + return parseClineSessionContent(file, metadataContent, messagesContent, platform, {}, messageSink) } function isMissingSessionPathError(error: unknown): boolean { @@ -68,7 +70,8 @@ export function parseClineSessionContent( metadataContent: string, messagesContent: string | null, platform: NodeJS.Platform = process.platform, - options: ParserSessionOptions = {} + options: ParserSessionOptions = {}, + messageSink?: TranscriptMessageSink ): AiVaultSession | null { const metadata = parseJsonRecord(metadataContent) if (!metadata) { @@ -76,7 +79,12 @@ export function parseClineSessionContent( } const pathSegments = file.path.replace(/\\/g, '/').split('/').filter(Boolean) const sessionId = extractString(metadata.session_id) ?? pathSegments.at(-2) ?? '' - const accumulator = createAccumulator({ agent: 'cline', file, sessionId }) + const accumulator = createAccumulator({ + agent: 'cline', + file, + sessionId, + messages: messageSink + }) accumulator.cwd = extractString(metadata.cwd) ?? extractString(metadata.workspace_root) accumulator.model = extractString(metadata.model) updateTimeline(accumulator, metadata.started_at) diff --git a/src/main/ai-vault/session-scanner-codex-parser.ts b/src/main/ai-vault/session-scanner-codex-parser.ts index 80a76ce0a92..02a385400de 100644 --- a/src/main/ai-vault/session-scanner-codex-parser.ts +++ b/src/main/ai-vault/session-scanner-codex-parser.ts @@ -22,6 +22,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { addCodexUsage, asRecord, @@ -29,18 +30,22 @@ import { extractModel, extractString, normalizeCodexUsage, - normalizeTitleText, parseJsonObject, subtractCodexUsage } from './session-scanner-values' import { remoteSessionContentLines } from './remote-session-content-lines' import { readCodexTimelineOnlyRecord } from './session-scanner-codex-record-fast-path' +import { + extractCodexSessionMetadataTitle, + isCodexWorkerSession +} from './session-scanner-codex-session-meta' export async function parseCodexSessionFile( file: FileWithMtime, platform: NodeJS.Platform = process.platform, codexHome: string | null = null, - executionHostId?: ExecutionHostId + executionHostId?: ExecutionHostId, + messages?: TranscriptMessageSink ): Promise { const lines = createInterface({ input: openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan'), @@ -53,6 +58,7 @@ export async function parseCodexSessionFile( platform, codexHome, executionHostId, + messages, titleReader: (sessionId) => readCodexSessionIndexTitle(file.path, codexHome, sessionId) }) } @@ -89,12 +95,16 @@ type CodexSessionParseState = { titleSource: 'meta' | 'user' | null } -function createCodexParseState(file: FileWithMtime): CodexSessionParseState { +function createCodexParseState( + file: FileWithMtime, + messages?: TranscriptMessageSink +): CodexSessionParseState { return { accumulator: createAccumulator({ agent: 'codex', file, - sessionId: sessionIdFromFileName(file.path) + sessionId: sessionIdFromFileName(file.path), + messages }), previousTotals: null, rejectedWorkerSession: false, @@ -257,10 +267,13 @@ async function finalizeCodexParseState( export function createCodexSessionResumeState( file: FileWithMtime, - codexHome: string | null + codexHome: string | null, + messages?: TranscriptMessageSink ): ResumableSessionParseState { - return codexResumeStateFromParseState(createCodexParseState(file), codexHome, (sessionId) => - readCodexSessionIndexTitle(file.path, codexHome, sessionId) + return codexResumeStateFromParseState( + createCodexParseState(file, messages), + codexHome, + (sessionId) => readCodexSessionIndexTitle(file.path, codexHome, sessionId) ) } @@ -298,8 +311,9 @@ async function parseCodexSessionLines(args: { executionHostId?: ExecutionHostId executionHostPlatform?: NodeJS.Platform | null titleReader?: (sessionId: string) => Promise + messages?: TranscriptMessageSink }): Promise { - const state = createCodexParseState(args.file) + const state = createCodexParseState(args.file, args.messages) for await (const line of args.lines) { consumeCodexRecordLine(state, line) if (state.rejectedWorkerSession) { @@ -314,21 +328,3 @@ async function parseCodexSessionLines(args: { executionHostPlatform: args.executionHostPlatform }) } - -function isCodexWorkerSession(payload: Record): boolean { - const threadSource = extractString(payload.thread_source) ?? extractString(payload.threadSource) - if (threadSource) { - return threadSource.toLowerCase() !== 'user' - } - - const source = asRecord(payload.source) - return Boolean(asRecord(source?.subagent)) -} - -function extractCodexSessionMetadataTitle(payload: Record): string | null { - return ( - normalizeTitleText(extractString(payload.title) ?? '') ?? - normalizeTitleText(extractString(payload.thread_name) ?? '') ?? - normalizeTitleText(extractString(payload.threadName) ?? '') - ) -} diff --git a/src/main/ai-vault/session-scanner-codex-session-meta.ts b/src/main/ai-vault/session-scanner-codex-session-meta.ts new file mode 100644 index 00000000000..f2cba3c97d0 --- /dev/null +++ b/src/main/ai-vault/session-scanner-codex-session-meta.ts @@ -0,0 +1,22 @@ +import { asRecord, extractString, normalizeTitleText } from './session-scanner-values' + +// Field readers for Codex's `session_meta` record, whose key spelling has drifted +// across Codex releases (snake_case rollouts, camelCase app-server rollouts). + +export function isCodexWorkerSession(payload: Record): boolean { + const threadSource = extractString(payload.thread_source) ?? extractString(payload.threadSource) + if (threadSource) { + return threadSource.toLowerCase() !== 'user' + } + + const source = asRecord(payload.source) + return Boolean(asRecord(source?.subagent)) +} + +export function extractCodexSessionMetadataTitle(payload: Record): string | null { + return ( + normalizeTitleText(extractString(payload.title) ?? '') ?? + normalizeTitleText(extractString(payload.thread_name) ?? '') ?? + normalizeTitleText(extractString(payload.threadName) ?? '') + ) +} diff --git a/src/main/ai-vault/session-scanner-copilot-parser.ts b/src/main/ai-vault/session-scanner-copilot-parser.ts index 7e5450393e0..239c5983457 100644 --- a/src/main/ai-vault/session-scanner-copilot-parser.ts +++ b/src/main/ai-vault/session-scanner-copilot-parser.ts @@ -8,6 +8,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { accumulatorFoldResumeState, addPreviewMessage, @@ -32,13 +33,14 @@ type ParserSessionOptions = { export async function parseCopilotSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const lines = createInterface({ input: openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan'), crlfDelay: Infinity }) - return parseCopilotSessionLines({ file, lines, platform }) + return parseCopilotSessionLines({ file, lines, platform, messages }) } export async function parseCopilotSessionContent( @@ -107,9 +109,17 @@ function consumeCopilotRecordLine(accumulator: SessionAccumulator, line: string) } } -export function createCopilotSessionResumeState(file: FileWithMtime): ResumableSessionParseState { +export function createCopilotSessionResumeState( + file: FileWithMtime, + messages?: TranscriptMessageSink +): ResumableSessionParseState { return accumulatorFoldResumeState( - createAccumulator({ agent: 'copilot', file, sessionId: sessionIdFromFileName(file.path) }), + createAccumulator({ + agent: 'copilot', + file, + sessionId: sessionIdFromFileName(file.path), + messages + }), consumeCopilotRecordLine ) } @@ -119,8 +129,9 @@ async function parseCopilotSessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ParserSessionOptions + messages?: TranscriptMessageSink }): Promise { - const state = createCopilotSessionResumeState(args.file) + const state = createCopilotSessionResumeState(args.file, args.messages) for await (const line of args.lines) { state.consumeLine(line) } diff --git a/src/main/ai-vault/session-scanner-cursor-chat-meta.test.ts b/src/main/ai-vault/session-scanner-cursor-chat-meta.test.ts new file mode 100644 index 00000000000..0516ff457a7 --- /dev/null +++ b/src/main/ai-vault/session-scanner-cursor-chat-meta.test.ts @@ -0,0 +1,552 @@ +import { appendFile, mkdir, mkdtemp, rm, stat, utimes, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { WslTranscriptFsError } from '../native-chat/wsl-transcript-fs-error' + +// Why: a refused WSL read is the one build failure that must not be cached. +let failNextChatsReaddir = false +let failNextChatsRootReaddir = false +let failMetaJsonReads = false +let failMetaJsonStats: false | true | 'eacces' = false +let chatsRootReads = 0 +vi.mock('../native-chat/wsl-transcript-fs-access', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + wslGatedReaddir: ( + ...args: Parameters + ): ReturnType => { + if (args[0].endsWith('chats')) { + chatsRootReads += 1 + if (failNextChatsRootReaddir) { + failNextChatsRootReaddir = false + return Promise.reject(new WslTranscriptFsError('timeout', 'wsl fs timed out')) + } + } + if (failNextChatsReaddir && args[0].includes('workspace-hash')) { + failNextChatsReaddir = false + return Promise.reject(new WslTranscriptFsError('timeout', 'wsl fs timed out')) + } + return actual.wslGatedReaddir(...args) + }, + wslGatedReadFile: ( + ...args: Parameters + ): ReturnType => { + if (failMetaJsonReads && String(args[0]).endsWith('meta.json')) { + return Promise.reject(new WslTranscriptFsError('timeout', 'wsl fs timed out')) + } + return actual.wslGatedReadFile(...args) + }, + wslGatedStat: ( + ...args: Parameters + ): ReturnType => { + if (failMetaJsonStats && String(args[0]).endsWith('meta.json')) { + return Promise.reject( + failMetaJsonStats === 'eacces' + ? Object.assign(new Error('permission denied'), { code: 'EACCES' }) + : new WslTranscriptFsError('timeout', 'wsl fs timed out') + ) + } + return actual.wslGatedStat(...args) + } + } +}) +import type * as WslTranscriptFsAccess from '../native-chat/wsl-transcript-fs-access' +import { + cursorChatMetaPath, + readCursorChatMeta, + resetCursorChatMetaIndexCacheForTests, + withCursorChatMetaScan +} from './session-scanner-cursor-chat-meta' +import { parseCursorSessionContent } from './session-scanner-cursor-parser' +import type { AiVaultScanIssue, AiVaultSession } from '../../shared/ai-vault-types' +import { AI_VAULT_AGENT_SOURCES } from './session-scanner-agent-sources' +import { discoverFiles } from './session-scanner-discovery' +import { scanAiVaultSessions } from './session-scanner' +import { + createSessionParseStats, + parseAgentSessionFileCached, + resetSessionParseCacheForTests, + seedSessionParseCache, + snapshotSessionParseCacheForPersistence, + type SessionParseStats +} from './session-scanner-parse-cache' +import { + getSessionParseCacheEntry, + type PersistedSessionParseCacheEntry +} from './session-parse-cache-store' +import { isolatedScanRoots } from './session-scanner-test-fixtures' +import type { FileWithMtime } from './session-scanner-types' +import type { SessionSidecarStat } from './session-sidecar-stat' + +// Cursor's real meta.json keys (~/.cursor/chats///meta.json, 2026-09). +type CursorMetaFixture = { + schemaVersion: number + createdAtMs: number + updatedAtMs: number + cwd: string + hasConversation: boolean + title?: string +} + +const CREATED_AT_MS = 1_787_039_612_017 +const UPDATED_AT_MS = 1_787_039_640_532 + +let tempRoots: string[] = [] + +afterEach(async () => { + resetCursorChatMetaIndexCacheForTests() + resetSessionParseCacheForTests() + failNextChatsRootReaddir = false + failMetaJsonReads = false + failMetaJsonStats = false + await Promise.all(tempRoots.map((root) => rm(root, { recursive: true, force: true }))) + tempRoots = [] +}) + +async function createCursorHome(): Promise { + const root = await mkdtemp(join(tmpdir(), 'orca-cursor-chat-meta-')) + tempRoots.push(root) + const cursorHome = join(root, '.cursor') + await mkdir(cursorHome, { recursive: true }) + return cursorHome +} + +async function writeTranscript( + cursorHome: string, + projectSlug: string, + chatId: string, + lines: string[] +): Promise { + const chatDir = join(cursorHome, 'projects', projectSlug, 'agent-transcripts', chatId) + await mkdir(chatDir, { recursive: true }) + const transcriptPath = join(chatDir, `${chatId}.jsonl`) + await writeFile(transcriptPath, lines.map((line) => `${line}\n`).join('')) + return transcriptPath +} + +async function writeChatMeta( + cursorHome: string, + workspaceHash: string, + chatId: string, + meta: Partial = {} +): Promise { + const chatDir = join(cursorHome, 'chats', workspaceHash, chatId) + await mkdir(chatDir, { recursive: true }) + const metaPath = join(chatDir, 'meta.json') + await writeFile( + metaPath, + JSON.stringify({ + schemaVersion: 1, + createdAtMs: CREATED_AT_MS, + updatedAtMs: UPDATED_AT_MS, + cwd: '/private/tmp/workspace', + hasConversation: true, + ...meta + } satisfies CursorMetaFixture) + ) + return metaPath +} + +function fileWithMtime(path: string): FileWithMtime { + return { path, mtimeMs: 1, modifiedAt: new Date(1).toISOString() } +} + +describe('cursor chat meta', () => { + it('resolves the meta.json under the workspace hash that holds the chat id', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'aa37220647fb7ce5eb044aa4bda60807', 'other-chat') + const metaPath = await writeChatMeta(cursorHome, '96fa26ac0f433670ebec73ecef20b47b', 'chat-1', { + title: 'Shell Command Hostname' + }) + const transcriptPath = await writeTranscript(cursorHome, 'private-tmp-workspace', 'chat-1', []) + + expect(await cursorChatMetaPath(transcriptPath)).toBe(metaPath) + expect(await readCursorChatMeta(transcriptPath)).toEqual({ + title: 'Shell Command Hostname', + cwd: '/private/tmp/workspace', + createdAt: new Date(CREATED_AT_MS).toISOString(), + updatedAt: new Date(UPDATED_AT_MS).toISOString() + }) + }) + + it('re-indexes after a chat appears under an already indexed workspace', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-first') + const firstTranscript = await writeTranscript(cursorHome, 'slug', 'chat-first', []) + expect(await cursorChatMetaPath(firstTranscript)).toBeDefined() + + const laterMetaPath = await writeChatMeta(cursorHome, 'workspace-hash', 'chat-later') + const laterTranscript = await writeTranscript(cursorHome, 'slug', 'chat-later', []) + + expect(await cursorChatMetaPath(laterTranscript)).toBe(laterMetaPath) + }) + + it('does not cache a metadata index whose build was refused by the WSL gate', async () => { + const cursorHome = await createCursorHome() + const metaPath = await writeChatMeta(cursorHome, 'workspace-hash', 'chat-refused') + const transcriptPath = await writeTranscript(cursorHome, 'slug', 'chat-refused', []) + + failNextChatsReaddir = true + // A refusal degrades to "no metadata" rather than taking the session down. + await expect(cursorChatMetaPath(transcriptPath)).resolves.toBeUndefined() + // The next scan rebuilds instead of replaying the rejected promise. + await expect(cursorChatMetaPath(transcriptPath)).resolves.toBe(metaPath) + }) + + it('validates the index once per scan, not once per transcript', async () => { + const cursorHome = await createCursorHome() + const transcripts: string[] = [] + for (const chatId of ['chat-a', 'chat-b', 'chat-c']) { + await writeChatMeta(cursorHome, 'workspace-hash', chatId) + transcripts.push(await writeTranscript(cursorHome, 'slug', chatId, [])) + } + chatsRootReads = 0 + + const inScan = await withCursorChatMetaScan(() => + Promise.all(transcripts.map((path) => cursorChatMetaPath(path))) + ) + expect(inScan.every(Boolean)).toBe(true) + expect(chatsRootReads).toBe(1) + + // Outside a scan every lookup re-validates, which is what the parse path needs. + chatsRootReads = 0 + await Promise.all(transcripts.map((path) => cursorChatMetaPath(path))) + expect(chatsRootReads).toBe(3) + }) + + it('yields nothing and does not throw when there is no chats tree', async () => { + const cursorHome = await createCursorHome() + const transcriptPath = await writeTranscript(cursorHome, 'slug', 'chat-orphan', []) + + await expect(cursorChatMetaPath(transcriptPath)).resolves.toBeUndefined() + await expect(readCursorChatMeta(transcriptPath)).resolves.toBeNull() + await expect(readCursorChatMeta('/nowhere/near/cursor/chat.jsonl')).resolves.toBeNull() + }) + + it('yields nothing and does not throw when meta.json is malformed', async () => { + const cursorHome = await createCursorHome() + const chatDir = join(cursorHome, 'chats', 'workspace-hash', 'chat-bad') + await mkdir(chatDir, { recursive: true }) + await writeFile(join(chatDir, 'meta.json'), '{ not json') + const transcriptPath = await writeTranscript(cursorHome, 'slug', 'chat-bad', []) + + await expect(readCursorChatMeta(transcriptPath)).resolves.toBeNull() + }) +}) + +async function cursorCandidate(cursorHome: string): Promise { + const issues: AiVaultScanIssue[] = [] + const discovery = await discoverFiles({ + rootDir: join(cursorHome, 'projects'), + limit: 10, + agent: 'cursor', + issues, + extensions: [...AI_VAULT_AGENT_SOURCES.cursor.extensions], + filePredicate: AI_VAULT_AGENT_SOURCES.cursor.filePredicate, + contentDependencyPath: AI_VAULT_AGENT_SOURCES.cursor.contentDependencyPath + }) + return discovery.files[0] +} + +/** The production path: the parse cache owns the sidecar merge, not the parser. */ +function parseCursorCached( + file: FileWithMtime, + stats: SessionParseStats = createSessionParseStats() +): Promise<{ session: AiVaultSession | null; stats: SessionParseStats }> { + return withCursorChatMetaScan(async () => { + const session = await parseAgentSessionFileCached( + { agent: 'cursor', file, codexHome: null }, + 'darwin', + stats + ) + return { session, stats } + }) +} + +async function writeCursorScanFixture(chatIds: string[]): Promise<{ + cursorHome: string + scanOptions: ReturnType & { cursorProjectsDir: string } +}> { + const cursorHome = await createCursorHome() + for (const chatId of chatIds) { + await writeChatMeta(cursorHome, 'workspace-hash', chatId, { cwd: `/tmp/ws-${chatId}` }) + await writeTranscript(cursorHome, 'slug', chatId, [ + JSON.stringify({ role: 'user', message: { content: [{ type: 'text', text: chatId }] } }) + ]) + } + const root = join(cursorHome, '..') + return { + cursorHome, + scanOptions: { + ...isolatedScanRoots(root), + cursorProjectsDir: join(cursorHome, 'projects') + } + } +} + +describe('cursor discovery sidecar observation', () => { + it('records meta.json beside the transcript stat instead of folding it in', async () => { + const cursorHome = await createCursorHome() + const metaPath = await writeChatMeta(cursorHome, 'workspace-hash', 'chat-7') + const transcriptPath = await writeTranscript(cursorHome, 'slug', 'chat-7', []) + + const before = await cursorCandidate(cursorHome) + const future = new Date(Date.now() + 10_000) + await utimes(metaPath, future, future) + const after = await cursorCandidate(cursorHome) + + // The transcript's own key is untouched by a sibling rewrite. + const transcriptStat = await stat(transcriptPath) + expect(after.mtimeMs).toBe(transcriptStat.mtimeMs) + expect(after.sizeBytes).toBe(transcriptStat.size) + expect(after.mtimeMs).toBe(before.mtimeMs) + // The sibling is observed separately, and it did move. + expect(before.sidecar).toMatchObject({ path: metaPath }) + expect(after.sidecar).toMatchObject({ path: metaPath }) + expect((after.sidecar as SessionSidecarStat).mtimeMs).toBeGreaterThan( + (before.sidecar as SessionSidecarStat).mtimeMs + ) + }) +}) + +describe('cursor sidecar enrichment', () => { + it('fills cwd, timestamps and title from meta.json', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-2', { title: 'Named From Meta' }) + await writeTranscript(cursorHome, 'slug', 'chat-2', [ + JSON.stringify({ role: 'assistant', message: { content: 'hello' } }) + ]) + resetSessionParseCacheForTests() + + const { session } = await parseCursorCached(await cursorCandidate(cursorHome)) + + expect(session?.cwd).toBe('/private/tmp/workspace') + expect(session?.title).toBe('Named From Meta') + expect(session?.createdAt).toBe(new Date(CREATED_AT_MS).toISOString()) + expect(session?.updatedAt).toBe(new Date(UPDATED_AT_MS).toISOString()) + }) + + it('keeps a transcript title and timestamps over meta.json', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-3', { title: 'Meta Title' }) + await writeTranscript(cursorHome, 'slug', 'chat-3', [ + JSON.stringify({ + role: 'user', + timestamp: '2026-01-01T00:00:00.000Z', + message: { content: 'transcript first prompt' } + }) + ]) + resetSessionParseCacheForTests() + + const { session } = await parseCursorCached(await cursorCandidate(cursorHome)) + + expect(session?.title).toBe('transcript first prompt') + expect(session?.createdAt).toBe('2026-01-01T00:00:00.000Z') + // cwd is never in the transcript, so it still comes from meta.json. + expect(session?.cwd).toBe('/private/tmp/workspace') + }) + + it('builds the resume command from the meta.json cwd', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-4', { cwd: '/repo/from-meta' }) + await writeTranscript(cursorHome, 'slug', 'chat-4', [ + JSON.stringify({ role: 'user', message: { content: 'hi' } }) + ]) + resetSessionParseCacheForTests() + + const { session } = await parseCursorCached(await cursorCandidate(cursorHome)) + + expect(session?.resumeCommand).toContain('/repo/from-meta') + }) + + it('leaves remote content parses to the transcript alone', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-5', { title: 'Meta Title' }) + const transcriptPath = await writeTranscript(cursorHome, 'slug', 'chat-5', []) + + const session = await parseCursorSessionContent( + fileWithMtime(transcriptPath), + `${JSON.stringify({ role: 'assistant', message: { content: 'remote' } })}\n`, + 'linux' + ) + + expect(session?.cwd).toBeNull() + expect(session?.title).not.toBe('Meta Title') + }) + + it('re-enriches without a parse when only the sidecar is rewritten', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-8', { cwd: '/repo/first' }) + await writeTranscript(cursorHome, 'slug', 'chat-8', [ + JSON.stringify({ role: 'user', message: { content: 'hi' } }) + ]) + resetSessionParseCacheForTests() + await parseCursorCached(await cursorCandidate(cursorHome)) + + const future = new Date(Date.now() + 10_000) + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-8', { cwd: '/repo/second' }) + await utimes(join(cursorHome, 'chats', 'workspace-hash', 'chat-8', 'meta.json'), future, future) + + const { session, stats } = await parseCursorCached(await cursorCandidate(cursorHome)) + + // The transcript is not re-read: the merge runs over the stored fold result. + expect(stats.reused).toBe(1) + expect(stats.fullParses).toBe(0) + expect(stats.incremental).toBe(0) + // A rewritten cwd REPLACES the merged one; `??=` on the cached session could + // never do this, because the cached cwd is already non-null. + expect(session?.cwd).toBe('/repo/second') + expect(session?.resumeCommand).toContain('/repo/second') + }) + + it('treats a persisted entry with no sidecar as unknown and enriches once', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-9', { cwd: '/repo/persisted' }) + await writeTranscript(cursorHome, 'slug', 'chat-9', [ + JSON.stringify({ role: 'user', message: { content: 'hi' } }) + ]) + resetSessionParseCacheForTests() + await parseCursorCached(await cursorCandidate(cursorHome)) + + // What a build older than the sidecar field wrote: no such key. + const persisted = snapshotSessionParseCacheForPersistence().map( + ([path, entry]): [string, PersistedSessionParseCacheEntry] => { + const { sidecar: _sidecar, ...rest } = entry + return [path, rest] + } + ) + resetSessionParseCacheForTests() + seedSessionParseCache(persisted) + + const { session, stats } = await parseCursorCached(await cursorCandidate(cursorHome)) + expect(stats.reused).toBe(0) + expect(session?.cwd).toBe('/repo/persisted') + }) +}) + +describe('cursor chat meta scan failures', () => { + it('lists cursor sessions without metadata when the chats tree is refused, then heals', async () => { + const { cursorHome, scanOptions } = await writeCursorScanFixture(['chat-a', 'chat-b']) + resetSessionParseCacheForTests() + + failNextChatsRootReaddir = true + const refused = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + const refusedCursor = refused.sessions.filter((session) => session.agent === 'cursor') + expect(refusedCursor).toHaveLength(2) + expect(refusedCursor.map((session) => session.cwd)).toEqual([null, null]) + // One issue for the chats root, not one per transcript. + expect(refused.issues).toHaveLength(1) + expect(refused.issues[0].path).toBe(join(cursorHome, 'chats')) + expect(refused.issues[0].agent).toBe('cursor') + + const healed = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + expect(healed.issues).toEqual([]) + expect( + healed.sessions + .filter((session) => session.agent === 'cursor') + .map((session) => session.cwd) + .sort() + ).toEqual(['/tmp/ws-chat-a', '/tmp/ws-chat-b']) + }) + + it('re-enriches after a refused meta.json read without losing the resume cursor', async () => { + const { scanOptions } = await writeCursorScanFixture(['chat-a']) + resetSessionParseCacheForTests() + + failMetaJsonReads = true + const refused = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + const listed = refused.sessions.find((session) => session.agent === 'cursor') + expect(listed?.cwd).toBeNull() + expect(refused.issues).toHaveLength(1) + + // The sibling alone is unknown; the transcript's work and its resume point + // are kept, so the next healthy scan merges without re-reading bytes. + const entry = getSessionParseCacheEntry(listed?.filePath ?? '') + expect(entry?.sidecar).toBe('unknown') + expect(entry?.resume).not.toBeNull() + + failMetaJsonReads = false + const healed = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + expect(healed.issues).toEqual([]) + expect(healed.sessions.find((session) => session.agent === 'cursor')?.cwd).toBe( + '/tmp/ws-chat-a' + ) + }) + + it('treats a local EACCES on the sidecar stat as unknown, not as absent', async () => { + const { scanOptions } = await writeCursorScanFixture(['chat-a']) + resetSessionParseCacheForTests() + + // On mac/Linux/Windows the gated stat is a bare fs stat, so a permissions + // failure is not a WslTranscriptFsError and must not read as "no sidecar". + failMetaJsonStats = 'eacces' + const refused = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + const listed = refused.sessions.find((session) => session.agent === 'cursor') + expect(listed?.sessionId).toBeTruthy() + expect(refused.issues).toHaveLength(1) + expect(refused.issues[0].agent).toBe('cursor') + expect(getSessionParseCacheEntry(listed?.filePath ?? '')?.sidecar).toBe('unknown') + + failMetaJsonStats = false + const healed = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + expect(healed.issues).toEqual([]) + expect(healed.sessions.find((session) => session.agent === 'cursor')?.cwd).toBe( + '/tmp/ws-chat-a' + ) + }) + + it('lists a cursor session whose meta.json stat is refused instead of dropping it', async () => { + const { scanOptions } = await writeCursorScanFixture(['chat-a']) + resetSessionParseCacheForTests() + + failMetaJsonStats = true + const refused = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + expect(refused.sessions.filter((session) => session.agent === 'cursor')).toHaveLength(1) + expect(refused.issues).toHaveLength(1) + expect(refused.issues[0].agent).toBe('cursor') + + failMetaJsonStats = false + const healed = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + expect(healed.issues).toEqual([]) + expect(healed.sessions.find((session) => session.agent === 'cursor')?.cwd).toBe( + '/tmp/ws-chat-a' + ) + }) + + it('reads the chats root once per scan across discovery and parse', async () => { + const { scanOptions } = await writeCursorScanFixture(['chat-a', 'chat-b', 'chat-c']) + resetSessionParseCacheForTests() + chatsRootReads = 0 + + const result = await scanAiVaultSessions({ ...scanOptions, platform: 'darwin', limit: 20 }) + + expect(result.sessions.filter((session) => session.agent === 'cursor')).toHaveLength(3) + expect(chatsRootReads).toBe(1) + }) + + it('resumes an appended transcript after a refused sidecar scan', async () => { + const cursorHome = await createCursorHome() + await writeChatMeta(cursorHome, 'workspace-hash', 'chat-r', { cwd: '/repo/resume' }) + const transcriptPath = await writeTranscript(cursorHome, 'slug', 'chat-r', [ + JSON.stringify({ role: 'user', message: { content: 'one' } }) + ]) + resetSessionParseCacheForTests() + await parseCursorCached(await cursorCandidate(cursorHome)) + + await appendFile( + transcriptPath, + `${JSON.stringify({ role: 'user', message: { content: 'two' } })}\n` + ) + failMetaJsonReads = true + const { stats: refusedStats } = await parseCursorCached(await cursorCandidate(cursorHome)) + expect(refusedStats.incremental).toBe(1) + + failMetaJsonReads = false + const { session, stats } = await parseCursorCached(await cursorCandidate(cursorHome)) + expect(stats.reused).toBe(1) + expect(stats.fullParses).toBe(0) + expect(session?.cwd).toBe('/repo/resume') + }) +}) diff --git a/src/main/ai-vault/session-scanner-cursor-chat-meta.ts b/src/main/ai-vault/session-scanner-cursor-chat-meta.ts new file mode 100644 index 00000000000..866792412eb --- /dev/null +++ b/src/main/ai-vault/session-scanner-cursor-chat-meta.ts @@ -0,0 +1,294 @@ +import { AsyncLocalStorage } from 'node:async_hooks' +import { basename, dirname, join } from 'node:path' +import { wslGatedReaddir, wslGatedStat } from '../native-chat/wsl-transcript-fs-access' +import { WslTranscriptFsError } from '../native-chat/wsl-transcript-fs-gate' +import { timestampIso } from './session-scanner-accumulator' +import { extractString, normalizeTitleText, readJsonObjectIfExists } from './session-scanner-values' + +// Cursor keeps a chat's transcript and its metadata in two unrelated trees: +// /projects//agent-transcripts//.jsonl holds the +// messages, while /chats///meta.json holds the cwd, +// title and timestamps. The md5 hashes the very cwd we are looking for, so the +// only way across is an index of the chat directories. + +const CURSOR_CHATS_DIR = 'chats' +const CURSOR_CHAT_META_FILE = 'meta.json' +const CURSOR_TRANSCRIPTS_DIR = 'agent-transcripts' +const CURSOR_PROJECTS_DIR = 'projects' +// Why: custom and WSL Cursor homes can vary over a long-lived main process. +const CURSOR_CHAT_META_INDEX_CACHE_MAX = 8 + +export type CursorChatMeta = { + title: string | null + cwd: string | null + createdAt: string | null + updatedAt: string | null +} + +type CursorChatMetaIndexEntry = { + signature: string + metaPathByChatId: Map +} + +const cursorChatMetaIndexCache = new Map>() + +type CursorChatMetaScan = { + index: Map>> + // Chats roots this scan could not read, reported once by the scan owner. + refusals: Map + // Transcripts whose own meta.json read was refused, so the metadata merged + // onto them is not what the file on disk says. + refusedTranscripts: Set +} + +// Why: validating the module cache costs a readdir of the chats root plus a stat +// per workspace, and it cannot be skipped because the signature is built from +// those stats. Discovery asks once per transcript and finalize asks again, so +// the scope has to span both phases for one scan to see the tree once. +const scanScopedIndex = new AsyncLocalStorage() + +export function resetCursorChatMetaIndexCacheForTests(): void { + cursorChatMetaIndexCache.clear() +} + +/** Runs one whole scan, discovery and parse; every Cursor transcript in it shares one index read. */ +export function withCursorChatMetaScan(fn: () => Promise): Promise { + return scanScopedIndex.run( + { index: new Map(), refusals: new Map(), refusedTranscripts: new Set() }, + fn + ) +} + +/** + * True when this transcript's own meta.json read was refused, so the caller + * records the sidecar as unknown rather than as the observation discovery made. + * The transcript's own work and its resume point are kept either way. + */ +export function wasCursorChatMetaRefused(transcriptPath: string): boolean { + return scanScopedIndex.getStore()?.refusedTranscripts.has(transcriptPath) ?? false +} + +/** Chats roots the current scan was refused, for the caller to report as scan issues. */ +export function cursorChatMetaRefusals(): { chatsRoot: string; message: string }[] { + const scan = scanScopedIndex.getStore() + return scan ? [...scan.refusals].map(([chatsRoot, message]) => ({ chatsRoot, message })) : [] +} + +/** Path a discovery stat can watch so a rewritten meta.json invalidates the parse cache. */ +export async function cursorChatMetaPath(transcriptPath: string): Promise { + const chatsRoot = cursorChatsRootFromTranscriptPath(transcriptPath) + const chatId = cursorChatIdFromTranscriptPath(transcriptPath) + if (!chatsRoot || !chatId) { + return undefined + } + const index = await readCursorChatMetaIndexOncePerScan(chatsRoot) + return index.get(chatId) +} + +function readCursorChatMetaIndexOncePerScan(chatsRoot: string): Promise> { + const scan = scanScopedIndex.getStore() + if (!scan) { + return readCursorChatMetaIndexOrNone(chatsRoot) + } + let pending = scan.index.get(chatsRoot) + if (!pending) { + pending = readCursorChatMetaIndexOrNone(chatsRoot) + scan.index.set(chatsRoot, pending) + } + return pending +} + +/** + * A refused WSL read is not "no chats", but it must not take the transcript + * down with it: before this join a stalled distro could not hide a Cursor + * session at all. Degrade to no metadata for the scan and report the root once. + * The session still lists from its transcript alone, and the sidecar is + * recorded as unknown, so the next healthy scan merges the real metadata in + * without re-reading a byte of the transcript. + */ +async function readCursorChatMetaIndexOrNone(chatsRoot: string): Promise> { + try { + return await readCursorChatMetaIndex(chatsRoot) + } catch (error) { + if (!(error instanceof WslTranscriptFsError)) { + throw error + } + recordCursorChatMetaRefusal(chatsRoot, error.message) + return new Map() + } +} + +function recordCursorChatMetaRefusal(chatsRoot: string, message: string): void { + const scan = scanScopedIndex.getStore() + if (scan && !scan.refusals.has(chatsRoot)) { + scan.refusals.set(chatsRoot, message) + } +} + +export async function readCursorChatMeta(transcriptPath: string): Promise { + const metaPath = await cursorChatMetaPath(transcriptPath) + if (!metaPath) { + return null + } + let record: Record | null + try { + record = await readJsonObjectIfExists(metaPath) + } catch (error) { + if (!(error instanceof WslTranscriptFsError)) { + throw error + } + // The session still lists, but unlike the index read this transcript's key + // already includes meta.json's stat, so the caller must not cache the + // un-enriched result. One issue per chats root, as for a refused index. + recordCursorChatMetaRefusal( + cursorChatsRootFromTranscriptPath(transcriptPath) ?? metaPath, + error.message + ) + scanScopedIndex.getStore()?.refusedTranscripts.add(transcriptPath) + return null + } + if (!record) { + return null + } + return { + title: normalizeTitleText(extractString(record.title) ?? ''), + cwd: extractString(record.cwd), + createdAt: timestampIso(record.createdAtMs), + updatedAt: timestampIso(record.updatedAtMs) + } +} + +function cursorChatIdFromTranscriptPath(transcriptPath: string): string | null { + const chatDir = dirname(transcriptPath) + return basename(dirname(chatDir)) === CURSOR_TRANSCRIPTS_DIR ? basename(chatDir) : null +} + +function cursorChatsRootFromTranscriptPath(transcriptPath: string): string | null { + let currentDir = dirname(transcriptPath) + while (currentDir && dirname(currentDir) !== currentDir) { + // The chats tree is a sibling of the projects tree, custom Cursor homes included. + if (basename(currentDir) === CURSOR_PROJECTS_DIR) { + return join(dirname(currentDir), CURSOR_CHATS_DIR) + } + currentDir = dirname(currentDir) + } + return null +} + +async function readCursorChatMetaIndex(chatsRoot: string): Promise> { + let workspaceDirs: string[] + try { + workspaceDirs = (await wslGatedReaddir(chatsRoot, 'scan')) + .filter((entry) => entry.isDirectory()) + .map((entry) => entry.name) + .sort() + } catch (error) { + // Why: a refused WSL read is not "no chats"; letting it through keeps the + // session out of the parse cache instead of caching it without metadata. + if (error instanceof WslTranscriptFsError) { + throw error + } + return new Map() + } + const signature = await readCursorChatsSignature(chatsRoot, workspaceDirs) + const cached = await readCachedCursorChatMetaIndex(chatsRoot, signature) + if (cached) { + return cached + } + const pending = buildCursorChatMetaIndex(chatsRoot, workspaceDirs).then((metaPathByChatId) => ({ + signature, + metaPathByChatId + })) + storeCursorChatMetaIndexEntry(chatsRoot, pending) + // Why: a rejected build (a refused WSL read) must not be served from the + // cache forever; the next scan rebuilds while this one still sees the error. + pending.catch(() => { + if (cursorChatMetaIndexCache.get(chatsRoot) === pending) { + cursorChatMetaIndexCache.delete(chatsRoot) + } + }) + return (await pending).metaPathByChatId +} + +// Why: a new chat only bumps its own workspace directory, so the chats root's +// own mtime would keep serving an index that is missing the newest sessions. +async function readCursorChatsSignature( + chatsRoot: string, + workspaceDirs: string[] +): Promise { + const parts = await Promise.all( + workspaceDirs.map(async (name) => { + try { + const dirStat = await wslGatedStat(join(chatsRoot, name), 'scan') + return `${name}:${dirStat.mtimeMs}` + } catch { + return `${name}:?` + } + }) + ) + return parts.join('|') +} + +async function buildCursorChatMetaIndex( + chatsRoot: string, + workspaceDirs: string[] +): Promise> { + const metaPathByChatId = new Map() + for (const workspaceDir of workspaceDirs) { + let chatDirs + try { + chatDirs = await wslGatedReaddir(join(chatsRoot, workspaceDir), 'scan') + } catch (error) { + if (error instanceof WslTranscriptFsError) { + throw error + } + continue + } + for (const chatDir of chatDirs) { + // Why: the same chat id never appears under two workspace hashes, so the + // first hit wins and a duplicate would only cost a wasted read. + if (chatDir.isDirectory() && !metaPathByChatId.has(chatDir.name)) { + metaPathByChatId.set( + chatDir.name, + join(chatsRoot, workspaceDir, chatDir.name, CURSOR_CHAT_META_FILE) + ) + } + } + } + return metaPathByChatId +} + +async function readCachedCursorChatMetaIndex( + chatsRoot: string, + signature: string +): Promise | undefined> { + const cached = cursorChatMetaIndexCache.get(chatsRoot) + if (!cached) { + return undefined + } + const entry = await cached + if (entry.signature !== signature) { + return undefined + } + // Why: a concurrent scan can replace this Promise while it resolves; only the + // still-current entry may refresh recency without bypassing the cap. + if (cursorChatMetaIndexCache.get(chatsRoot) === cached) { + cursorChatMetaIndexCache.delete(chatsRoot) + cursorChatMetaIndexCache.set(chatsRoot, cached) + } + return entry.metaPathByChatId +} + +function storeCursorChatMetaIndexEntry( + chatsRoot: string, + pending: Promise +): void { + cursorChatMetaIndexCache.delete(chatsRoot) + cursorChatMetaIndexCache.set(chatsRoot, pending) + if (cursorChatMetaIndexCache.size > CURSOR_CHAT_META_INDEX_CACHE_MAX) { + const oldest = cursorChatMetaIndexCache.keys().next() + if (!oldest.done) { + cursorChatMetaIndexCache.delete(oldest.value) + } + } +} diff --git a/src/main/ai-vault/session-scanner-cursor-parser.ts b/src/main/ai-vault/session-scanner-cursor-parser.ts index 0f44aa923fa..bfa15caa530 100644 --- a/src/main/ai-vault/session-scanner-cursor-parser.ts +++ b/src/main/ai-vault/session-scanner-cursor-parser.ts @@ -8,6 +8,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { accumulatorFoldResumeState, addPreviewContent, @@ -30,13 +31,14 @@ type ParserSessionOptions = { export async function parseCursorSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const lines = createInterface({ input: openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan'), crlfDelay: Infinity }) - return parseCursorSessionLines({ file, lines, platform }) + return parseCursorSessionLines({ file, lines, platform, messages }) } export async function parseCursorSessionContent( @@ -75,9 +77,17 @@ function consumeCursorRecordLine(accumulator: SessionAccumulator, line: string): } } -export function createCursorSessionResumeState(file: FileWithMtime): ResumableSessionParseState { +export function createCursorSessionResumeState( + file: FileWithMtime, + messages?: TranscriptMessageSink +): ResumableSessionParseState { return accumulatorFoldResumeState( - createAccumulator({ agent: 'cursor', file, sessionId: sessionIdFromFileName(file.path) }), + createAccumulator({ + agent: 'cursor', + file, + sessionId: sessionIdFromFileName(file.path), + messages + }), consumeCursorRecordLine ) } @@ -87,8 +97,9 @@ async function parseCursorSessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ParserSessionOptions + messages?: TranscriptMessageSink }): Promise { - const state = createCursorSessionResumeState(args.file) + const state = createCursorSessionResumeState(args.file, args.messages) for await (const line of args.lines) { state.consumeLine(line) } diff --git a/src/main/ai-vault/session-scanner-devin-parser.ts b/src/main/ai-vault/session-scanner-devin-parser.ts index a43603bca47..12e40ef3fc2 100644 --- a/src/main/ai-vault/session-scanner-devin-parser.ts +++ b/src/main/ai-vault/session-scanner-devin-parser.ts @@ -2,6 +2,7 @@ import { wslGatedReadFile } from '../native-chat/wsl-transcript-fs-access' import type { AiVaultSession } from '../../shared/ai-vault-types' import type { ExecutionHostId } from '../../shared/execution-host' import type { FileWithMtime } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { addPreviewContent, createAccumulator, @@ -25,20 +26,34 @@ type ParserSessionOptions = { export async function parseDevinSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { - return parseDevinSessionContent( + return parseDevinSessionRecord( file, await wslGatedReadFile(file.path, 'utf-8', 'scan'), - platform + platform, + {}, + messages ) } +/** Remote transcript content, streamed from a host that has no reader attached. */ export function parseDevinSessionContent( file: FileWithMtime, content: string, platform: NodeJS.Platform = process.platform, options: ParserSessionOptions = {} +): AiVaultSession | null { + return parseDevinSessionRecord(file, content, platform, options) +} + +function parseDevinSessionRecord( + file: FileWithMtime, + content: string, + platform: NodeJS.Platform, + options: ParserSessionOptions, + messages?: TranscriptMessageSink ): AiVaultSession | null { const record = asRecord(JSON.parse(content) as unknown) if (!record) { @@ -48,7 +63,7 @@ export function parseDevinSessionContent( extractString(record.session_id) ?? extractString(record.sessionId) ?? sessionIdFromFileName(file.path) - const accumulator = createAccumulator({ agent: 'devin', file, sessionId }) + const accumulator = createAccumulator({ agent: 'devin', file, sessionId, messages }) const agentRecord = asRecord(record.agent) accumulator.model = extractString(agentRecord?.model_name) ?? diff --git a/src/main/ai-vault/session-scanner-discovery.ts b/src/main/ai-vault/session-scanner-discovery.ts index a5cf386043e..f7835dbdcc5 100644 --- a/src/main/ai-vault/session-scanner-discovery.ts +++ b/src/main/ai-vault/session-scanner-discovery.ts @@ -1,10 +1,12 @@ import type { Dirent } from 'node:fs' import { extname, join } from 'node:path' +import { SessionNewestFiles } from './session-newest-files' +import type { SessionSidecarObservation } from './session-sidecar-stat' import type { AiVaultAgent, AiVaultScanIssue } from '../../shared/ai-vault-types' import { wslGatedReaddir, wslGatedStat } from '../native-chat/wsl-transcript-fs-access' import { WslTranscriptFsError } from '../native-chat/wsl-transcript-fs-gate' import { recordSessionScanIssue } from './session-scan-issues' -import type { FileWithMtime, SessionFileDiscovery } from './session-scanner-types' +import type { SessionFileDiscovery } from './session-scanner-types' import { errorMessage } from './session-scanner-values' export async function discoverFiles(args: { @@ -14,16 +16,55 @@ export async function discoverFiles(args: { issues: AiVaultScanIssue[] extensions: string[] filePredicate?: (path: string) => boolean - contentDependencyPath?: (path: string) => string + contentDependencyPath?: (path: string) => string | undefined | Promise directoryPredicate?: (name: string, depth: number) => boolean }): Promise { - let paths: string[] + const files = new SessionNewestFiles(args.limit) + let refusedSidecar = false try { - paths = await walkSessionFiles(args.rootDir, args.agent, args.issues, { - extensions: new Set(args.extensions), - filePredicate: args.filePredicate, - directoryPredicate: args.directoryPredicate - }) + await forEachSessionFile( + args.rootDir, + args.agent, + args.issues, + { + extensions: new Set(args.extensions), + filePredicate: args.filePredicate, + directoryPredicate: args.directoryPredicate + }, + async (path) => { + try { + const fileStat = await wslGatedStat(path, 'scan') + const sidecarPath = await args.contentDependencyPath?.(path) + const sidecar = await observeSessionSidecar(sidecarPath) + if (sidecar === 'unknown' && !refusedSidecar) { + // One issue per root: a refused sibling is a property of the tree, + // not of each transcript that happens to point at it. + refusedSidecar = true + recordSessionScanIssue(args.issues, { + agent: args.agent, + path: sidecarPath ?? args.rootDir, + message: 'Session metadata could not be read this scan.' + }) + } + files.add({ + path, + mtimeMs: fileStat.mtimeMs, + modifiedAt: new Date(fileStat.mtimeMs).toISOString(), + sizeBytes: fileStat.size, + sidecar, + dev: fileStat.dev, + ino: fileStat.ino, + nlink: fileStat.nlink + }) + } catch (err) { + recordSessionScanIssue(args.issues, { + agent: args.agent, + path, + message: errorMessage(err) + }) + } + } + ) } catch (err) { // Why: discoverAiVaultSessionSources fans out with Promise.all, so one // stalled distro would otherwise reject the whole vault scan — including @@ -38,68 +79,74 @@ export async function discoverFiles(args: { }) return { agent: args.agent, rootDir: args.rootDir, files: [] } } - const files: FileWithMtime[] = [] - for (const path of paths) { - try { - const fileStat = await wslGatedStat(path, 'scan') - const dependencyStat = await optionalContentDependencyStat(args.contentDependencyPath?.(path)) - const mtimeMs = Math.max(fileStat.mtimeMs, dependencyStat?.mtimeMs ?? 0) - files.push({ - path, - mtimeMs, - modifiedAt: new Date(mtimeMs).toISOString(), - sizeBytes: fileStat.size + (dependencyStat?.size ?? 0), - dev: fileStat.dev, - ino: fileStat.ino, - nlink: fileStat.nlink - }) - } catch (err) { - recordSessionScanIssue(args.issues, { - agent: args.agent, - path, - message: errorMessage(err) - }) - } - } - return { - agent: args.agent, - rootDir: args.rootDir, - files: files.sort((left, right) => right.mtimeMs - left.mtimeMs).slice(0, args.limit) - } + return { agent: args.agent, rootDir: args.rootDir, files: files.newest() } } -async function optionalContentDependencyStat( +/** + * A sibling that cannot be statted is not "no sibling": it must not take the + * transcript down with it, and it must not read as absent either, or the parse + * cache would treat a session enriched from a file nobody can see as current + * forever. Only a genuinely missing path is `'none'`; every other failure — + * a stalled WSL distro, EACCES, EIO — is `'unknown'`. + */ +async function observeSessionSidecar( filePath: string | undefined -): Promise<{ mtimeMs: number; size: number } | null> { +): Promise { if (!filePath) { - return null + return 'none' } try { const fileStat = await wslGatedStat(filePath, 'scan') - return { mtimeMs: fileStat.mtimeMs, size: fileStat.size } + return { path: filePath, mtimeMs: fileStat.mtimeMs, sizeBytes: fileStat.size } } catch (error) { - if (error instanceof WslTranscriptFsError) { - throw error - } - return null + return isMissingSidecarError(error) ? 'none' : 'unknown' } } +function isMissingSidecarError(error: unknown): boolean { + if (error instanceof WslTranscriptFsError) { + return false + } + const code = + error && typeof error === 'object' && 'code' in error && typeof error.code === 'string' + ? error.code + : null + return code === 'ENOENT' || code === 'ENOTDIR' +} + +export type SessionFileWalkOptions = { + extensions: Set + filePredicate?: (path: string) => boolean + // Return false to skip descending into a directory; depth 0 is a child of + // rootDir, so pruned subtrees are never stat'd or parsed. + directoryPredicate?: (name: string, depth: number) => boolean + readDirectory?: (dirPath: string) => Promise + signal?: AbortSignal +} + +/** Collecting form for callers that want every match; bounded scans stream. */ export async function walkSessionFiles( dirPath: string, agent: AiVaultAgent, issues: AiVaultScanIssue[], - options: { - extensions: Set - filePredicate?: (path: string) => boolean - // Return false to skip descending into a directory; depth 0 is a child of - // rootDir, so pruned subtrees are never stat'd or parsed. - directoryPredicate?: (name: string, depth: number) => boolean - readDirectory?: (dirPath: string) => Promise - signal?: AbortSignal - }, - depth = 0 + options: SessionFileWalkOptions ): Promise { + const files: string[] = [] + await forEachSessionFile(dirPath, agent, issues, options, async (path) => { + files.push(path) + }) + return files +} + +/** Streams matches to `onFile` so a bounded consumer never retains the whole tree. */ +export async function forEachSessionFile( + dirPath: string, + agent: AiVaultAgent, + issues: AiVaultScanIssue[], + options: SessionFileWalkOptions, + onFile: (path: string) => Promise, + depth = 0 +): Promise { options.signal?.throwIfAborted() let entries try { @@ -113,10 +160,9 @@ export async function walkSessionFiles( if (error instanceof WslTranscriptFsError) { throw error } - return [] + return } - const files: string[] = [] for (const entry of entries) { options.signal?.throwIfAborted() const fullPath = join(dirPath, entry.name) @@ -124,7 +170,7 @@ export async function walkSessionFiles( // Skip whole subtrees an agent never wants (e.g. subagent transcripts), // avoiding the readdir cost of descending into them. if (options.directoryPredicate?.(entry.name, depth) ?? true) { - files.push(...(await walkSessionFiles(fullPath, agent, issues, options, depth + 1))) + await forEachSessionFile(fullPath, agent, issues, options, onFile, depth + 1) } continue } @@ -133,8 +179,7 @@ export async function walkSessionFiles( options.extensions.has(extname(entry.name).toLowerCase()) && (options.filePredicate?.(fullPath) ?? true) ) { - files.push(fullPath) + await onFile(fullPath) } } - return files } diff --git a/src/main/ai-vault/session-scanner-droid-parser.ts b/src/main/ai-vault/session-scanner-droid-parser.ts index c896b47a3e8..748a8a6bcc9 100644 --- a/src/main/ai-vault/session-scanner-droid-parser.ts +++ b/src/main/ai-vault/session-scanner-droid-parser.ts @@ -8,6 +8,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { accumulatorFoldResumeState, addPreviewMessage, @@ -32,12 +33,13 @@ type ParserSessionOptions = { export async function parseDroidSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const input = openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan') const lines = createInterface({ input, crlfDelay: Infinity }) try { - return await parseDroidSessionLines({ file, lines, platform }) + return await parseDroidSessionLines({ file, lines, platform, messages }) } finally { // readline.close() leaves the underlying stream open; destroy it so a // mid-parse throw cannot leak the gated transcript handle. @@ -94,9 +96,17 @@ function consumeDroidRecordLine(accumulator: SessionAccumulator, line: string): } } -export function createDroidSessionResumeState(file: FileWithMtime): ResumableSessionParseState { +export function createDroidSessionResumeState( + file: FileWithMtime, + messages?: TranscriptMessageSink +): ResumableSessionParseState { return accumulatorFoldResumeState( - createAccumulator({ agent: 'droid', file, sessionId: sessionIdFromFileName(file.path) }), + createAccumulator({ + agent: 'droid', + file, + sessionId: sessionIdFromFileName(file.path), + messages + }), consumeDroidRecordLine ) } @@ -106,8 +116,9 @@ async function parseDroidSessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ParserSessionOptions + messages?: TranscriptMessageSink }): Promise { - const state = createDroidSessionResumeState(args.file) + const state = createDroidSessionResumeState(args.file, args.messages) for await (const line of args.lines) { state.consumeLine(line) } diff --git a/src/main/ai-vault/session-scanner-gemini-parsers.ts b/src/main/ai-vault/session-scanner-gemini-parsers.ts index d8d8a485d8f..efc9eef9e10 100644 --- a/src/main/ai-vault/session-scanner-gemini-parsers.ts +++ b/src/main/ai-vault/session-scanner-gemini-parsers.ts @@ -8,6 +8,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { accumulatorFoldResumeState, addPreviewContent, @@ -27,16 +28,19 @@ import { export async function parseGeminiSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { if (file.path.endsWith('.jsonl')) { - return parseGeminiJsonlSessionFile(file, platform) + return parseGeminiJsonlSessionFile(file, platform, messages) } return parseGeminiJsonSessionContent( file, await wslGatedReadFile(file.path, 'utf-8', 'scan'), - platform + platform, + {}, + messages ) } @@ -62,7 +66,8 @@ function parseGeminiJsonSessionContent( file: FileWithMtime, content: string, platform: NodeJS.Platform, - options: ResumableParseFinalizeOptions = {} + options: ResumableParseFinalizeOptions = {}, + messages?: TranscriptMessageSink ): AiVaultSession | null { const record = asRecord(JSON.parse(content) as unknown) if (!record) { @@ -71,7 +76,8 @@ function parseGeminiJsonSessionContent( const accumulator = createAccumulator({ agent: 'gemini', file, - sessionId: extractString(record.sessionId) ?? sessionIdFromFileName(file.path) + sessionId: extractString(record.sessionId) ?? sessionIdFromFileName(file.path), + messages }) updateTimeline(accumulator, extractString(record.startTime)) updateTimeline(accumulator, extractString(record.lastUpdated)) @@ -83,13 +89,14 @@ function parseGeminiJsonSessionContent( export async function parseGeminiJsonlSessionFile( file: FileWithMtime, - platform: NodeJS.Platform + platform: NodeJS.Platform, + messages?: TranscriptMessageSink ): Promise { const lines = createInterface({ input: openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan'), crlfDelay: Infinity }) - return parseGeminiJsonlSessionLines({ file, lines, platform }) + return parseGeminiJsonlSessionLines({ file, lines, platform, messages }) } function consumeGeminiJsonlRecordLine(accumulator: SessionAccumulator, line: string): void { @@ -114,10 +121,16 @@ function consumeGeminiJsonlRecordLine(accumulator: SessionAccumulator, line: str // Resumable only for the JSONL log format; Gemini's legacy single-JSON // session documents are rewritten in place and must be re-read whole. export function createGeminiJsonlSessionResumeState( - file: FileWithMtime + file: FileWithMtime, + messages?: TranscriptMessageSink ): ResumableSessionParseState { return accumulatorFoldResumeState( - createAccumulator({ agent: 'gemini', file, sessionId: sessionIdFromFileName(file.path) }), + createAccumulator({ + agent: 'gemini', + file, + sessionId: sessionIdFromFileName(file.path), + messages + }), consumeGeminiJsonlRecordLine ) } @@ -127,8 +140,9 @@ async function parseGeminiJsonlSessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ResumableParseFinalizeOptions + messages?: TranscriptMessageSink }): Promise { - const state = createGeminiJsonlSessionResumeState(args.file) + const state = createGeminiJsonlSessionResumeState(args.file, args.messages) for await (const line of args.lines) { state.consumeLine(line) } diff --git a/src/main/ai-vault/session-scanner-graph-parsers.ts b/src/main/ai-vault/session-scanner-graph-parsers.ts index d99893acaa2..581e885fd96 100644 --- a/src/main/ai-vault/session-scanner-graph-parsers.ts +++ b/src/main/ai-vault/session-scanner-graph-parsers.ts @@ -10,6 +10,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { accumulatorFoldResumeState, addPreviewContent, @@ -38,7 +39,8 @@ type ParserSessionOptions = { export async function parseRovoSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const metadata = asRecord( JSON.parse(await wslGatedReadFile(file.path, 'utf-8', 'scan')) as unknown @@ -49,7 +51,8 @@ export async function parseRovoSessionFile( const accumulator = createAccumulator({ agent: 'rovo', file, - sessionId: basename(dirname(file.path)) + sessionId: basename(dirname(file.path)), + messages }) accumulator.title = firstString(metadata, ['title', 'name', 'summary']) accumulator.cwd = firstString(metadata, [ @@ -174,12 +177,13 @@ export type MessageGraphAgent = 'openclaw' | 'pi' | 'omp' | 'prime-agent' export async function parseMessageGraphSessionFile( agent: MessageGraphAgent, file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const input = openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan') const lines = createInterface({ input, crlfDelay: Infinity }) try { - return await parseMessageGraphSessionLines({ agent, file, lines, platform }) + return await parseMessageGraphSessionLines({ agent, file, lines, platform, messages }) } finally { // readline.close() leaves the underlying stream open; destroy it so a // mid-parse throw cannot leak the gated transcript handle. @@ -245,10 +249,11 @@ function consumeMessageGraphRecordLine(accumulator: SessionAccumulator, line: st export function createMessageGraphSessionResumeState( agent: MessageGraphAgent, - file: FileWithMtime + file: FileWithMtime, + messages?: TranscriptMessageSink ): ResumableSessionParseState { const state = accumulatorFoldResumeState( - createAccumulator({ agent, file, sessionId: sessionIdFromFileName(file.path) }), + createAccumulator({ agent, file, sessionId: sessionIdFromFileName(file.path), messages }), consumeMessageGraphRecordLine ) // Why: only OMP materializes task-subagent transcripts beside its sessions @@ -263,8 +268,9 @@ async function parseMessageGraphSessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ParserSessionOptions + messages?: TranscriptMessageSink }): Promise { - const state = createMessageGraphSessionResumeState(args.agent, args.file) + const state = createMessageGraphSessionResumeState(args.agent, args.file, args.messages) for await (const line of args.lines) { state.consumeLine(line) } diff --git a/src/main/ai-vault/session-scanner-grok-parser.ts b/src/main/ai-vault/session-scanner-grok-parser.ts index 94e41e4b052..d9e88077c39 100644 --- a/src/main/ai-vault/session-scanner-grok-parser.ts +++ b/src/main/ai-vault/session-scanner-grok-parser.ts @@ -4,6 +4,7 @@ import { dirname, join } from 'node:path' import { createInterface } from 'node:readline' import type { AiVaultSession } from '../../shared/ai-vault-types' import type { FileWithMtime, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { addPreviewMessage, createAccumulator, @@ -34,7 +35,8 @@ const GROK_USER_QUERY_PREVIEW_SCAN_LIMIT = 4096 export async function parseGrokSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const record = asRecord(JSON.parse(await wslGatedReadFile(file.path, 'utf-8', 'scan')) as unknown) if (!record) { @@ -42,7 +44,7 @@ export async function parseGrokSessionFile( } const info = asRecord(record.info) const sessionId = extractString(info?.id) ?? sessionIdFromFileName(dirname(file.path)) - const accumulator = createAccumulator({ agent: 'grok', file, sessionId }) + const accumulator = createAccumulator({ agent: 'grok', file, sessionId, messages }) accumulator.cwd = extractString(info?.cwd) accumulator.title = normalizeTitleText(extractString(record.generated_title) ?? '') ?? diff --git a/src/main/ai-vault/session-scanner-hermes-parser.ts b/src/main/ai-vault/session-scanner-hermes-parser.ts index 017fd9c8fdc..f532fa8241d 100644 --- a/src/main/ai-vault/session-scanner-hermes-parser.ts +++ b/src/main/ai-vault/session-scanner-hermes-parser.ts @@ -2,6 +2,7 @@ import { wslGatedReadFile } from '../native-chat/wsl-transcript-fs-access' import type { AiVaultSession } from '../../shared/ai-vault-types' import type { ExecutionHostId } from '../../shared/execution-host' import type { FileWithMtime } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { addPreviewContent, createAccumulator, @@ -24,20 +25,34 @@ type ParserSessionOptions = { export async function parseHermesSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { - return parseHermesSessionContent( + return parseHermesSessionRecord( file, await wslGatedReadFile(file.path, 'utf-8', 'scan'), - platform + platform, + {}, + messages ) } +/** Remote transcript content, streamed from a host that has no reader attached. */ export async function parseHermesSessionContent( file: FileWithMtime, content: string, platform: NodeJS.Platform = process.platform, options: ParserSessionOptions = {} +): Promise { + return parseHermesSessionRecord(file, content, platform, options) +} + +async function parseHermesSessionRecord( + file: FileWithMtime, + content: string, + platform: NodeJS.Platform, + options: ParserSessionOptions, + messages?: TranscriptMessageSink ): Promise { const record = asRecord(JSON.parse(content) as unknown) if (!record) { @@ -46,7 +61,8 @@ export async function parseHermesSessionContent( const accumulator = createAccumulator({ agent: 'hermes', file, - sessionId: extractString(record.session_id) ?? sessionIdFromFileName(file.path) + sessionId: extractString(record.session_id) ?? sessionIdFromFileName(file.path), + messages }) accumulator.model = extractString(record.model) accumulator.cwd = extractString(record.cwd) diff --git a/src/main/ai-vault/session-scanner-kimi-parser.ts b/src/main/ai-vault/session-scanner-kimi-parser.ts index ba6deaa8fb8..cb5a93279ad 100644 --- a/src/main/ai-vault/session-scanner-kimi-parser.ts +++ b/src/main/ai-vault/session-scanner-kimi-parser.ts @@ -16,6 +16,7 @@ import { readKimiWorkDirBySessionId } from './session-scanner-kimi-paths' import type { FileWithMtime, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { asRecord, extractContentText, @@ -32,7 +33,8 @@ import { // session_index.jsonl; model/messages/tokens come from the wire transcript. export async function parseKimiSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { let stateRecord: Record | null try { @@ -53,7 +55,7 @@ export async function parseKimiSessionFile( } const sessionId = kimiSessionIdFromStatePath(file.path) - const accumulator = createAccumulator({ agent: 'kimi', file, sessionId }) + const accumulator = createAccumulator({ agent: 'kimi', file, sessionId, messages }) // Why: Kimi sessions are work-dir-scoped — the resume command must `cd` into // the original directory or the CLI rejects it. That path lives only in the diff --git a/src/main/ai-vault/session-scanner-opencode-parser.ts b/src/main/ai-vault/session-scanner-opencode-parser.ts index 30223e70866..2bc0245bcb0 100644 --- a/src/main/ai-vault/session-scanner-opencode-parser.ts +++ b/src/main/ai-vault/session-scanner-opencode-parser.ts @@ -3,6 +3,7 @@ import { WslTranscriptFsError } from '../native-chat/wsl-transcript-fs-gate' import { join } from 'node:path' import type { AiVaultSession } from '../../shared/ai-vault-types' import type { FileWithMtime, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { addPreviewMessage, createAccumulator, @@ -25,14 +26,15 @@ import { export async function parseOpenCodeSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const record = asRecord(JSON.parse(await wslGatedReadFile(file.path, 'utf-8', 'scan')) as unknown) if (!record) { return null } const sessionId = extractString(record.id) ?? sessionIdFromFileName(file.path) - const accumulator = createAccumulator({ agent: 'opencode', file, sessionId }) + const accumulator = createAccumulator({ agent: 'opencode', file, sessionId, messages }) accumulator.title = normalizeTitleText(extractString(record.title) ?? '') accumulator.cwd = extractString(record.directory) updateTimeline(accumulator, timeObjectValue(record.time, 'created')) diff --git a/src/main/ai-vault/session-scanner-opencode-sqlite-schema.ts b/src/main/ai-vault/session-scanner-opencode-sqlite-schema.ts new file mode 100644 index 00000000000..b7420c80043 --- /dev/null +++ b/src/main/ai-vault/session-scanner-opencode-sqlite-schema.ts @@ -0,0 +1,29 @@ +import type SyncDatabase from '../sqlite/sync-database' +import { columnExists, tableExists } from '../opencode-usage/schema-helpers' + +// Why: OpenCode's schema has moved more than once, so every read probes for the +// columns it names. These are the two shapes the session parser depends on; +// keeping them here stops each reader from inventing its own partial gate. + +/** Enough of `message` to count a session's turns. */ +export function canCountOpenCodeMessages(db: SyncDatabase): boolean { + return ( + tableExists(db, 'message') && + columnExists(db, 'message', 'session_id') && + columnExists(db, 'message', 'data') + ) +} + +/** Enough of `message`×`part` to read a session's parts in turn order. */ +export function canReadOpenCodeMessageParts(db: SyncDatabase): boolean { + return ( + canCountOpenCodeMessages(db) && + columnExists(db, 'message', 'id') && + // Every parts read orders by it; unprobed, a schema without it throws mid-read. + columnExists(db, 'message', 'time_created') && + tableExists(db, 'part') && + columnExists(db, 'part', 'message_id') && + columnExists(db, 'part', 'time_created') && + columnExists(db, 'part', 'data') + ) +} diff --git a/src/main/ai-vault/session-scanner-opencode-sqlite-worker-client.ts b/src/main/ai-vault/session-scanner-opencode-sqlite-worker-client.ts index e8fc78907ba..7d7772bcf1d 100644 --- a/src/main/ai-vault/session-scanner-opencode-sqlite-worker-client.ts +++ b/src/main/ai-vault/session-scanner-opencode-sqlite-worker-client.ts @@ -1,4 +1,4 @@ -import type { Worker } from 'node:worker_threads' +import { LazyWorkerThreadHost, type WorkerThreadFactory } from '../lazy-worker-thread-host' import type { AiVaultScanIssue, AiVaultSession } from '../../shared/ai-vault-types' import type { OpenCodeSqliteListRequest, @@ -11,9 +11,10 @@ import type { SessionFileCandidate } from './session-scanner-types' import { errorMessage } from './session-scanner-values' // Why (#8864): a lazily-spawned, unref'd worker runs OpenCode SQLite reads off -// the main-process event loop. Lifecycle (idle teardown, FIFO one-at-a-time -// dispatch, per-call timeouts, respawn-on-fault) mirrors src/main/speech/ -// stt-service.ts. The default spawn + shared singleton live in +// the main-process event loop. This module owns the request half (FIFO +// one-at-a-time dispatch, per-call timeouts, respawn-on-fault); the thread's +// own lifetime belongs to LazyWorkerThreadHost, shared with the port-scan probe +// client. The default spawn + shared singleton live in // session-scanner-opencode-sqlite-worker-spawn.ts. export const LIST_TIMEOUT_MS = 30_000 @@ -25,8 +26,6 @@ export const IDLE_TEARDOWN_MS = 30_000 // fresh scan burst starts from idle (so the cap is per-scan, not process-wide). export const MAX_CONSECUTIVE_DEATHS = 3 -export type WorkerFactory = () => Worker - // Omit collapses to the shared keys, so omit each member and let // the client stamp the correlation id. type OpenCodeSqliteRequestBody = @@ -53,20 +52,27 @@ class OpenCodeSqliteWorkerUnavailableError extends Error {} * no worker can be spawned rather than moving SQLite work onto the main thread. */ export class OpenCodeSqliteWorkerClient { - private worker: Worker | null = null private active: PendingCall | null = null private queue: PendingCall[] = [] - private idleTimer: NodeJS.Timeout | null = null private consecutiveDeaths = 0 private nextId = 1 - private loggedWorkerUnavailable = false - private cleanupWorkerListeners: (() => void) | null = null - private readonly workerFactory: WorkerFactory - private readonly log: (message: string) => void + private readonly host: LazyWorkerThreadHost - constructor(options: { workerFactory: WorkerFactory; log?: (message: string) => void }) { - this.workerFactory = options.workerFactory - this.log = options.log ?? ((message) => console.warn(message)) + constructor(options: { workerFactory: WorkerThreadFactory; log?: (message: string) => void }) { + const log = options.log ?? ((message: string) => console.warn(message)) + this.host = new LazyWorkerThreadHost({ + factory: options.workerFactory, + idleTeardownMs: IDLE_TEARDOWN_MS, + onMessage: (response) => this.onMessage(response), + onError: (error) => this.onWorkerFault(error), + onExit: (code) => this.onWorkerExit(code), + isIdle: () => !this.active && this.queue.length === 0, + // Why (#8864): never fall back to synchronous SQLite reads here; a missing + // bundle or resource-exhausted spawn must omit OpenCode history rather than + // reintroduce the main-process hang this worker boundary prevents. + onUnavailable: (err) => + log(`OpenCode SQLite worker unavailable; skipping its history. ${errorMessage(err)}`) + }) } /** @@ -167,7 +173,7 @@ export class OpenCodeSqliteWorkerClient { if (this.active || this.queue.length === 0) { return } - const worker = this.ensureWorker() + const worker = this.host.ensure() if (!worker) { this.failQueuedAsUnavailable() return @@ -177,7 +183,7 @@ export class OpenCodeSqliteWorkerClient { return } this.active = call - this.clearIdleTimer() + this.host.clearIdleTimer() // Timeout clock starts at dispatch (not enqueue): a batch may enqueue up to // 8 parses at once, and a queue-inclusive timeout would fire falsely. call.timer = setTimeout(() => this.onTimeout(call), call.timeoutMs) @@ -185,39 +191,6 @@ export class OpenCodeSqliteWorkerClient { worker.postMessage(call.request) } - private ensureWorker(): Worker | null { - if (this.worker) { - return this.worker - } - try { - const worker = this.workerFactory() - const onMessage = (response: OpenCodeSqliteWorkerResponse): void => this.onMessage(response) - const onError = (error: Error): void => this.onWorkerFault(error) - const onExit = (code: number): void => this.onWorkerExit(code) - worker.on('message', onMessage) - worker.on('error', onError) - worker.on('exit', onExit) - this.cleanupWorkerListeners = () => { - worker.off('message', onMessage) - worker.off('error', onError) - worker.off('exit', onExit) - } - // Never keep the app alive for a scan worker. - worker.unref?.() - this.worker = worker - return worker - } catch (err) { - // Why (#8864): never fall back to synchronous SQLite reads here; a missing - // bundle or resource-exhausted spawn must omit OpenCode history rather than - // reintroduce the main-process hang this worker boundary prevents. - if (!this.loggedWorkerUnavailable) { - this.loggedWorkerUnavailable = true - this.log(`OpenCode SQLite worker unavailable; skipping its history. ${errorMessage(err)}`) - } - return null - } - } - private onMessage(response: OpenCodeSqliteWorkerResponse): void { const call = this.active if (!call || call.request.id !== response.id) { @@ -243,7 +216,7 @@ export class OpenCodeSqliteWorkerClient { // A clean self-exit is not a death, but the stale handle must be dropped // or the next dispatch would post into the dead worker and stall to timeout. if (code === 0 && !this.active && this.queue.length === 0) { - this.destroyWorker() + this.host.destroy() return } this.onWorkerFault(new Error(`OpenCode SQLite worker exited with code ${code}`)) @@ -251,7 +224,7 @@ export class OpenCodeSqliteWorkerClient { private onWorkerFault(error: Error): void { const failed = this.active - this.destroyWorker() + this.host.destroy() this.consecutiveDeaths++ if (failed) { this.settle(failed, () => failed.reject(error)) @@ -302,46 +275,7 @@ export class OpenCodeSqliteWorkerClient { if (this.queue.length > 0) { this.pump() } else { - this.scheduleIdleTeardown() + this.host.scheduleIdleTeardown() } } - - private scheduleIdleTeardown(): void { - this.clearIdleTimer() - if (!this.worker) { - return - } - this.idleTimer = setTimeout(() => this.teardownIfIdle(), IDLE_TEARDOWN_MS) - this.idleTimer.unref?.() - } - - private teardownIfIdle(): void { - this.idleTimer = null - // Only tear down with nothing active AND nothing queued: a request arriving - // as the timer fires must never be lost to a self-exiting worker. - if (this.active || this.queue.length > 0) { - return - } - this.destroyWorker() - } - - private clearIdleTimer(): void { - if (this.idleTimer) { - clearTimeout(this.idleTimer) - this.idleTimer = null - } - } - - private destroyWorker(): void { - this.clearIdleTimer() - const worker = this.worker - this.worker = null - if (!worker) { - return - } - this.cleanupWorkerListeners?.() - this.cleanupWorkerListeners = null - worker.removeAllListeners() - void worker.terminate().catch(() => undefined) - } } diff --git a/src/main/ai-vault/session-scanner-opencode-sqlite.ts b/src/main/ai-vault/session-scanner-opencode-sqlite.ts index 322a38f214f..607534186da 100644 --- a/src/main/ai-vault/session-scanner-opencode-sqlite.ts +++ b/src/main/ai-vault/session-scanner-opencode-sqlite.ts @@ -10,6 +10,10 @@ import { shouldCaptureFullFirstUserPrompt } from './session-scanner-first-user-prompt' import { readOpenCodeDatabase } from './session-scanner-opencode-sqlite-open' +import { + canCountOpenCodeMessages, + canReadOpenCodeMessageParts +} from './session-scanner-opencode-sqlite-schema' import { normalizeTitleText } from './session-scanner-values' import type SyncDatabase from '../sqlite/sync-database' import { columnExists, tableExists } from '../opencode-usage/schema-helpers' @@ -70,14 +74,6 @@ function sessionNumberColumnSelect(db: SyncDatabase, columnName: string): string return columnExists(db, 'session', columnName) ? `s.${columnName}` : '0' } -function canCountOpenCodeMessages(db: SyncDatabase): boolean { - return ( - tableExists(db, 'message') && - columnExists(db, 'message', 'session_id') && - columnExists(db, 'message', 'data') - ) -} - function buildSessionQuery(db: SyncDatabase): string { const messageCountSubquery = canCountOpenCodeMessages(db) ? `(SELECT COUNT(*) FROM message m @@ -154,14 +150,7 @@ function extractPartText(partData: string): string | null { } function readFirstUserPromptFromOpenCodeDb(db: SyncDatabase, sessionId: string): string | null { - if ( - !canCountOpenCodeMessages(db) || - !tableExists(db, 'part') || - !columnExists(db, 'message', 'id') || - !columnExists(db, 'part', 'message_id') || - !columnExists(db, 'part', 'time_created') || - !columnExists(db, 'part', 'data') - ) { + if (!canReadOpenCodeMessageParts(db)) { return null } @@ -206,14 +195,7 @@ function readFirstUserPromptFromOpenCodeDb(db: SyncDatabase, sessionId: string): } function buildPreviewQuery(db: SyncDatabase): string | null { - if ( - !canCountOpenCodeMessages(db) || - !tableExists(db, 'part') || - !columnExists(db, 'message', 'id') || - !columnExists(db, 'part', 'message_id') || - !columnExists(db, 'part', 'time_created') || - !columnExists(db, 'part', 'data') - ) { + if (!canReadOpenCodeMessageParts(db)) { return null } return `SELECT json_extract(m.data, '$.role') AS role, diff --git a/src/main/ai-vault/session-scanner-parse-cache-agents.test.ts b/src/main/ai-vault/session-scanner-parse-cache-agents.test.ts index 9208b490fb1..cc086dd50e2 100644 --- a/src/main/ai-vault/session-scanner-parse-cache-agents.test.ts +++ b/src/main/ai-vault/session-scanner-parse-cache-agents.test.ts @@ -205,6 +205,61 @@ describe('codex-specific resume behavior', () => { }) describe('non-resumable formats keep reuse-only caching', () => { + it('re-parses cline when only its messages sidecar changed', async () => { + const root = await makeTempDir() + const sessionDir = join(root, 'cline-1') + await mkdir(sessionDir, { recursive: true }) + const metadataPath = join(sessionDir, 'cline-1.json') + const messagesPath = join(sessionDir, 'cline-1.messages.json') + await writeFile( + metadataPath, + JSON.stringify({ + session_id: 'cline-1', + cwd: '/tmp/cline', + started_at: '2026-05-01T10:00:00Z' + }) + ) + const writeMessages = (text: string): Promise => + writeFile( + messagesPath, + JSON.stringify({ + updated_at: '2026-05-01T10:00:01Z', + messages: [{ role: 'user', content: [{ type: 'text', text }] }] + }) + ) + await writeMessages('first ask') + + // Cline reads the sidecar as part of its parse, so a change to it has to + // re-parse; there is no metadata-only merge to re-run. + const candidate = async (): Promise => { + const base = await candidateFor('cline', metadataPath) + const sidecarStat = await stat(messagesPath) + return { + ...base, + file: { + ...base.file, + sidecar: { + path: messagesPath, + mtimeMs: sidecarStat.mtimeMs, + sizeBytes: sidecarStat.size + } + } + } + } + + const stats = createSessionParseStats() + const seeded = await parseAgentSessionFileCached(await candidate(), process.platform, stats) + expect(seeded?.title).toBe('first ask') + await parseAgentSessionFileCached(await candidate(), process.platform, stats) + expect(stats).toMatchObject({ fullParses: 1, reused: 1 }) + + await writeMessages('second ask, rather longer than the first') + const rewritten = await parseAgentSessionFileCached(await candidate(), process.platform, stats) + + expect(rewritten?.title).toBe('second ask, rather longer than the first') + expect(stats).toMatchObject({ fullParses: 2, reused: 1 }) + }) + it('re-parses a changed grok summary fully and reuses it when unchanged', async () => { const root = await makeTempDir() const sessionDir = join(root, 'session-1') diff --git a/src/main/ai-vault/session-scanner-parse-cache.ts b/src/main/ai-vault/session-scanner-parse-cache.ts index 139940ba005..46fb4754a32 100644 --- a/src/main/ai-vault/session-scanner-parse-cache.ts +++ b/src/main/ai-vault/session-scanner-parse-cache.ts @@ -1,7 +1,6 @@ -import { readTranscriptSlice } from '../native-chat/wsl-transcript-fs-access' import type { AiVaultSession } from '../../shared/ai-vault-types' +import { inSessionParseFileLane } from './session-parse-file-lane' import { createAntigravitySessionResumeState } from './session-scanner-antigravity-parser' -import { parseAgentSessionFile } from './session-scanner-agent-parser' import { createCodexSessionResumeState } from './session-scanner-codex-parser' import { createDroidSessionResumeState } from './session-scanner-droid-parser' import { createMessageGraphSessionResumeState } from './session-scanner-graph-parsers' @@ -13,27 +12,30 @@ import { countSubagentTranscripts } from './session-scanner-subagent-transcripts import { countOmpSubagentTranscripts } from './session-scanner-omp-subagent-transcripts' import type { ResumableSessionParseState, SessionFileCandidate } from './session-scanner-types' import { refreshCachedCodexTitle } from './session-scanner-codex-cached-title' -import { consumeCompleteJsonlLines } from './session-scanner-jsonl-reader' +import { + getSessionParseCacheEntry, + storeSessionParseCacheEntry, + type SessionParseCacheEntry +} from './session-parse-cache-store' +import type { TranscriptMessageSink } from './session-transcript-consumers' +import { sidecarUnchanged } from './session-sidecar-stat' +import { + enrichSessionFromSidecar, + sidecarEnrichesWithoutReparse +} from './session-scanner-sidecar-enrichment' +import { + readResumableTranscript, + readWholeTranscript, + type TranscriptReadStats +} from './session-transcript-reader' -// Sized past the default recency cap (1000) plus the in-scope cap (2000) so a -// full steady-state result set stays resident between forced rescans. -const MAX_CACHE_ENTRIES = 4096 -const NEWLINE_BYTE = 0x0a - -type ResumePoint = { - state: ResumableSessionParseState - // Byte offset just past the last complete ('\n'-terminated) line consumed; - // a trailing unterminated line is deliberately left before this point. - byteOffset: number -} - -type SessionParseCacheEntry = { - mtimeMs: number - sizeBytes: number | null - platform: NodeJS.Platform - session: AiVaultSession | null - resume: ResumePoint | null -} +export { + invalidateSessionParseCacheEntry, + resetSessionParseCacheForTests, + seedSessionParseCache, + snapshotSessionParseCacheForPersistence, + type PersistedSessionParseCacheEntry +} from './session-parse-cache-store' // Incremental append-parsing applies only to transcripts that are append-only // JSONL line-folds. Whole-JSON documents (grok/rovo/devin/hermes/gemini-json) @@ -44,31 +46,32 @@ type SessionParseCacheEntry = { // cached state instead, never pay for a throwaway accumulator. function resumableStateFactoryFor( candidate: SessionFileCandidate -): (() => ResumableSessionParseState) | null { +): ((messages: TranscriptMessageSink) => ResumableSessionParseState) | null { switch (candidate.agent) { case 'claude': - return () => createClaudeSessionResumeState(candidate.file) + return (messages) => createClaudeSessionResumeState(candidate.file, messages) case 'codex': - return () => createCodexSessionResumeState(candidate.file, candidate.codexHome) + return (messages) => + createCodexSessionResumeState(candidate.file, candidate.codexHome, messages) case 'cursor': - return () => createCursorSessionResumeState(candidate.file) + return (messages) => createCursorSessionResumeState(candidate.file, messages) case 'copilot': - return () => createCopilotSessionResumeState(candidate.file) + return (messages) => createCopilotSessionResumeState(candidate.file, messages) case 'droid': - return () => createDroidSessionResumeState(candidate.file) + return (messages) => createDroidSessionResumeState(candidate.file, messages) case 'openclaw': case 'pi': case 'omp': case 'prime-agent': { const agent = candidate.agent - return () => createMessageGraphSessionResumeState(agent, candidate.file) + return (messages) => createMessageGraphSessionResumeState(agent, candidate.file, messages) } case 'gemini': return candidate.file.path.endsWith('.jsonl') - ? () => createGeminiJsonlSessionResumeState(candidate.file) + ? (messages) => createGeminiJsonlSessionResumeState(candidate.file, messages) : null case 'antigravity': - return () => createAntigravitySessionResumeState(candidate.file) + return (messages) => createAntigravitySessionResumeState(candidate.file, messages) case 'devin': case 'grok': case 'hermes': @@ -80,245 +83,138 @@ function resumableStateFactoryFor( } } -export type SessionParseStats = { +export type SessionParseStats = TranscriptReadStats & { reused: number - incremental: number - fullParses: number - // Transcripts the parser already excluded (Codex workers), re-listed after a - // write and dismissed without reading. Counted apart from `incremental` so a - // scan span still shows how much work the early stop actually removed. - earlyStopped: number - bytesRead: number } export function createSessionParseStats(): SessionParseStats { return { reused: 0, incremental: 0, fullParses: 0, earlyStopped: 0, bytesRead: 0 } } -const cache = new Map() - -export function resetSessionParseCacheForTests(): void { - cache.clear() -} - -// Drops one entry after its file is deleted. Cleanliness, not correctness: -// discovery walks disk first, so a trashed file is never rediscovered anyway. -export function invalidateSessionParseCacheEntry(path: string): void { - cache.delete(path) -} - -// Persisted subset of a cache entry: the non-serializable `resume` parser -// state is dropped (see session-parse-cache-persistence.ts). -export type PersistedSessionParseCacheEntry = Omit - -export function snapshotSessionParseCacheForPersistence(): [ - string, - PersistedSessionParseCacheEntry -][] { - return [...cache].map(([path, entry]): [string, PersistedSessionParseCacheEntry] => [ - path, - { - mtimeMs: entry.mtimeMs, - sizeBytes: entry.sizeBytes, - platform: entry.platform, - session: entry.session - } - ]) -} - -// Seeded entries carry `resume: null`: after a restart an unchanged file is a -// cache hit; a file that changed while the app was closed pays one full -// (not incremental) re-parse. -export function seedSessionParseCache( - entries: Iterable<[string, PersistedSessionParseCacheEntry]> -): void { - const list = [...entries] - // Snapshot order is oldest→newest (LRU); an over-cap list keeps the newest - // tail rather than seeding the oldest entries and dropping the tail. - for (const [path, entry] of list.slice(Math.max(0, list.length - MAX_CACHE_ENTRIES))) { - if (cache.size >= MAX_CACHE_ENTRIES) { - return - } - // In-process entries are always fresher than persisted ones; never clobber. - if (cache.has(path)) { - continue - } - cache.set(path, { - mtimeMs: entry.mtimeMs, - sizeBytes: entry.sizeBytes, - platform: entry.platform, - session: entry.session, - resume: null - }) - } -} - -function storeEntry(path: string, entry: SessionParseCacheEntry): void { - cache.delete(path) - cache.set(path, entry) - if (cache.size > MAX_CACHE_ENTRIES) { - const oldest = cache.keys().next() - if (!oldest.done) { - cache.delete(oldest.value) - } - } -} - /** - * Parse a session file, reusing prior work where the file is provably - * unchanged (mtime+size) and, for append-only JSONL transcripts (Claude, - * Codex, Cursor, Copilot, Droid, OpenClaw/Pi/OMP, Gemini-JSONL), resuming the - * parse from the last consumed byte when the file only grew. This is what - * keeps the renderer's ~5s forced rescans from re-reading gigabytes of - * transcripts (STA-1278/STA-1417: main process pegging one core during - * multi-agent workloads). + * The session list's cursor over the transcript reader: it remembers what each + * file looked like when it was last listed, reuses that work where the file is + * provably unchanged (mtime+size), and otherwise asks the reader to resume from + * the last consumed byte or re-read the file whole. This is what keeps the + * renderer's ~5s forced rescans from re-reading gigabytes of transcripts + * (STA-1278/STA-1417: main process pegging one core during multi-agent + * workloads). Other consumers of the reader keep their own equivalent cursor + * and never consult this one. */ export async function parseAgentSessionFileCached( candidate: SessionFileCandidate, platform: NodeJS.Platform, stats?: SessionParseStats ): Promise { - const { file } = candidate - const entry = cache.get(file.path) + // The whole lookup-read-store sequence runs in the lane: a concurrent parse of + // the same path shares this entry's resume point and its message channel. + return inSessionParseFileLane(candidate.file.path, () => + parseCachedInLane(candidate, platform, stats) + ) +} - const unchanged = +async function parseCachedInLane( + candidate: SessionFileCandidate, + platform: NodeJS.Platform, + stats?: SessionParseStats +): Promise { + const { file } = candidate + const entry = getSessionParseCacheEntry(file.path) + + const transcriptUnchanged = entry !== undefined && entry.platform === platform && entry.mtimeMs === file.mtimeMs && (entry.sizeBytes === null || file.sizeBytes === undefined || entry.sizeBytes === file.sizeBytes) - if (unchanged) { - if (stats) { - stats.reused++ + if (transcriptUnchanged) { + if (sidecarUnchanged(entry.sidecar, file.sidecar)) { + return reuseCachedSession(candidate, entry, stats) } - // A zero-turn transcript usually never changes again, but its sibling - // subagent dir (Claude `/subagents/`, OMP's same-named artifact - // dir) can gain files after the parent's last write (a still-running - // subagent finishing). The mtime+size key can't see that, so refresh the - // cheap directory count on reuse. - if (entry.session && entry.session.messageCount === 0) { - const subagentTranscriptCount = - candidate.agent === 'claude' - ? await countSubagentTranscripts(file.path) - : candidate.agent === 'omp' - ? await countOmpSubagentTranscripts(file.path) - : null - if ( - subagentTranscriptCount !== null && - subagentTranscriptCount !== entry.session.subagentTranscriptCount - ) { - entry.session = { ...entry.session, subagentTranscriptCount } + // Only the sibling moved. For an agent whose sibling just adds metadata, + // re-merge it onto the stored fold result; the transcript is not re-read. + if (sidecarEnrichesWithoutReparse(candidate) && entry.foldSession !== undefined) { + const enriched = await enrichSessionFromSidecar(candidate, entry.foldSession, platform) + entry.session = enriched.session + entry.sidecar = enriched.refused ? 'unknown' : file.sidecar + storeSessionParseCacheEntry(file.path, entry) + if (stats) { + stats.reused++ } + return entry.session } - // Codex titles come from session_index.jsonl, which mtime+size can't see. - // Remote counterpart: remote-session-scanner.ts's reusedCodexTitleRefresh. - if (entry.session && candidate.agent === 'codex') { - entry.session = await refreshCachedCodexTitle(candidate, entry.session) - } - storeEntry(file.path, entry) - return entry.session } const stateFactory = resumableStateFactoryFor(candidate) if (stateFactory) { - const parsed = await parseResumableCandidate({ + const read = await readResumableTranscript({ candidate, platform, - entry, - stats, - stateFactory + resume: entry?.platform === platform ? entry.resume : null, + stateFactory, + stats }) - storeEntry(file.path, parsed) - return parsed.session + const enriched = await enrichSessionFromSidecar(candidate, read.session, platform) + storeSessionParseCacheEntry(file.path, { + mtimeMs: file.mtimeMs, + sizeBytes: file.sizeBytes ?? null, + platform, + session: enriched.session, + // A refused sibling leaves the transcript's own work cached and resumable; + // only the sibling is recorded as unknown, so the next healthy scan + // re-merges it without re-reading the transcript. + sidecar: enriched.refused ? 'unknown' : file.sidecar, + foldSession: read.session, + resume: read.resume + }) + return enriched.session } - if (stats) { - stats.fullParses++ - stats.bytesRead += file.sizeBytes ?? 0 - } - const session = await parseAgentSessionFile(candidate, platform) - storeEntry(file.path, { + const session = await readWholeTranscript({ candidate, platform, stats }) + storeSessionParseCacheEntry(file.path, { mtimeMs: file.mtimeMs, sizeBytes: file.sizeBytes ?? null, platform, session, + // A whole-file parse reads the sibling itself, so a change to it re-parses. + sidecar: file.sidecar, + foldSession: session, resume: null }) return session } -async function parseResumableCandidate(args: { - candidate: SessionFileCandidate - platform: NodeJS.Platform - entry: SessionParseCacheEntry | undefined +async function reuseCachedSession( + candidate: SessionFileCandidate, + entry: SessionParseCacheEntry, stats?: SessionParseStats - stateFactory: () => ResumableSessionParseState -}): Promise { - const { file } = args.candidate - const resume = args.entry?.platform === args.platform ? args.entry.resume : null - const canResume = - resume !== null && - resume !== undefined && - typeof file.sizeBytes === 'number' && - file.sizeBytes >= resume.byteOffset && - (resume.byteOffset === 0 || (await endsWithNewlineAt(file.path, resume.byteOffset))) - - // Clone before consuming: a failed read must not corrupt the cached state, - // or the next resume would double-count the lines applied before the error. - const state = canResume ? resume.state.clone() : args.stateFactory() - const startOffset = canResume ? resume.byteOffset : 0 - // Mirrors the reader's entry guard so a dismissed transcript is not reported - // as an incremental parse that read nothing. - const stoppedBeforeRead = state.shouldStop?.() === true - if (args.stats) { - if (stoppedBeforeRead) { - args.stats.earlyStopped++ - } else if (canResume) { - args.stats.incremental++ - } else { - args.stats.fullParses++ +): Promise { + if (stats) { + stats.reused++ + } + // A zero-turn transcript usually never changes again, but its sibling + // subagent dir (Claude `/subagents/`, OMP's same-named artifact + // dir) can gain files after the parent's last write (a still-running + // subagent finishing). The mtime+size key can't see that, so refresh the + // cheap directory count on reuse. + if (entry.session && entry.session.messageCount === 0) { + const subagentTranscriptCount = + candidate.agent === 'claude' + ? await countSubagentTranscripts(candidate.file.path) + : candidate.agent === 'omp' + ? await countOmpSubagentTranscripts(candidate.file.path) + : null + if ( + subagentTranscriptCount !== null && + subagentTranscriptCount !== entry.session.subagentTranscriptCount + ) { + entry.session = { ...entry.session, subagentTranscriptCount } } } - - const readResult = await consumeCompleteJsonlLines({ - path: file.path, - start: startOffset, - onLine: (line) => state.consumeLine(line), - // Bound: the optional hooks are declared as methods, so a parser written - // with method syntax must not lose `this` on the way into the reader. - onLineBytes: state.consumeLineBytes?.bind(state), - shouldStop: state.shouldStop?.bind(state) - }) - if (args.stats) { - args.stats.bytesRead += readResult.bytesRead - } - - // The stat this scan displays is current even when nothing new was consumed. - state.touchFile(file) - - // Keep parity with the one-shot parser: a final unterminated line is shown, - // but stays out of the resumable state so the (possibly still-growing) line - // is re-read once complete instead of being half-counted. - let displayState = state - if (readResult.trailingPartialLine !== null) { - displayState = state.clone() - displayState.consumeLine(readResult.trailingPartialLine) - } - - return { - mtimeMs: file.mtimeMs, - sizeBytes: file.sizeBytes ?? null, - platform: args.platform, - session: await displayState.finalize(args.platform), - resume: { state, byteOffset: readResult.consumedThrough } + // Codex titles come from session_index.jsonl, which mtime+size can't see. + // Remote counterpart: remote-session-scanner.ts's reusedCodexTitleRefresh. + if (entry.session && candidate.agent === 'codex') { + entry.session = await refreshCachedCodexTitle(candidate, entry.session) } -} - -// A resume point is only valid if it still sits just past a line break; -// anything else means the file was rewritten, not appended. Heuristic: a -// grown rewrite keeping '\n' at exactly this byte would slip through, but -// agent transcripts are append-only so that trade is accepted (worst case is -// a stale vault row until the file is next truncated or the app restarts). -async function endsWithNewlineAt(path: string, offset: number): Promise { - const slice = await readTranscriptSlice(path, offset - 1, 1, 'scan') - return slice.length === 1 && slice[0] === NEWLINE_BYTE + storeSessionParseCacheEntry(candidate.file.path, entry) + return entry.session } diff --git a/src/main/ai-vault/session-scanner-primary-parsers.ts b/src/main/ai-vault/session-scanner-primary-parsers.ts index f554268c7e5..9783f8a0520 100644 --- a/src/main/ai-vault/session-scanner-primary-parsers.ts +++ b/src/main/ai-vault/session-scanner-primary-parsers.ts @@ -10,6 +10,7 @@ import type { ResumableSessionParseState, SessionAccumulator } from './session-scanner-types' +import type { TranscriptMessageSink } from './session-transcript-consumers' import { addPreviewContent, createAccumulator, @@ -42,12 +43,16 @@ export type ClaudeSessionParseState = { firstUserTitle: string | null } -export function createClaudeSessionParseState(file: FileWithMtime): ClaudeSessionParseState { +export function createClaudeSessionParseState( + file: FileWithMtime, + messages?: TranscriptMessageSink +): ClaudeSessionParseState { return { accumulator: createAccumulator({ agent: 'claude', file, - sessionId: sessionIdFromFileName(file.path) + sessionId: sessionIdFromFileName(file.path), + messages }), metaTitle: null, generatedTitle: null, @@ -188,8 +193,11 @@ export async function finalizeClaudeSessionParseState( return finalizeSession(snapshot.accumulator, platform, options) } -export function createClaudeSessionResumeState(file: FileWithMtime): ResumableSessionParseState { - return claudeResumeStateFromParseState(createClaudeSessionParseState(file)) +export function createClaudeSessionResumeState( + file: FileWithMtime, + messages?: TranscriptMessageSink +): ResumableSessionParseState { + return claudeResumeStateFromParseState(createClaudeSessionParseState(file, messages)) } function claudeResumeStateFromParseState( @@ -207,13 +215,14 @@ function claudeResumeStateFromParseState( export async function parseClaudeSessionFile( file: FileWithMtime, - platform: NodeJS.Platform = process.platform + platform: NodeJS.Platform = process.platform, + messages?: TranscriptMessageSink ): Promise { const lines = createInterface({ input: openTranscriptReadStream(file.path, { encoding: 'utf-8' }, 'scan'), crlfDelay: Infinity }) - return parseClaudeSessionLines({ file, lines, platform }) + return parseClaudeSessionLines({ file, lines, platform, messages }) } export async function parseClaudeSessionContent( @@ -236,8 +245,9 @@ async function parseClaudeSessionLines(args: { lines: AsyncIterable | Iterable platform: NodeJS.Platform options?: ParserSessionOptions + messages?: TranscriptMessageSink }): Promise { - const state = createClaudeSessionParseState(args.file) + const state = createClaudeSessionParseState(args.file, args.messages) for await (const line of args.lines) { consumeClaudeSessionLine(state, line) } diff --git a/src/main/ai-vault/session-scanner-sidecar-enrichment.ts b/src/main/ai-vault/session-scanner-sidecar-enrichment.ts new file mode 100644 index 00000000000..983e200e987 --- /dev/null +++ b/src/main/ai-vault/session-scanner-sidecar-enrichment.ts @@ -0,0 +1,79 @@ +import type { AiVaultSession } from '../../shared/ai-vault-types' +import { buildAiVaultResumeCommand } from '../../shared/ai-vault-resume-command' +import { generatedSessionTitle } from './session-scanner-accumulator' +import { readCursorChatMeta, wasCursorChatMetaRefused } from './session-scanner-cursor-chat-meta' +import type { SessionFileCandidate } from './session-scanner-types' + +/** + * Merges an agent's sibling file onto the session its transcript alone + * produced. Kept out of the fold and applied here so it is a pure function of + * (fold result, sibling): re-running it starts from what the transcript said, + * never from a previous merge, so a rewritten sibling replaces the fields it + * supplied last time instead of losing to them. + * + * The reuse-time counterpart of session-scanner-codex-cached-title.ts, for + * agents whose sibling the transcript key cannot see. + */ + +export type SidecarEnrichment = { + session: AiVaultSession | null + /** The sibling could not be read; the caller records the observation as unknown. */ + refused: boolean +} + +/** True when the sibling only adds metadata, so a change to it needs no re-parse. */ +export function sidecarEnrichesWithoutReparse(candidate: SessionFileCandidate): boolean { + return candidate.agent === 'cursor' +} + +export async function enrichSessionFromSidecar( + candidate: SessionFileCandidate, + foldSession: AiVaultSession | null, + platform: NodeJS.Platform +): Promise { + if (candidate.agent !== 'cursor' || !foldSession) { + return { session: foldSession, refused: false } + } + const meta = await readCursorChatMeta(candidate.file.path) + if (!meta) { + return { session: foldSession, refused: wasCursorChatMetaRefused(candidate.file.path) } + } + return { session: mergeCursorChatMeta(foldSession, meta, platform), refused: false } +} + +/** Fills only what the transcript never recorded; its own records always win. */ +export function mergeCursorChatMeta( + session: AiVaultSession, + meta: { + title: string | null + cwd: string | null + createdAt: string | null + updatedAt: string | null + }, + platform: NodeJS.Platform +): AiVaultSession { + // A generated title means the fold found none, so the sibling's may stand in. + const named = session.title !== generatedSessionTitle(session.agent, session.sessionId) + const cwd = session.cwd ?? meta.cwd + const merged: AiVaultSession = { + ...session, + title: named ? session.title : (meta.title ?? session.title), + cwd, + createdAt: session.createdAt ?? meta.createdAt, + updatedAt: session.updatedAt ?? meta.updatedAt + } + if (cwd === session.cwd) { + return merged + } + // The resume command embeds the cwd, so it has to be rebuilt with it. + return { + ...merged, + resumeCommand: buildAiVaultResumeCommand({ + agent: merged.agent, + sessionId: merged.sessionId, + resumeFilePath: merged.filePath, + cwd, + platform + }) + } +} diff --git a/src/main/ai-vault/session-scanner-source-discovery.ts b/src/main/ai-vault/session-scanner-source-discovery.ts index fef37ebcda0..d62fdd4d19d 100644 --- a/src/main/ai-vault/session-scanner-source-discovery.ts +++ b/src/main/ai-vault/session-scanner-source-discovery.ts @@ -17,6 +17,8 @@ export async function discoverAiVaultSessionSources(args: { const { options, limitPerAgent, issues } = args const wslHomeDirs = normalizedWslHomeDirs(options.wslHomeDirs) + // The Cursor chat-meta scan scope is owned by scanAiVaultSessions: it has to + // span parse as well, and finalize runs after this returns. return Promise.all([ // Why: OpenCode 1.17.x migrated sessions from per-session JSON files to a // SQLite DB. discoverOpenCodeSessions runs both the file scanner (legacy) diff --git a/src/main/ai-vault/session-scanner-types.ts b/src/main/ai-vault/session-scanner-types.ts index 6216e3d7cf8..b1d480aa944 100644 --- a/src/main/ai-vault/session-scanner-types.ts +++ b/src/main/ai-vault/session-scanner-types.ts @@ -5,6 +5,8 @@ import type { AiVaultSessionPreviewMessage } from '../../shared/ai-vault-types' import type { ExecutionHostId } from '../../shared/execution-host' +import type { TranscriptMessageSink } from './session-transcript-consumers' +import type { SessionSidecarObservation } from './session-sidecar-stat' export type AiVaultScanOptions = { claudeProjectsDir?: string @@ -54,8 +56,12 @@ export type FileWithMtime = { modifiedAt: string // Present when discovery statted the file; lets the parse cache detect // unchanged/truncated files without a second stat. Synthetic candidates - // such as OpenCode SQLite rows omit it. + // such as OpenCode SQLite rows omit it. The transcript's own length: a byte + // offset into it may be compared against this directly. sizeBytes?: number + // What discovery saw of the agent's sibling file, tracked apart from the + // transcript's own stat (see session-sidecar-stat.ts). + sidecar?: SessionSidecarObservation // Present when discovery can prove filesystem identity. Codex dual-root // scans use a multi-link inode to collapse only actual hardlink aliases. dev?: number @@ -108,6 +114,9 @@ export type ResumableSessionParseState = { export type SessionAccumulator = { agent: AiVaultAgent + // Every decoded message this fold sees also goes here, for the reader's + // consumers. Shared by clones on purpose: one read, one message stream. + messages: TranscriptMessageSink sessionId: string title: string | null fallbackTitle: string | null diff --git a/src/main/ai-vault/session-scanner.ts b/src/main/ai-vault/session-scanner.ts index 3d27dccd1ef..8c3ac157e4f 100644 --- a/src/main/ai-vault/session-scanner.ts +++ b/src/main/ai-vault/session-scanner.ts @@ -6,19 +6,13 @@ import type { import { LOCAL_EXECUTION_HOST_ID, type ExecutionHostId } from '../../shared/execution-host' import { withSpan } from '../observability/tracer' import { sessionSortTime } from './session-scanner-accumulator' -import { - codexRolloutHardlinkIdentity, - dedupeCodexRolloutAliases, - dedupeCodexSessionsBySessionId -} from './codex-session-root-dedup' -import { readCodexRolloutSessionMetaId } from '../codex/codex-rollout-session-meta' +import { dedupeCodexSessionsBySessionId } from './codex-session-root-dedup' import { createAntigravityWorkspaceResolver, readLocalAntigravityHistory, type AntigravityWorkspaceResolver } from './session-scanner-antigravity-history' -import { antigravityHistoryPathForBrainDir } from './session-scanner-antigravity-paths' -import { codexHomeForSessionsDir } from './session-scanner-codex-paths' +import { sessionCandidatesFromDiscoveries } from './session-scanner-candidates' import { ensureSessionParseCacheLoaded, scheduleSessionParseCachePersist @@ -30,10 +24,8 @@ import { } from './session-scanner-parse-cache' import { recordSessionScanIssue } from './session-scan-issues' import { discoverInScopeClaudeFiles } from './session-scanner-scope-discovery' -import { - DEFAULT_CODEX_HOME_DIR, - discoverAiVaultSessionSources -} from './session-scanner-source-discovery' +import { discoverAiVaultSessionSources } from './session-scanner-source-discovery' +import { cursorChatMetaRefusals, withCursorChatMetaScan } from './session-scanner-cursor-chat-meta' import type { AiVaultScanOptions, SessionFileCandidate, @@ -62,103 +54,87 @@ export async function scanAiVaultSessions( // The span makes scan cost visible in the local trace file: STA-1278-style // "one core pegged" reports need to show whether transcript scanning is the // subsystem burning CPU, and how much of each scan the cache absorbed. - return withSpan('aiVault.scan', async (span) => { - const limit = options.unlimited - ? Number.POSITIVE_INFINITY - : clampPositiveInteger(options.limit, DEFAULT_AI_VAULT_SCAN_LIMIT) - const limitPerAgent = options.unlimited - ? Number.POSITIVE_INFINITY - : clampPositiveInteger(options.limitPerAgent, limit * SESSION_PARSE_CANDIDATE_MULTIPLIER) - const platform = options.platform ?? process.platform - const executionHostId = options.executionHostId ?? LOCAL_EXECUTION_HOST_ID - const issues: AiVaultScanIssue[] = [] - const parseStats = createSessionParseStats() - const antigravityWorkspaceResolver = createAntigravityWorkspaceResolver( - readLocalAntigravityHistory - ) - // Why: persisted entries must be seeded before any candidate is parsed, or - // the cold scan gains nothing from the cache file (#9210). - throwIfAiVaultScanCancelled(options.signal) - await ensureSessionParseCacheLoaded() - const discoveries = await discoverAiVaultSessionSources({ options, limitPerAgent, issues }) - throwIfAiVaultScanCancelled(options.signal) + // The Cursor chat-meta scope spans discovery AND parse: its sibling meta.json + // is looked up in both phases, and one scan must read the chats tree once. + return withSpan('aiVault.scan', (span) => + withCursorChatMetaScan(async () => { + const limit = options.unlimited + ? Number.POSITIVE_INFINITY + : clampPositiveInteger(options.limit, DEFAULT_AI_VAULT_SCAN_LIMIT) + const limitPerAgent = options.unlimited + ? Number.POSITIVE_INFINITY + : clampPositiveInteger(options.limitPerAgent, limit * SESSION_PARSE_CANDIDATE_MULTIPLIER) + const platform = options.platform ?? process.platform + const executionHostId = options.executionHostId ?? LOCAL_EXECUTION_HOST_ID + const issues: AiVaultScanIssue[] = [] + const parseStats = createSessionParseStats() + const antigravityWorkspaceResolver = createAntigravityWorkspaceResolver( + readLocalAntigravityHistory + ) + // Why: persisted entries must be seeded before any candidate is parsed, or + // the cold scan gains nothing from the cache file (#9210). + throwIfAiVaultScanCancelled(options.signal) + await ensureSessionParseCacheLoaded() + const discoveries = await discoverAiVaultSessionSources({ options, limitPerAgent, issues }) + throwIfAiVaultScanCancelled(options.signal) - const candidates = await dedupeCodexRolloutAliases( - discoveries - .flatMap((discovery) => - discovery.files.map((file): SessionFileCandidate => ({ - agent: discovery.agent, - file, - codexHome: - discovery.agent === 'codex' - ? codexHomeForSessionsDir( - discovery.rootDir, - options.defaultCodexHomeDir ?? DEFAULT_CODEX_HOME_DIR - ) - : null, - antigravityHistoryPath: - discovery.agent === 'antigravity' - ? antigravityHistoryPathForBrainDir(discovery.rootDir) - : undefined - })) - ) - .sort((left, right) => right.file.mtimeMs - left.file.mtimeMs), - { - isCodex: (candidate) => candidate.agent === 'codex', - getFilePath: (candidate) => candidate.file.path, - getCodexHome: (candidate) => candidate.codexHome, - getHardlinkIdentity: (candidate) => codexRolloutHardlinkIdentity(candidate.file) - }, - (filePath) => readCodexRolloutSessionMetaId(filePath, options.signal, 'scan'), - options.signal - ) + const candidates = await sessionCandidatesFromDiscoveries(discoveries, options) - const parsedSessions = await parseSessionCandidates({ - candidates: candidates.slice(0, limit * SESSION_PARSE_CANDIDATE_MULTIPLIER), - limit, - platform, - executionHostId, - issues, - parseStats, - signal: options.signal, - antigravityWorkspaceResolver + const parsedSessions = await parseSessionCandidates({ + candidates: candidates.slice(0, limit * SESSION_PARSE_CANDIDATE_MULTIPLIER), + limit, + platform, + executionHostId, + issues, + parseStats, + signal: options.signal, + antigravityWorkspaceResolver + }) + + const cappedSessions = dedupeCodexSessionsBySessionId(parsedSessions) + .sort((left, right) => sessionSortTime(right) - sessionSortTime(left)) + .slice(0, limit) + + const scopeSessions = await scanInScopeSessions({ + discoveries, + scopePaths: options.scopePaths ?? [], + limit, + alreadyParsedFilePaths: new Set(cappedSessions.map((session) => session.filePath)), + platform, + executionHostId, + issues, + parseStats, + signal: options.signal + }) + // Scope discovery can return without parsing anything, so an abort landing + // here would otherwise persist and return a cancelled scan as complete. + throwIfAiVaultScanCancelled(options.signal) + for (const refusal of cursorChatMetaRefusals()) { + // One issue per refused chats root, not one per Cursor transcript. + recordSessionScanIssue(issues, { + agent: 'cursor', + path: refusal.chatsRoot, + message: refusal.message + }) + } + + span.setAttribute('candidates', candidates.length) + span.setAttribute('reused', parseStats.reused) + span.setAttribute('incremental', parseStats.incremental) + span.setAttribute('fullParses', parseStats.fullParses) + span.setAttribute('earlyStopped', parseStats.earlyStopped) + span.setAttribute('bytesRead', parseStats.bytesRead) + span.setAttribute('issues', issues.length) + + scheduleSessionParseCachePersist(parseStats) + + return { + sessions: mergeSessions(cappedSessions, scopeSessions), + issues: issues.map((issue) => ({ executionHostId, ...issue })), + scannedAt: new Date().toISOString() + } }) - - const cappedSessions = dedupeCodexSessionsBySessionId(parsedSessions) - .sort((left, right) => sessionSortTime(right) - sessionSortTime(left)) - .slice(0, limit) - - const scopeSessions = await scanInScopeSessions({ - discoveries, - scopePaths: options.scopePaths ?? [], - limit, - alreadyParsedFilePaths: new Set(cappedSessions.map((session) => session.filePath)), - platform, - executionHostId, - issues, - parseStats, - signal: options.signal - }) - // Scope discovery can return without parsing anything, so an abort landing - // here would otherwise persist and return a cancelled scan as complete. - throwIfAiVaultScanCancelled(options.signal) - - span.setAttribute('candidates', candidates.length) - span.setAttribute('reused', parseStats.reused) - span.setAttribute('incremental', parseStats.incremental) - span.setAttribute('fullParses', parseStats.fullParses) - span.setAttribute('earlyStopped', parseStats.earlyStopped) - span.setAttribute('bytesRead', parseStats.bytesRead) - span.setAttribute('issues', issues.length) - - scheduleSessionParseCachePersist(parseStats) - - return { - sessions: mergeSessions(cappedSessions, scopeSessions), - issues: issues.map((issue) => ({ executionHostId, ...issue })), - scannedAt: new Date().toISOString() - } - }) + ) } // In-scope sessions are guaranteed regardless of the recency cap, so the global diff --git a/src/main/ai-vault/session-sidecar-stat.test.ts b/src/main/ai-vault/session-sidecar-stat.test.ts new file mode 100644 index 00000000000..e8d384d6f6b --- /dev/null +++ b/src/main/ai-vault/session-sidecar-stat.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it } from 'vitest' +import { sidecarUnchanged, type SessionSidecarObservation } from './session-sidecar-stat' + +const META = { path: '/chats/a/meta.json', mtimeMs: 100, sizeBytes: 20 } as const +const OTHER = { path: '/chats/a/meta.json', mtimeMs: 101, sizeBytes: 20 } as const + +type Named = [label: string, value: SessionSidecarObservation | undefined] + +const ENTRIES: Named[] = [ + ['undefined', undefined], + ["'none'", 'none'], + ["'unknown'", 'unknown'], + ['object', { ...META }] +] +const OBSERVED: Named[] = [ + ['undefined', undefined], + ["'none'", 'none'], + ["'unknown'", 'unknown'], + ['same object', { ...META }], + ['different object', { ...OTHER }] +] + +// entry (row) x observed (column). A `true` cell is a cache hit. +const TRUTH_TABLE: Record> = { + undefined: { + undefined: true, + "'none'": true, + "'unknown'": false, + 'same object': false, + 'different object': false + }, + "'none'": { + undefined: true, + "'none'": true, + "'unknown'": false, + 'same object': false, + 'different object': false + }, + "'unknown'": { + undefined: false, + "'none'": false, + "'unknown'": false, + 'same object': false, + 'different object': false + }, + object: { + undefined: false, + "'none'": false, + "'unknown'": false, + 'same object': true, + 'different object': false + } +} + +describe.each(ENTRIES)('cached %s', (entryLabel, entry) => { + it.each(OBSERVED)(`vs observed %s`, (observedLabel, observed) => { + expect(sidecarUnchanged(entry, observed)).toBe(TRUTH_TABLE[entryLabel][observedLabel]) + }) +}) + +it('treats a vanished sidecar as a change, not as "never had one"', () => { + expect(sidecarUnchanged({ ...META }, 'none')).toBe(false) +}) + +it('never concludes anything from an unreadable sidecar, in either position', () => { + expect(sidecarUnchanged('unknown', 'none')).toBe(false) + expect(sidecarUnchanged('unknown', { ...META })).toBe(false) + expect(sidecarUnchanged({ ...META }, 'unknown')).toBe(false) +}) + +it('keeps an agent with no sidecar at all a cache hit', () => { + expect(sidecarUnchanged(undefined, undefined)).toBe(true) + expect(sidecarUnchanged(undefined, 'none')).toBe(true) +}) diff --git a/src/main/ai-vault/session-sidecar-stat.ts b/src/main/ai-vault/session-sidecar-stat.ts new file mode 100644 index 00000000000..e176da342e2 --- /dev/null +++ b/src/main/ai-vault/session-sidecar-stat.ts @@ -0,0 +1,49 @@ +// Why: some agents keep part of a session beside its transcript — Cursor's +// chat meta.json, Cline's messages file. Folding that file's stat into the +// transcript's own mtime/size makes one key mean two things, so a byte offset +// into the transcript can no longer be compared against it and a refused read +// of the sibling takes the transcript down with it. The sidecar is observed +// separately and compared separately. + +export type SessionSidecarStat = { + path: string + mtimeMs: number + sizeBytes: number +} + +export type SessionSidecarObservation = + | SessionSidecarStat + /** This agent declares no sidecar, or it does not exist. */ + | 'none' + /** It could not be read this scan; nothing may be concluded from its absence. */ + | 'unknown' + +/** + * Whether a cached observation still describes what discovery just saw. + * + * Asymmetric on purpose: `file` is observed now, so a missing value means the + * agent has no sidecar, while `entry` may predate the field (an entry seeded + * from a cache file an older build wrote), so a missing value means unknown. + * + * `'none'` is a claim, not an absence of one: a sidecar that was there and is + * gone changed, and one that was unreadable last time is still unknown now. + */ +export function sidecarUnchanged( + entry: SessionSidecarObservation | undefined, + file: SessionSidecarObservation | undefined +): boolean { + const observed = file ?? 'none' + if (observed === 'unknown' || entry === 'unknown') { + return false + } + if (observed === 'none') { + // Absent now: a hit only if it was absent before, or the agent never had one. + return entry === undefined || entry === 'none' + } + return ( + typeof entry === 'object' && + entry.path === observed.path && + entry.mtimeMs === observed.mtimeMs && + entry.sizeBytes === observed.sizeBytes + ) +} diff --git a/src/main/ai-vault/session-title-file-reader.ts b/src/main/ai-vault/session-title-file-reader.ts index 15228d132bb..bfbb4e073dd 100644 --- a/src/main/ai-vault/session-title-file-reader.ts +++ b/src/main/ai-vault/session-title-file-reader.ts @@ -31,6 +31,9 @@ async function readOneTitle( if (!stats.isFile() || signal?.aborted) { return null } + // Why: this key is a raw lstat with no content dependency, so it only + // matches the scanner's for providers that declare none — today claude and + // codex, which is all this request type carries. const session = await parseAgentSessionFileCached( { agent: request.agent, diff --git a/src/main/ai-vault/session-transcript-channel.ts b/src/main/ai-vault/session-transcript-channel.ts new file mode 100644 index 00000000000..1b1b3950949 --- /dev/null +++ b/src/main/ai-vault/session-transcript-channel.ts @@ -0,0 +1,92 @@ +import { + hasTranscriptConsumers, + transcriptConsumers, + type TranscriptMessage, + type TranscriptMessageSink, + type TranscriptReadConsumer, + type TranscriptReadOutcome, + type TranscriptReadStart +} from './session-transcript-consumers' + +/** + * The sink a parser pushes into, and the fan-out to every registered consumer. + * + * One channel belongs to one file for as long as its resumable parse state + * lives, because the cached state (and every clone of it) holds this reference. + * A read re-points the channel at that read's consumers instead of replacing it. + */ +export class TranscriptMessageChannel implements TranscriptMessageSink { + private readers: TranscriptReadConsumer[] = [] + + private muted = false + + /** True while a read is open with at least one consumer attached. */ + get active(): boolean { + return this.readers.length > 0 + } + + beginRead(start: TranscriptReadStart): void { + this.muted = false + this.readers = [] + // Keeps a scan with no consumers allocation-free on its hottest path. + if (!hasTranscriptConsumers()) { + return + } + for (const consumer of transcriptConsumers()) { + try { + const reader = consumer.beginRead(start) + if (reader) { + this.readers.push(reader) + } + } catch { + // A consumer that cannot open this read simply does not see it. + } + } + } + + push(message: TranscriptMessage): void { + if (this.muted || this.readers.length === 0) { + return + } + // A throwing consumer is dropped for the rest of the read rather than + // failing the parse; it then gets no `finish`, so it never records a cursor + // for a stream it did not see in full. + let index = 0 + while (index < this.readers.length) { + try { + this.readers[index].message(message) + index++ + } catch { + this.readers.splice(index, 1) + } + } + } + + /** + * Suppresses emission for a display-only re-read: the trailing unterminated + * line is shown in the list but is re-read once complete, so emitting it here + * would hand every consumer the same line twice. `fn` must be synchronous. + */ + mute(fn: () => T): T { + const previous = this.muted + this.muted = true + try { + return fn() + } finally { + this.muted = previous + } + } + + finishRead(outcome: TranscriptReadOutcome): void { + const readers = this.readers + this.readers = [] + this.muted = false + for (const reader of readers) { + try { + reader.finish(outcome) + } catch { + // A consumer failure must never fail the session list. + } + } + } +} diff --git a/src/main/ai-vault/session-transcript-consumers.test.ts b/src/main/ai-vault/session-transcript-consumers.test.ts new file mode 100644 index 00000000000..cf9a6068673 --- /dev/null +++ b/src/main/ai-vault/session-transcript-consumers.test.ts @@ -0,0 +1,330 @@ +import { appendFile, mkdir, mkdtemp, rm, stat, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, expect, it, vi } from 'vitest' +import { scanAiVaultSessions } from './session-scanner' +import { + parseAgentSessionFileCached, + resetSessionParseCacheForTests +} from './session-scanner-parse-cache' +import { isolatedScanRoots, jsonLines } from './session-scanner-test-fixtures' +import type { FileWithMtime, SessionFileCandidate } from './session-scanner-types' +import { readWholeTranscript } from './session-transcript-reader' + +const OPENCODE_SQLITE_SESSION = { + id: 'local:opencode:sqlite-session:db', + agent: 'opencode' as const, + sessionId: 'sqlite-session' +} + +// Stands in for the worker thread: the point is that its messages never come +// back over the channel, not what the SQLite read returns. +vi.mock('./session-scanner-opencode-sqlite-worker-spawn', async (importOriginal) => ({ + ...(await importOriginal()), + parseOpenCodeSqliteSessionViaWorker: () => Promise.resolve(OPENCODE_SQLITE_SESSION) +})) +import type * as OpenCodeSqliteWorkerSpawn from './session-scanner-opencode-sqlite-worker-spawn' +import { + registerTranscriptConsumer, + resetTranscriptConsumersForTests, + type TranscriptMessage, + type TranscriptReadOutcome, + type TranscriptReadStart +} from './session-transcript-consumers' + +type RecordedRead = { + start: TranscriptReadStart + messages: TranscriptMessage[] + outcome: TranscriptReadOutcome | null +} + +function recordingConsumer(): { reads: RecordedRead[]; unregister: () => void } { + const reads: RecordedRead[] = [] + const unregister = registerTranscriptConsumer({ + beginRead: (start) => { + const read: RecordedRead = { start, messages: [], outcome: null } + reads.push(read) + return { + message: (message) => read.messages.push(message), + finish: (outcome) => { + read.outcome = outcome + } + } + } + }) + return { reads, unregister } +} + +function textsFor(reads: RecordedRead[], agent: string): string[] { + return reads + .filter((read) => read.start.candidate.agent === agent) + .flatMap((read) => read.messages.map((message) => `${message.role}:${message.text}`)) +} + +let tempRoots: string[] = [] + +afterEach(async () => { + resetTranscriptConsumersForTests() + resetSessionParseCacheForTests() + await Promise.all(tempRoots.map((root) => rm(root, { recursive: true, force: true }))) + tempRoots = [] +}) + +function claudeTurns(from: number, to: number): unknown[] { + const records: unknown[] = [] + for (let index = from; index <= to; index++) { + records.push({ + type: 'user', + sessionId: 'claude-session', + timestamp: `2026-05-01T10:0${index}:00.000Z`, + cwd: '/tmp/claude', + message: { role: 'user', content: `ask ${index}` } + }) + records.push({ + type: 'assistant', + sessionId: 'claude-session', + timestamp: `2026-05-01T10:0${index}:01.000Z`, + message: { + role: 'assistant', + content: [ + { type: 'text', text: `reply ${index}` }, + { type: 'tool_use', name: 'Bash', input: { command: `ls ${index}` } } + ] + } + }) + } + return records +} + +async function writeClaudeFixture(): Promise<{ + root: string + roots: ReturnType + transcript: string +}> { + const root = await mkdtemp(join(tmpdir(), 'orca-transcript-consumers-')) + tempRoots.push(root) + const roots = isolatedScanRoots(root) + const transcript = join(roots.claudeProjectsDir, 'project', 'claude-session.jsonl') + await mkdir(join(roots.claudeProjectsDir, 'project'), { recursive: true }) + await writeFile(transcript, `${jsonLines(claudeTurns(1, 4))}\n`) + return { root, roots, transcript } +} + +it('delivers one message stream to every registered consumer', async () => { + const { roots } = await writeClaudeFixture() + const first = recordingConsumer() + const second = recordingConsumer() + + const result = await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + + expect(result.issues).toEqual([]) + const stream = textsFor(first.reads, 'claude') + expect(stream).toEqual(textsFor(second.reads, 'claude')) + expect(stream).toEqual([ + 'user:ask 1', + 'assistant:reply 1', + 'tool:Bash: ls 1', + 'user:ask 2', + 'assistant:reply 2', + 'tool:Bash: ls 2', + 'user:ask 3', + 'assistant:reply 3', + 'tool:Bash: ls 3', + 'user:ask 4', + 'assistant:reply 4', + 'tool:Bash: ls 4' + ]) + // The list's own fold keeps only the newest five preview turns, so the stream + // is demonstrably the reader's, not a projection of the session row. + const session = result.sessions.find((entry) => entry.agent === 'claude') + expect(session?.previewMessages).toHaveLength(5) + expect(session?.messageCount).toBe(8) +}) + +it('leaves the session list identical whether or not a consumer is registered', async () => { + const withoutConsumer = await writeClaudeFixture() + const bare = await scanAiVaultSessions({ + ...withoutConsumer.roots, + platform: 'darwin', + limit: 20 + }) + + resetSessionParseCacheForTests() + recordingConsumer() + const observed = await scanAiVaultSessions({ + ...withoutConsumer.roots, + platform: 'darwin', + limit: 20 + }) + + expect(observed.sessions).toEqual(bare.sessions) +}) + +it('replays only the appended lines on a resumed read', async () => { + const { roots, transcript } = await writeClaudeFixture() + const consumer = recordingConsumer() + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + const firstRead = consumer.reads.at(-1) + expect(firstRead?.start.mode).toBe('replace') + expect(firstRead?.start.previousByteOffset).toBe(0) + expect(firstRead?.outcome?.incomplete).toBe(false) + + await appendFile(transcript, `${jsonLines(claudeTurns(5, 5))}\n`) + consumer.reads.length = 0 + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + + const resumed = consumer.reads.find((read) => read.start.candidate.agent === 'claude') + expect(resumed?.start.mode).toBe('append') + expect(resumed?.start.previousByteOffset).toBe(firstRead?.outcome?.byteOffset) + expect(textsFor(consumer.reads, 'claude')).toEqual([ + 'user:ask 5', + 'assistant:reply 5', + 'tool:Bash: ls 5' + ]) +}) + +it('publishes a trailing unterminated line once, when it is complete', async () => { + const { roots, transcript } = await writeClaudeFixture() + const consumer = recordingConsumer() + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + + // A half-written record: the list shows it, the stream must not carry it yet. + const [partial] = claudeTurns(5, 5) + await appendFile(transcript, JSON.stringify(partial)) + consumer.reads.length = 0 + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + expect(textsFor(consumer.reads, 'claude')).toEqual([]) + + await appendFile(transcript, '\n') + consumer.reads.length = 0 + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + expect(textsFor(consumer.reads, 'claude')).toEqual(['user:ask 5']) +}) + +it('keeps the session list working when a consumer throws', async () => { + const { roots } = await writeClaudeFixture() + registerTranscriptConsumer({ + beginRead: () => ({ + message: () => { + throw new Error('consumer exploded') + }, + finish: () => undefined + }) + }) + const healthy = recordingConsumer() + + const result = await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + + expect(result.issues).toEqual([]) + expect(result.sessions.find((entry) => entry.agent === 'claude')?.messageCount).toBe(8) + expect(textsFor(healthy.reads, 'claude')).toHaveLength(12) +}) + +it('skips a read a consumer declines without disturbing the others', async () => { + const { roots } = await writeClaudeFixture() + registerTranscriptConsumer({ beginRead: () => null }) + const healthy = recordingConsumer() + + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + + expect(textsFor(healthy.reads, 'claude')).toHaveLength(12) +}) + +async function claudeCandidate(transcript: string): Promise { + const stats = await stat(transcript) + const file: FileWithMtime = { + path: transcript, + mtimeMs: stats.mtimeMs, + modifiedAt: stats.mtime.toISOString(), + sizeBytes: stats.size + } + return { agent: 'claude', file, codexHome: null } +} + +it('serializes overlapping parses of one path so no consumer read is orphaned', async () => { + const { transcript } = await writeClaudeFixture() + // Seed a resume point: the channel it stores is what concurrent reads share. + await parseAgentSessionFileCached(await claudeCandidate(transcript), 'darwin') + + await appendFile(transcript, `${jsonLines(claudeTurns(5, 5))}\n`) + const consumer = recordingConsumer() + const appended = await claudeCandidate(transcript) + + const [first, second] = await Promise.all([ + parseAgentSessionFileCached(appended, 'darwin'), + parseAgentSessionFileCached(appended, 'darwin') + ]) + + // Every read that opened must also close, or its consumer keeps a half-read + // stream forever and never learns the outcome. + expect(consumer.reads.filter((read) => read.outcome === null)).toEqual([]) + expect(consumer.reads).toHaveLength(1) + expect(consumer.reads[0].start.mode).toBe('append') + expect(textsFor(consumer.reads, 'claude')).toEqual([ + 'user:ask 5', + 'assistant:reply 5', + 'tool:Bash: ls 5' + ]) + // The later caller reuses the stored entry rather than moving the cursor back. + expect(first?.messageCount).toBe(10) + expect(second?.messageCount).toBe(10) +}) + +it('reports a read whose parser cannot publish its messages as not complete', async () => { + const root = await mkdtemp(join(tmpdir(), 'orca-transcript-opencode-')) + tempRoots.push(root) + const dbPath = join(root, 'opencode.db') + await writeFile(dbPath, '') + const consumer = recordingConsumer() + + const session = await readWholeTranscript({ + candidate: { + agent: 'opencode', + codexHome: null, + file: { + path: `${dbPath}#sqlite-session`, + mtimeMs: 1, + modifiedAt: new Date(1).toISOString(), + sizeBytes: 10 + } + }, + platform: 'darwin' + }) + + expect(session).toEqual(OPENCODE_SQLITE_SESSION) + expect(consumer.reads).toHaveLength(1) + expect(consumer.reads[0].messages).toEqual([]) + expect(consumer.reads[0].outcome?.incomplete).toBe(true) +}) + +it('reports the transcript size, not the cache key, as a whole-file read offset', async () => { + const root = await mkdtemp(join(tmpdir(), 'orca-transcript-cline-')) + tempRoots.push(root) + const roots = isolatedScanRoots(root) + // Cline is whole-file and declares a sibling content dependency, so its cache + // key covers two files while the read covers one. + const sessionDir = join(roots.clineSessionsDir, 'cline-session') + await mkdir(sessionDir, { recursive: true }) + const metadataPath = join(sessionDir, 'cline-session.json') + await writeFile( + metadataPath, + JSON.stringify({ + session_id: 'cline-session', + started_at: '2026-05-01T10:00:00.000Z', + cwd: '/tmp/cline' + }) + ) + await writeFile( + join(sessionDir, 'cline-session.messages.json'), + JSON.stringify({ + updated_at: '2026-05-01T10:00:01.000Z', + messages: [{ role: 'user', content: [{ type: 'text', text: 'x'.repeat(400) }] }] + }) + ) + const consumer = recordingConsumer() + + await scanAiVaultSessions({ ...roots, platform: 'darwin', limit: 20 }) + + const read = consumer.reads.find((entry) => entry.start.candidate.agent === 'cline') + expect(read?.outcome?.byteOffset).toBe((await stat(metadataPath)).size) +}) diff --git a/src/main/ai-vault/session-transcript-consumers.ts b/src/main/ai-vault/session-transcript-consumers.ts new file mode 100644 index 00000000000..6707b298586 --- /dev/null +++ b/src/main/ai-vault/session-transcript-consumers.ts @@ -0,0 +1,85 @@ +import type { AiVaultSession } from '../../shared/ai-vault-types' +import type { SessionFileCandidate } from './session-scanner-types' + +// Why: the transcript reader owns discovery, per-file cursors and decoding; a +// consumer only folds the message stream. Registering a second consumer (a +// search index, a digest) must not require touching the reader or the parse +// cache, so the reader publishes reads rather than knowing who reads them. + +export type TranscriptMessageRole = 'user' | 'assistant' | 'tool' + +export type TranscriptMessage = { + role: TranscriptMessageRole + /** Untruncated decoded text; caps and redaction are consumer policy. */ + text: string + timestamp: string | null +} + +/** Where a parser hands its decoded messages; the reader supplies the instance. */ +export type TranscriptMessageSink = { + /** False when nobody is listening: parsers skip the extraction entirely. */ + readonly active: boolean + push(message: TranscriptMessage): void +} + +export const NO_TRANSCRIPT_MESSAGES: TranscriptMessageSink = { + active: false, + push: () => undefined +} + +export type TranscriptReadStart = { + candidate: SessionFileCandidate + /** `replace`: the whole file is being re-read; `append`: a resumed read. */ + mode: 'replace' | 'append' + /** Byte offset the messages of this read continue from. */ + previousByteOffset: number +} + +export type TranscriptReadOutcome = { + /** Null when the parser rejected the file (an excluded Codex worker transcript). */ + session: AiVaultSession | null + /** Byte offset just past the last complete line this read consumed. */ + byteOffset: number + /** + * The messages of this read are not the whole span: the read failed part way, + * or the parser decodes where the channel cannot reach it. A consumer must + * not record a cursor for an incomplete read. + */ + incomplete: boolean +} + +/** One consumer's view of one file read. */ +export type TranscriptReadConsumer = { + message(message: TranscriptMessage): void + finish(outcome: TranscriptReadOutcome): void +} + +export type TranscriptConsumer = { + /** + * Open this read, or return null to ignore it. A consumer whose own cursor is + * behind `previousByteOffset` declines here and re-reads on its own schedule; + * it must never ask another consumer where it is. + */ + beginRead(start: TranscriptReadStart): TranscriptReadConsumer | null +} + +const consumers = new Set() + +export function registerTranscriptConsumer(consumer: TranscriptConsumer): () => void { + consumers.add(consumer) + return () => { + consumers.delete(consumer) + } +} + +export function transcriptConsumers(): readonly TranscriptConsumer[] { + return [...consumers] +} + +export function hasTranscriptConsumers(): boolean { + return consumers.size > 0 +} + +export function resetTranscriptConsumersForTests(): void { + consumers.clear() +} diff --git a/src/main/ai-vault/session-transcript-message-content.test.ts b/src/main/ai-vault/session-transcript-message-content.test.ts new file mode 100644 index 00000000000..fa8c3934409 --- /dev/null +++ b/src/main/ai-vault/session-transcript-message-content.test.ts @@ -0,0 +1,61 @@ +import { expect, it } from 'vitest' +import { transcriptMessagesFromContent } from './session-transcript-message-content' + +const AT = '2026-05-01T10:00:00.000Z' + +it('keeps a plain string turn under the record role', () => { + expect(transcriptMessagesFromContent('user', 'just words', AT)).toEqual([ + { role: 'user', text: 'just words', timestamp: AT } + ]) +}) + +it('drops turns whose role a consumer cannot use', () => { + expect(transcriptMessagesFromContent('system', 'boot', AT)).toEqual([]) + expect(transcriptMessagesFromContent('unknown', 'noise', AT)).toEqual([]) +}) + +it('joins text blocks and appends tool blocks as their own messages', () => { + expect( + transcriptMessagesFromContent( + 'assistant', + [ + { type: 'text', text: 'first' }, + { type: 'tool_use', name: 'Bash', input: { command: 'ls -la', description: 'ignored' } }, + { type: 'thinking', text: 'second' }, + { type: 'image', source: {} } + ], + AT + ) + ).toEqual([ + { role: 'assistant', text: 'first\nsecond', timestamp: AT }, + { role: 'tool', text: 'Bash: ls -la', timestamp: AT } + ]) +}) + +it('reads a tool result carried on a user record as a tool message', () => { + expect( + transcriptMessagesFromContent( + 'user', + [{ type: 'tool_result', content: [{ type: 'text', text: 'exit 0' }] }], + AT + ) + ).toEqual([{ role: 'tool', text: 'exit 0', timestamp: AT }]) +}) + +it('names a tool call even with no recognisable argument', () => { + expect( + transcriptMessagesFromContent('assistant', [{ type: 'tool_use', name: 'Read', input: {} }], AT) + ).toEqual([{ role: 'tool', text: 'Read', timestamp: AT }]) +}) + +it('emits nothing for blank or absent content', () => { + expect(transcriptMessagesFromContent('user', ' ', AT)).toEqual([]) + expect(transcriptMessagesFromContent('user', null, AT)).toEqual([]) + expect(transcriptMessagesFromContent('assistant', [{ type: 'tool_use' }], AT)).toEqual([]) +}) + +it('does not apply the list preview cap', () => { + const long = 'x'.repeat(5000) + const [message] = transcriptMessagesFromContent('user', [{ type: 'text', text: long }], AT) + expect(message.text).toHaveLength(5000) +}) diff --git a/src/main/ai-vault/session-transcript-message-content.ts b/src/main/ai-vault/session-transcript-message-content.ts new file mode 100644 index 00000000000..2dda8493c31 --- /dev/null +++ b/src/main/ai-vault/session-transcript-message-content.ts @@ -0,0 +1,138 @@ +import { asRecord } from './session-scanner-record-value' +import { sliceAtCodeUnitLimit } from './session-scanner-text-normalization' +import type { AiVaultSessionPreviewMessage } from '../../shared/ai-vault-types' +import type { TranscriptMessage, TranscriptMessageRole } from './session-transcript-consumers' + +// Safety bound only: a consumer applies its own caps. Matches the first-prompt +// copy path's ceiling so one pathological paste cannot dominate a scan. +const TRANSCRIPT_MESSAGE_TEXT_LIMIT = 256 * 1024 +const TOOL_ARGUMENT_SCAN_LIMIT = 2000 + +const TEXT_BLOCK_TYPES = new Set(['text', 'input_text', 'output_text', 'thinking', 'reasoning']) +// The argument that identifies what a tool call actually did. +const TOOL_INPUT_KEYS = ['command', 'cmd', 'file_path', 'path', 'pattern', 'query', 'description'] + +type PreviewRole = AiVaultSessionPreviewMessage['role'] + +/** Only conversational roles reach consumers; system/unknown turns are noise. */ +export function transcriptMessageRole(role: PreviewRole): TranscriptMessageRole | null { + return role === 'user' || role === 'assistant' || role === 'tool' ? role : null +} + +export function toolCallText(name: unknown, input: unknown): string | null { + const toolName = typeof name === 'string' && name.trim() ? name.trim() : null + const inputRecord = asRecord(input) + let argument: string | null = null + if (inputRecord) { + for (const key of TOOL_INPUT_KEYS) { + const value = inputRecord[key] + if (typeof value === 'string' && value.trim()) { + argument = value + break + } + } + } else if (typeof input === 'string' && input.trim()) { + argument = input + } + if (!toolName && !argument) { + return null + } + const bounded = argument ? sliceAtCodeUnitLimit(argument, TOOL_ARGUMENT_SCAN_LIMIT) : null + return toolName && bounded ? `${toolName}: ${bounded}` : (toolName ?? bounded) +} + +/** Flattens a tool_result body (a string, or an array of text blocks). */ +function toolResultText(content: unknown): string | null { + if (typeof content === 'string') { + return content.trim() ? content : null + } + if (!Array.isArray(content)) { + return null + } + const parts: string[] = [] + let length = 0 + for (const item of content) { + const text = typeof item === 'string' ? item : asRecord(item)?.text + if (typeof text === 'string' && text) { + parts.push(text) + length += text.length + if (length >= TRANSCRIPT_MESSAGE_TEXT_LIMIT) { + break + } + } + } + const joined = parts.join('\n') + return joined.trim() ? joined : null +} + +/** + * Splits one provider content value into the messages it decodes to. Text + * blocks keep the record's role; tool_use and tool_result blocks become `tool` + * messages whichever record carried them (Claude stores tool results on user + * records), so a consumer never has to know a provider's record shapes. + */ +export function transcriptMessagesFromContent( + role: PreviewRole, + content: unknown, + timestamp: string | null +): TranscriptMessage[] { + const messages: TranscriptMessage[] = [] + const textRole = transcriptMessageRole(role) + if (typeof content === 'string') { + const text = boundedText(content) + return text && textRole ? [{ role: textRole, text, timestamp }] : [] + } + const blocks = Array.isArray(content) ? content : content != null ? [content] : [] + const textParts: string[] = [] + for (const block of blocks) { + if (typeof block === 'string') { + textParts.push(block) + continue + } + const item = asRecord(block) + if (!item) { + continue + } + const type = typeof item.type === 'string' ? item.type : null + if (type === 'tool_use') { + pushMessage(messages, 'tool', toolCallText(item.name, item.input), timestamp) + continue + } + if (type === 'tool_result') { + pushMessage(messages, 'tool', toolResultText(item.content), timestamp) + continue + } + if (type !== null && !TEXT_BLOCK_TYPES.has(type)) { + continue + } + const text = typeof item.text === 'string' ? item.text : item.content + if (typeof text === 'string' && text) { + textParts.push(text) + } + } + if (textRole && textParts.length > 0) { + // The record's own words lead; its tool blocks follow in transcript order. + const text = boundedText(textParts.join('\n')) + if (text) { + messages.unshift({ role: textRole, text, timestamp }) + } + } + return messages +} + +function pushMessage( + messages: TranscriptMessage[], + role: TranscriptMessageRole, + text: string | null, + timestamp: string | null +): void { + const bounded = text === null ? null : boundedText(text) + if (bounded) { + messages.push({ role, text: bounded, timestamp }) + } +} + +export function boundedText(value: string): string | null { + const bounded = sliceAtCodeUnitLimit(value, TRANSCRIPT_MESSAGE_TEXT_LIMIT) + return bounded.trim() ? bounded : null +} diff --git a/src/main/ai-vault/session-transcript-reader.ts b/src/main/ai-vault/session-transcript-reader.ts new file mode 100644 index 00000000000..228b0c832a3 --- /dev/null +++ b/src/main/ai-vault/session-transcript-reader.ts @@ -0,0 +1,150 @@ +import { readTranscriptSlice } from '../native-chat/wsl-transcript-fs-access' +import type { AiVaultSession } from '../../shared/ai-vault-types' +import { parseAgentSessionFile, parserPublishesMessages } from './session-scanner-agent-parser' +import { consumeCompleteJsonlLines } from './session-scanner-jsonl-reader' +import type { ResumableSessionParseState, SessionFileCandidate } from './session-scanner-types' +import type { SessionParseResumePoint } from './session-parse-cache-store' +import { TranscriptMessageChannel } from './session-transcript-channel' + +const NEWLINE_BYTE = 0x0a + +// Why: this layer owns reading a transcript and nothing else. It decides where +// a read starts, drives the parser, publishes the decoded messages to every +// registered consumer, and reports where the read ended. Which of those results +// are cached, listed or indexed belongs to the callers. + +export type TranscriptReadStats = { + incremental: number + fullParses: number + // Transcripts the parser already excluded (Codex workers), re-listed after a + // write and dismissed without reading. Counted apart from `incremental` so a + // scan span still shows how much work the early stop actually removed. + earlyStopped: number + bytesRead: number +} + +export type ResumableTranscriptRead = { + session: AiVaultSession | null + /** The fold to resume from next time, and the channel bound to it. */ + resume: SessionParseResumePoint +} + +/** + * Read an append-only transcript, resuming from `resume` when the file only + * grew and the recorded offset still sits on a line boundary. Anything else + * (a rewrite, a truncation, a platform change) re-reads the whole file. + */ +export async function readResumableTranscript(args: { + candidate: SessionFileCandidate + platform: NodeJS.Platform + resume: SessionParseResumePoint | null + stateFactory: (messages: TranscriptMessageChannel) => ResumableSessionParseState + stats?: TranscriptReadStats +}): Promise { + const { file } = args.candidate + const resume = args.resume + const canResume = + resume !== null && + typeof file.sizeBytes === 'number' && + file.sizeBytes >= resume.byteOffset && + (resume.byteOffset === 0 || (await endsWithNewlineAt(file.path, resume.byteOffset))) + + // Clone before consuming: a failed read must not corrupt the cached state, + // or the next resume would double-count the lines applied before the error. + const channel = canResume ? resume.channel : new TranscriptMessageChannel() + const state = canResume ? resume.state.clone() : args.stateFactory(channel) + const startOffset = canResume ? resume.byteOffset : 0 + // Mirrors the reader's entry guard so a dismissed transcript is not reported + // as an incremental parse that read nothing. + const stoppedBeforeRead = state.shouldStop?.() === true + if (args.stats) { + if (stoppedBeforeRead) { + args.stats.earlyStopped++ + } else if (canResume) { + args.stats.incremental++ + } else { + args.stats.fullParses++ + } + } + + channel.beginRead({ + candidate: args.candidate, + mode: canResume ? 'append' : 'replace', + previousByteOffset: startOffset + }) + try { + const readResult = await consumeCompleteJsonlLines({ + path: file.path, + start: startOffset, + onLine: (line) => state.consumeLine(line), + // Bound: the optional hooks are declared as methods, so a parser written + // with method syntax must not lose `this` on the way into the reader. + onLineBytes: state.consumeLineBytes?.bind(state), + shouldStop: state.shouldStop?.bind(state) + }) + if (args.stats) { + args.stats.bytesRead += readResult.bytesRead + } + + // The stat this scan displays is current even when nothing new was consumed. + state.touchFile(file) + + // Keep parity with the one-shot parser: a final unterminated line is shown, + // but stays out of the resumable state so the (possibly still-growing) line + // is re-read once complete instead of being half-counted. + let displayState = state + if (readResult.trailingPartialLine !== null) { + const partialLine = readResult.trailingPartialLine + displayState = state.clone() + channel.mute(() => displayState.consumeLine(partialLine)) + } + + const session = await displayState.finalize(args.platform) + channel.finishRead({ session, byteOffset: readResult.consumedThrough, incomplete: false }) + return { + session, + resume: { state, byteOffset: readResult.consumedThrough, channel } + } + } catch (error) { + channel.finishRead({ session: null, byteOffset: startOffset, incomplete: true }) + throw error + } +} + +/** + * Read a transcript whose format is rewritten in place rather than appended + * (whole-JSON documents, Kimi's state doc, OpenCode). There is no cursor to + * keep, so every read is a whole-file `replace`. + */ +export async function readWholeTranscript(args: { + candidate: SessionFileCandidate + platform: NodeJS.Platform + stats?: TranscriptReadStats +}): Promise { + const { file } = args.candidate + if (args.stats) { + args.stats.fullParses++ + args.stats.bytesRead += file.sizeBytes ?? 0 + } + const publishes = parserPublishesMessages(args.candidate) + const channel = new TranscriptMessageChannel() + channel.beginRead({ candidate: args.candidate, mode: 'replace', previousByteOffset: 0 }) + try { + const session = await parseAgentSessionFile(args.candidate, args.platform, channel) + channel.finishRead({ session, byteOffset: file.sizeBytes ?? 0, incomplete: !publishes }) + return session + } catch (error) { + channel.finishRead({ session: null, byteOffset: 0, incomplete: true }) + throw error + } +} + +// A resume point is only valid if it still sits just past a line break; +// anything else means the file was rewritten, not appended. Heuristic: a +// grown rewrite keeping '\n' at exactly this byte would slip through, but +// agent transcripts are append-only so that trade is accepted (worst case is +// a stale vault row until the file is next truncated or the app restarts). +async function endsWithNewlineAt(path: string, offset: number): Promise { + const slice = await readTranscriptSlice(path, offset - 1, 1, 'scan') + return slice.length === 1 && slice[0] === NEWLINE_BYTE +} diff --git a/src/main/lazy-worker-thread-host.ts b/src/main/lazy-worker-thread-host.ts new file mode 100644 index 00000000000..01f15eb1c66 --- /dev/null +++ b/src/main/lazy-worker-thread-host.ts @@ -0,0 +1,102 @@ +import type { Worker } from 'node:worker_threads' + +export type WorkerThreadFactory = () => Worker + +/** + * Owns the lifetime of one lazily-spawned worker thread: spawn on demand, + * listener wiring, teardown, and idle expiry. It holds no request state, so + * every decision about which call a message belongs to stays with its client, + * and a failed spawn is reported rather than thrown so the client can fail its + * queued calls closed instead of moving the work back onto the main thread. + */ +export class LazyWorkerThreadHost { + private worker: Worker | null = null + private idleTimer: NodeJS.Timeout | null = null + private cleanupListeners: (() => void) | null = null + private reportedUnavailable = false + + constructor( + private readonly options: { + factory: WorkerThreadFactory + idleTeardownMs: number + onMessage: (response: TResponse) => void + onError: (error: Error) => void + onExit: (code: number) => void + /** Nothing active and nothing queued, checked again when the idle timer fires. */ + isIdle: () => boolean + /** First spawn failure only: a repeating one must not repeat the log. */ + onUnavailable: (error: unknown) => void + } + ) {} + + get current(): Worker | null { + return this.worker + } + + /** The live worker, spawning one if needed; null when no worker can be had. */ + ensure(): Worker | null { + if (this.worker) { + return this.worker + } + try { + const worker = this.options.factory() + const onMessage = (response: TResponse): void => this.options.onMessage(response) + const onError = (error: Error): void => this.options.onError(error) + const onExit = (code: number): void => this.options.onExit(code) + worker.on('message', onMessage) + worker.on('error', onError) + worker.on('exit', onExit) + this.cleanupListeners = () => { + worker.off('message', onMessage) + worker.off('error', onError) + worker.off('exit', onExit) + } + // Never keep the app alive for background work. + worker.unref?.() + this.worker = worker + return worker + } catch (err) { + if (!this.reportedUnavailable) { + this.reportedUnavailable = true + this.options.onUnavailable(err) + } + return null + } + } + + destroy(): void { + this.clearIdleTimer() + const worker = this.worker + this.worker = null + if (!worker) { + return + } + this.cleanupListeners?.() + this.cleanupListeners = null + worker.removeAllListeners() + void worker.terminate().catch(() => undefined) + } + + scheduleIdleTeardown(): void { + this.clearIdleTimer() + if (!this.worker) { + return + } + this.idleTimer = setTimeout(() => { + this.idleTimer = null + // Re-checked here: a request arriving as the timer fires must never be + // lost to a self-exiting worker. + if (this.options.isIdle()) { + this.destroy() + } + }, this.options.idleTeardownMs) + this.idleTimer.unref?.() + } + + clearIdleTimer(): void { + if (this.idleTimer) { + clearTimeout(this.idleTimer) + this.idleTimer = null + } + } +} diff --git a/src/main/ports/port-scan-command-client.ts b/src/main/ports/port-scan-command-client.ts index 9c01b4dccda..27232ee8833 100644 --- a/src/main/ports/port-scan-command-client.ts +++ b/src/main/ports/port-scan-command-client.ts @@ -2,6 +2,7 @@ import { existsSync } from 'node:fs' import { getAppEnvironment, hasAppEnvironment } from '../../shared/app-environment' import { join } from 'node:path' import { Worker } from 'node:worker_threads' +import { LazyWorkerThreadHost, type WorkerThreadFactory } from '../lazy-worker-thread-host' import { PORT_SCAN_COMMAND_TIMEOUT_MS, PortScanCommandTimeoutError, @@ -11,11 +12,10 @@ import { // Why (#11161): a lazily-spawned, unref'd worker runs the port scan's probe // spawns off the Electron main-process event loop, because libuv performs -// process creation inline on the calling thread. Lifecycle (FIFO one-at-a-time -// dispatch, per-call deadlines, respawn-on-fault, idle teardown, fail-closed) -// mirrors src/main/ai-vault/session-scanner-opencode-sqlite-worker-client.ts; -// the duplicated ~150 lines are cheaper than a premature shared abstraction, so -// a third adopter should extract one. +// process creation inline on the calling thread. This module owns the request +// half (FIFO one-at-a-time dispatch, per-call deadlines, respawn-on-fault); the +// thread's own lifetime belongs to LazyWorkerThreadHost, shared with +// src/main/ai-vault/session-scanner-opencode-sqlite-worker-client.ts. // // This module used to contain the literal text require('electron'), which fails the // plain-Node entry guard even inside a try/catch. It reads the AppEnvironment port @@ -36,7 +36,7 @@ export const MAX_CONSECUTIVE_DEATHS = 3 export const MAX_QUEUED_CALLS = 8 export type PortScanCommandResult = { stdout: string; spawnMs: number } -export type PortScanWorkerFactory = () => Worker +export type PortScanWorkerFactory = WorkerThreadFactory // Distinguishes "no worker at all" from a timeout or crash so the scanner can // log it once and callers never mistake it for a command timeout. @@ -62,20 +62,27 @@ type PendingCall = { * be spawned rather than moving process creation back onto the main thread. */ export class PortScanCommandClient { - private worker: Worker | null = null private active: PendingCall | null = null private queue: PendingCall[] = [] - private idleTimer: NodeJS.Timeout | null = null private consecutiveDeaths = 0 private nextId = 1 - private loggedWorkerUnavailable = false - private cleanupWorkerListeners: (() => void) | null = null - private readonly workerFactory: PortScanWorkerFactory - private readonly log: (message: string) => void + private readonly host: LazyWorkerThreadHost constructor(options: { workerFactory: PortScanWorkerFactory; log?: (message: string) => void }) { - this.workerFactory = options.workerFactory - this.log = options.log ?? ((message) => console.warn(message)) + const log = options.log ?? ((message: string) => console.warn(message)) + this.host = new LazyWorkerThreadHost({ + factory: options.workerFactory, + idleTeardownMs: IDLE_TEARDOWN_MS, + onMessage: (response) => this.onMessage(response), + onError: (error) => this.onWorkerFault(error), + onExit: (code) => this.onWorkerExit(code), + isIdle: () => !this.active && this.queue.length === 0, + // Why (#11161): never fall back to in-process execFile here; a missing + // bundle must report port scanning as unavailable rather than reintroduce + // the main-thread freeze this worker boundary exists to prevent. + onUnavailable: (err) => + log(`[workspace-ports] probe worker unavailable. ${errorMessage(err)}`) + }) } /** @@ -109,7 +116,7 @@ export class PortScanCommandClient { if (this.active || this.queue.length === 0) { return } - const worker = this.ensureWorker() + const worker = this.host.ensure() if (!worker) { this.failQueuedAsUnavailable() return @@ -119,7 +126,7 @@ export class PortScanCommandClient { return } this.active = call - this.clearIdleTimer() + this.host.clearIdleTimer() // Why (#11161): one at a time. uv_spawn blocks the worker's own loop, so a // second concurrent request would have its deadline armed while the first // spawn is still stalling the thread, producing a false timeout. @@ -128,39 +135,6 @@ export class PortScanCommandClient { worker.postMessage(call.request) } - private ensureWorker(): Worker | null { - if (this.worker) { - return this.worker - } - try { - const worker = this.workerFactory() - const onMessage = (response: PortScanCommandResponse): void => this.onMessage(response) - const onError = (error: Error): void => this.onWorkerFault(error) - const onExit = (code: number): void => this.onWorkerExit(code) - worker.on('message', onMessage) - worker.on('error', onError) - worker.on('exit', onExit) - this.cleanupWorkerListeners = () => { - worker.off('message', onMessage) - worker.off('error', onError) - worker.off('exit', onExit) - } - // Never keep the app alive for a port scan. - worker.unref?.() - this.worker = worker - return worker - } catch (err) { - // Why (#11161): never fall back to in-process execFile here; a missing - // bundle must report port scanning as unavailable rather than reintroduce - // the main-thread freeze this worker boundary exists to prevent. - if (!this.loggedWorkerUnavailable) { - this.loggedWorkerUnavailable = true - this.log(`[workspace-ports] probe worker unavailable. ${errorMessage(err)}`) - } - return null - } - } - private onMessage(response: PortScanCommandResponse): void { const call = this.active if (!call || call.request.id !== response.id) { @@ -191,7 +165,7 @@ export class PortScanCommandClient { // A clean self-exit is not a death, but the stale handle must be dropped or // the next dispatch would post into a dead worker and stall to its deadline. if (code === 0 && !this.active && this.queue.length === 0) { - this.destroyWorker() + this.host.destroy() return } this.onWorkerFault(new Error(`Port scan probe worker exited with code ${code}`)) @@ -199,7 +173,7 @@ export class PortScanCommandClient { private onWorkerFault(error: Error): void { const failed = this.active - this.destroyWorker() + this.host.destroy() this.consecutiveDeaths++ if (failed) { this.settle(failed, () => failed.reject(error)) @@ -248,50 +222,11 @@ export class PortScanCommandClient { if (this.queue.length > 0) { this.pump() } else { - this.scheduleIdleTeardown() + // Terminating can orphan a probe child mid-spawn; the worker reaps what it + // can on exit, and every probe here is short-lived. + this.host.scheduleIdleTeardown() } } - - private scheduleIdleTeardown(): void { - this.clearIdleTimer() - if (!this.worker) { - return - } - this.idleTimer = setTimeout(() => this.teardownIfIdle(), IDLE_TEARDOWN_MS) - this.idleTimer.unref?.() - } - - private teardownIfIdle(): void { - this.idleTimer = null - // Only tear down with nothing active AND nothing queued: a request arriving - // as the timer fires must never be lost to a self-exiting worker. - if (this.active || this.queue.length > 0) { - return - } - this.destroyWorker() - } - - private clearIdleTimer(): void { - if (this.idleTimer) { - clearTimeout(this.idleTimer) - this.idleTimer = null - } - } - - private destroyWorker(): void { - this.clearIdleTimer() - const worker = this.worker - this.worker = null - if (!worker) { - return - } - this.cleanupWorkerListeners?.() - this.cleanupWorkerListeners = null - worker.removeAllListeners() - // Terminating can orphan a probe child mid-spawn; the worker reaps what it - // can on exit, and every probe here is short-lived. - void worker.terminate().catch(() => undefined) - } } function errorMessage(error: unknown): string {