mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
Jinwoo Hong
2c984bcdaa
* feat(ai-vault-search): add the session search index schema and row modules
The FTS5 index that PR 2 folds the transcript reader's message stream into:
two FTS tables behind visibility views, the identifier shadow column, the
resumable fork digest, redaction at the row-insert choke point, and the
bounded compaction, warm-up and retention lanes.
Schema version starts at 1: this branch drops the query log and the fts5vocab
table, which the query engine reintroduces with its own bump.
* feat(ai-vault-search): stage index writes and publish them atomically
The writer stages a read's rows against an unpublished session row and a batch
id, then flips the whole read visible in one transaction: both FTS tables are
written together per row, and publish nulls the batch pointer before dropping
the batch so a recycled rowid can never name a later in-flight write.
Buffering replaces the branch's streamed producer. The transcript reader pushes
messages synchronously, so a staged write cannot make it wait; rows are held to
128 of them or 256 KB and then flushed in one transaction, which bounds both the
retained bytes and one stall.
The store owns the database and the set of files the index is behind on. It
carries no query, coverage or scheduling: draining that set is the indexer's.
* feat(ai-vault-search): register the index as a transcript reader consumer
The index keeps its own per-file cursor in the `files` table and never consults
the parse cache; the branch's AsyncLocalStorage capture scope is gone.
Three refusals, each leaving the cursor where it was and recording the file for
a later whole re-read: an append whose predecessor offset is not this index's
own cursor (or whose dev/ino changed) is declined in `beginRead`, a staging
failure ends the read's rows without throwing back at the reader, and an
`incomplete` outcome never publishes. A null session drops the file's rows and
still advances the cursor, because the file was read through.
Nothing in production registers it: the indexer decides when the index is live.
* test(ai-vault-search): measure the index's disk and latency cost
The write benchmark indexes a synthetic corpus through the real reader and
reports rows/s, per-table bytes per transcript MB, write amplification and
rebuild time; the retention benchmark compares the batched purge against a
whole-file delete. The corpus generator ships with them, so the numbers are
reproducible on any host and no real transcript is ever read.
* fix(ai-vault-search): decline a source the message channel cannot reach
An OpenCode SQLite candidate decodes inside a worker, so the reader reports
every read of it as incomplete. Staging a batch first meant one staging session
and one tombstone per scan of every such session, forever. Declining in
`beginRead` also keeps it out of the re-read set, because no re-read helps: the
index cannot cover that source until its capture path lands.
* fix(ai-vault-search): redact a message before it is cut into rows
A credential is a shape, and a shape split across two chunks matches neither
half: an 8000-char boundary landing inside a PEM block or a JWT indexed the key
material in full, identifier shadow terms included. Redaction moves ahead of
chunking, and the row type is branded so only `searchMessageRows` can mint a row
an insert site accepts.
`upsertFile` stops erasing dev/ino when a candidate carries none. A host that
cannot stat identity (SSH, WSL, a degraded Windows stat) was overwriting a
proven identity with NULL, which made the next rename-replace of that path
undetectable.
* test(ai-vault-search): pin each index guard on its own
Three guards were each shadowed by another, so mutating any one of them left
every test green. Driving the store directly separates the writer's own
predecessor-offset check from the consumer's cursor check, and a stubbed store
proves `beginRead` refuses before the writer is ever asked.
`publishable` gets the case it never had: a second read of the same path
publishes first, and the stale stage is tombstoned rather than resurrected. It
covers the overlap the parse file lane normally prevents, so the invalidation
slot being per-path is not load-bearing on its own.
Two ratchets: every provider fingerprint needs a fixture that reaches past
`SECRET_ANCHOR`, and every FTS read in the index modules must subtract staged
rows through a visibility view.
* fix(ai-vault-search): rebuild an index that cannot be trusted, and index retention
Opening self-heals. A file too torn to open, or one whose recovery fails, is
unlinked with its sidecars and reopened once; a second failure throws. Before
this a torn index broke open permanently, even though the index is a cache over
the transcripts and throwing it away costs only a re-scan.
A `meta` table with no readable version row is now `stale`, not `fresh`. Seeding
the current version over it would have kept whatever rows the old schema left.
`fresh` means no `meta` table at all, and no longer triggers a rebuild: the
first open of a new profile was doing create-remove-create.
Recovery moves inside open, because retiring a batch that outlived its writer is
part of opening the index rather than of using it. Its append case gets the test
it never had: 200 rows staged onto a live session and abandoned leave the
published generation intact and nothing else.
Also an index on files(mtime_ms), so the retention pass seeks the expiring end
of the list instead of scanning and sorting it, a pragma read-back test, and a
sessions.file_path comment that says what is actually true of OpenCode.
* fix(ai-vault-search): key a session's cwd the way the sidebar already does
`cwd_key` had its own normalizer, which qualified a WSL cwd with its distro:
`/home/me/repo` was stored as `//wsl/ubuntu/home/me/repo` while the sidebar's
`folderGroupKey` stored `/home/me/repo`. Any later join between an indexed hit
and a sidebar group would have returned nothing for exactly the WSL users the
qualification was meant to help. The key is now the shared normalizer verbatim,
pinned against `folderGroupKey` for POSIX, Windows drive, /mnt/c, both WSL UNC
aliases, a Linux path, and the root.
The collision it guarded against is real: two distros both spell
`/home/me/repo`. It is also a collision every SSH host has, neither key
qualifies for SSH, and the honest fix is a column naming the execution host
rather than a path key that only some hosts spell differently.
`/` now keys as `/` instead of the empty string. An empty key cannot be told
apart from a session with no cwd, and the scope filter builds its child prefix
as `key + '/'`, which would have been `//`.
* test(ai-vault-search): widen the FTS visibility ratchet past whole literals
Concatenating or interpolating a table name hid it from the per-literal scan,
so `'SELECT rowid FROM ' + table` passed. The unit is now the file, the scan
covers src/main and src/relay rather than one directory, and both bypass shapes
are checker unit tests. Exemptions carry a reason and the census fails on one
that has stopped being needed, which is how the schema module lost its.
* fix(ai-vault-search): keep a failed rebuild's real cause, and read per statement
Closing the stale handle before the unlink left `db` dangling: if the unlink or
the reopen threw, the failure path closed it a second time and node:sqlite's
ERR_INVALID_STATE replaced the real cause. Nothing classifies that as worth
retrying, so an index a virus scanner or a second Orca was holding would never
rebuild. The handle is nulled while none is open.
The visibility ratchet moves from the file to the statement. A file is far too
coarse for what it exists to guard: the query module will name both views
somewhere, and that whitelisted every raw read in it. Statements come from
literals and their concatenation chains, split on `;`, and a table name
assembled at runtime counts as a read in any file that names an FTS table. The
roots now cover cli, shared and preload as well.
Half a recorded identity is now stated to be no identity. `remote-session-file-stat`
spreads dev and ino independently and the COALESCE preserves whichever half a
host could prove, so one number cannot tell a rename-replace from a same-file
re-read; comparing it would decline healthy resumes on a coincidence.
`discard` clearing the staging slot gets a test: without it the map grew one
entry per path for the store's life.
* fix(ai-vault-search): give a behind consumer a way to get the read it needs
`markStale` promised a whole re-read that nothing could deliver. The reader
picks append or replace from the session list's resume point, so with an empty
index and a warm parse cache every read arrives as `append`, the consumer
declines every one, and nothing is ever indexed. That is the state on first
enablement inside a running app.
`requestWholeTranscriptRead` in the reader drops that path's resume point, which
is the one lever that changes the next read's mode, and it lives with the cache
that owns it rather than with the consumer that wants it. `takeStale` documents
that its paths need it before a scan is re-dispatched.
Pausing no longer empties the re-read set. `acceptsCandidate` answers whether a
write may start now and was being used for both jobs, so reconfiguring retention
while paused dropped every entry and a declined read during a pause was never
recorded at all. Retention alone prunes; a paused decline is recorded, bounded,
with the oldest dropped and the drops counted, because a set that silently
forgets is worse than one that says it is incomplete.
A read that decodes no session now tombstones its staged rows before the store
schedules cleanup. The cleanup lane reads the tombstone table the moment it is
scheduled, so the old order left that batch on disk until some later write, and
for the last read before a shutdown that is never.
* fix(ai-vault-search): create the directory the index lives in
Nothing made `<userData>/ai-vault-search/`, and SQLite's failure for a missing
parent is `unable to open database file`, which is correctly not classified as
corruption — so the retry never fired and the feature stranded on any profile
that had never held an index.
* refactor(ai-vault-search): store transcript content as written
Decision: the index does not redact. A secret in a transcript is already
plaintext under the user's home directory and is treated as compromised, so the
index is a second copy of content the user already holds, not a new exposure.
The reference agent-session-search products do not redact either. What a snippet
may carry once it leaves this machine is a transport policy and belongs where
the wire is, not in the write path.
Removes the redaction module and its census, the branded row type that existed
to prove redaction had run before an insert, and the two chunk-boundary tests.
`insertSearchMessage` takes a plain chunk again and identifier shadow terms come
off the raw text. `src/main/observability/redactor.ts` goes back to what main
has, so this PR no longer touches it at all.
Chunking and the fork digest are unchanged; the digest always folded raw message
text, ahead of chunking, so its tests hold as written.
Cost: redaction was 14 ms per 10.5 MB of transcript, about 3% of a rebuild and
below the benchmark's run-to-run spread, so the write numbers are unchanged at
roughly 22-25k rows/s and 22-27 MB/s.
* fix(ai-vault-search): hide a tombstoned session's messages, not only its row
`visible_sessions` already subtracted session-keyed tombstones; the message
half filtered on the batch pointer alone. A published row outlives its session
row until the cleanup lane reaches it, so between a `replace` publish and that
drain both generations answered, and a removed file kept answering after its
session was gone.
Both views now subtract the same set, over a partial index so neither read
scans the tombstone table.
* fix(ai-vault-search): drain the rows a read that never published staged
Only `writePublished` and `removeFile` scheduled the cleanup lane. A read that
staged rows and then declined to publish them tombstoned its batch and told
nobody, so 256 rows of a 300-message incomplete read sat in `messages` and both
FTS tables until some unrelated write happened to schedule a pass. Open-time
recovery had the same hole: it wrote the tombstones and drained none.
The abandoned branch now schedules the drain the published branch already got,
and opening schedules one pass for what recovery just tombstoned. Recovery no
longer adds a batch tombstone when the session-keyed one already covers those
rows, which it did once more on every reopen.
* fix(ai-vault-search): continue the cursor of a file that decoded no session
A read that decodes no session — an excluded Codex worker transcript — advances
the cursor and leaves `files.session_row_id` null. Every later append was then
declined, so a file that only ever grows would be re-read whole on every pass
for the rest of its life.
An append now only has to continue this index's own cursor; it creates the
session row when there is none to resume. That also collapses the `append` flag,
which meant both "resume this session" and "do not own it", into the resumed row
id itself.
* refactor(ai-vault-search): drop the index path module nothing calls
No caller in this PR or the two that follow it: the composition root that would
capture the userData path is PR 3b's.
* refactor(ai-vault-search): read the schema version as stale or not
Only the stale answer was ever acted on; the fresh/current split named two ways
of being fine.
* fix(ai-vault-search): read the resumed session id off an optional row
* refactor(ai-vault-search): leave page warmup and the freshness check to PR 4
`warm()` and `session-search-page-warmup.ts` have their first caller in the
query engine, which is what knows which pages are worth warming; the module
itself went with the previous commit. `isSessionSearchFileCurrent` has its first
caller in the reconciler. `session-search-file-cursor.ts` stays, because
`fileIdentity` and both its types are load-bearing here.
* refactor(ai-vault-search): stop pruning the re-read set on a retention change
The prune walked the whole set to drop what the next `beginRead` would refuse
anyway: `acceptsCandidate` applies the window when the re-read is dispatched,
and `markStale` applies it again before recording anything. The bound stays.
* fix(ai-vault-search): keep a batch tombstone from outliving its batch row
Draining a session-keyed tombstone deletes the session's batch rows, but left
any batch-keyed tombstone naming them in place. `search_write_batches` empties
on every publish, so ids restart low and the next read takes the freed rowid;
the stale tombstone then deleted that live batch's staged rows and the session
published missing messages, with no re-read to fill the gap.
The session drain now clears those tombstones in the same transaction.
* refactor(ai-vault-search): commit a file's rows and its cursor in one transaction
The staged-publish model is replaced by one SQLite transaction per file in WAL
mode. A read buffers its decoded rows and writes them, its session and its
cursor together; a reader on another handle sees the last committed state, which
is the "never a torn session" guarantee the staging machinery was built to
provide. A crash rolls the whole file back and it is re-read.
Gone with it: `search_write_batches`, `search_pending_deletes`,
`messages.batch_id`, `sessions.index_ready`, both `visible_*` views, the
open-time recovery pass, the cleanup lane and `settled()`, and the ratchet test
that made every query site read through a view. Rounds 2 through 6 were all
seams between those pieces.
A file whose rows exceed `SESSION_SEARCH_COMMIT_CHARS` is cut into chunks. The
reader only hands out a byte offset when a read finishes, so a chunk records one
no append can continue from: its rows answer searches as a coherent prefix of
the session, and the next whole read replaces them.
Retention keeps no record of unfinished work. It cuts a session loose from its
file in one small transaction, which is what stops it answering, then reclaims
its rows in bounded batches and hands the freed pages back as it goes. Rows
whose session row is gone are the record of what an interrupted purge left.
Deleted for want of a caller in PR 2, 3 or 4: the WAL budget (a buffered write
has no staging window to grow one across), the compaction module (folded into
the drain), the `sessions_content_hash` index (fork folding runs in JS over rows
PR 4 already holds), `lastWriteAt`, `failures`, `SessionSearchStoreOptions`,
`openStageCount` and `chunkMessageText`.
* test(ai-vault-search): price a per-file commit and the ceiling that bounds it
The write benchmark now times every transaction, because with one per file that
is the whole stall a file costs. A second phase indexes a single synthetic
100 MB transcript, which is what the commit ceiling is chosen against.
* fix(ai-vault-search): stop a fenced write reopening a transaction per message
A chunk write that finds the file record moved under it — a `removeFile`, or an
overlapping read of the same path — can never land anything afterwards. It kept
buffering, so every remaining message re-entered `BEGIN IMMEDIATE` only to roll
back, once per message for the rest of a file that may be a hundred megabytes.
It now stops at the first refusal and drops what it holds.
* fix(ai-vault-search): never hand a live session the id of a purged one
`sessions.id` was a plain rowid alias, so SQLite reissued it as max+1. That id
names rows in `messages` for far longer than the row itself lives: retention
cuts a session loose in one transaction and reclaims its messages over many, and
a session created inside that window was handed a freed id and adopted whatever
of the purged conversation the drain had not reached. The drain then skipped
those rows for good, because their session row exists again, and a search
answered for a purged transcript under a live session's name.
Reachable with no crash at all: an append of a growing transcript the parser
decoded no session from creates its session row mid-purge. AUTOINCREMENT is the
class fix; the schema version goes to 3 so a file written by an earlier commit
of this branch rebuilds rather than keeping a plain-rowid table under a
`CREATE TABLE IF NOT EXISTS`. `messages.id` was checked and is not exposed to
the same window: a row and its two FTS entries always go in one transaction.
Two smaller ones in the same pass. `removeFile` did not fence a read of a path
this index had never written: `current()` compared a cursor, and on an unknown
path the absent row and the absent expectation are both undefined, so the write
recreated the source after its owner had proven it gone. The writer now counts
removals per path and a write compares that count, which is a positive fence
rather than an inferred one. And `drainOrphanedMessages`,
`CONTENT_HASH_MESSAGE_LIMIT` and `CONTENT_HASH_MIN_MESSAGES` are no longer
exported: nothing outside their own modules reads them.
* fix(ai-vault-search): report a half-written file as held, not as unknown
`indexedFile` returned null for a file a chunked read left partway through, the
same answer it gives for a path the index has never seen. A caller asking "do
you hold this file" therefore read a half-written one as new, asked for whatever
read the parse cache offered, and the reader picked append — which the consumer
then declined. Only the stale set healed it, a cycle later.
It now reports the record with a null `byteOffset`, and
`requiresWholeRead(indexed)` names that state. Null rather than an added flag
because the mtime and size on that record are the file's real ones: a freshness
check comparing only those would call a half-written file current, and a boolean
is easy to not read, while every site that does arithmetic on the offset has to
say what null means at compile time.
The test also found the writer accepting an append that passed the partial
sentinel back as its predecessor offset. Nothing in the reader produces a
negative offset, but the sentinel is not a byte offset and no caller should be
able to continue it; `beginWrite` now refuses it outright.
* fix(ai-vault-search): cut a chunk at whitespace, never mid-token
The 8,000-char chunker backed up only to a newline, and only when one sat in
the second half of the window. A wrapped paragraph, a CJK transcript or a
minified log has no newline there, so the cut landed inside whatever word
straddled the target: the user's term was filed as two halves and matched
neither. It now backs up to the last Unicode whitespace in the second half,
and falls through to the target when there is none, because 4,000 characters
without a space is not a word.
A phrase that straddles a chunk boundary is still not matched. Chunks are
separate FTS rows and FTS5 cannot span them; that is stated at the chunker.
* feat(ai-vault-search): cap an indexed tool row at 3,072 characters
Tool output is 80-97 % of a transcript's bytes and a single message may be a
quarter of a megabyte, so without a cap the index, the buffer a read holds and
the transaction it commits are all sized by how much a tool printed rather
than by how much is worth searching. A `tool` message now becomes one row of
at most 3,072 characters, kept from the head, where the command and the first
lines of its output are. User and assistant text is never capped.
Measured on the write benchmark, 40 sessions with tool output at 95 % of
message text (the real band), per transcript MB: index 1,334,126 -> 353,973
bytes, write amplification 1.27x -> 0.34x, rows 18,804 -> 9,600, rebuild
1,676 -> 379 ms, largest transaction 49.7 -> 12.1 ms. On the default corpus,
whose tool results are 1.4 KB and so under the cap, nothing changes at all:
2,167,887 bytes per transcript MB on both arms.
The corpus generator takes `toolResultWords` and the benchmark reads
ORCA_SEARCH_BENCH_TOOL_WORDS so both arms are the same harness.
* fix(ai-vault-search): check the commit ceiling per row, not per message
The buffered-chars check ran after a whole message had been folded into rows,
so a single message could carry a transaction as far past the ceiling as it
was large. A conversation turn is one message and can be megabytes; the
ceiling exists to bound how long one commit holds the process and how large a
WAL it produces, and neither bound survived a message that overshot it.
* style(ai-vault-search): undo a stray reformat of untouched assertions
Three assertions in the file-write test were expanded by a formatter run that
was not the repo's, and the expansion survived because a multi-line object
literal is preserved once it exists. Restored to what they were.
* feat(ai-vault-search): write a session's identity with its first commit
A chunked read created its session row with an empty session id, an empty
title, an empty resume command and null cwd and timestamps, and only filled
them in on the final commit. Those chunks answer searches the moment they
land, so until the read ended every hit they produced named a session nothing
could identify — and a crash between chunks left it that way for good, since
the re-read that heals it is the same read starting over.
The reader already knows the id, cwd and timestamps from a transcript's
opening lines; it just had nowhere to put them. `TranscriptReadStart` now
carries an optional `identity()` the consumer calls during the read, backed by
an optional `identity()` on `ResumableSessionParseState`: one line in the
shared accumulator fold (which covers cursor, copilot, droid, gemini,
antigravity and the graph parsers) and one each in the Claude and Codex folds,
which keep their own state. A chunk commit writes what it returns; the final
commit overwrites it from the decoded session, so the mid-read title stays
provisional.
Two `cwd` guards in the Codex fold collapse into the `?? ` form the `branch`
line beside one of them already used. `extractString` never returns an empty
string, so it is the same assignment in one line instead of four, and it is
what keeps the file under the 300-line cap with the identity accessor added.
* refactor(ai-vault-search): drop conversation_fts for a column filter
The second FTS table existed for query latency alone: a column-filtered
`messages_fts MATCH '{user_text assistant_text}: q'` returns the identical
rowid set, which PR 2's round 5 review proved and PR 4's benchmark re-checked
per query. PR 4 then measured the filter at 1.16-1.36x the p95 of the second
table on a 105 MB corpus across two points in the tool-output band, under the
2x bar the decision was set at, and closes the stack's open decision 3.
It cost a quarter of the index on a corpus whose tool output is half the
message text. With the tool-row cap it is a smaller saving than the decision
was framed around, but it is still a table, a write per conversational row and
a delete per reclaimed one.
Schema version 4: `CREATE TABLE IF NOT EXISTS` is a no-op over an existing
index, so only the bump makes a file that still carries the table go.
* feat(ai-vault-search): expose the index handle a composed reader queries
PR 4's engine reads the index this store owns. One getter lets it compose over
the store's handle instead of opening a second connection to the same file,
and it carries the two rules this PR measured: never hold a read transaction
across an await, and no `.iterate()` outliving its statement. Either pins a
read snapshot, and a checkpoint cannot pass one, so the WAL grows without
bound for as long as it is held (10 MB to 266 MB on the write benchmark).
* fix(ai-vault-search): cut a chunk at punctuation, not only whitespace
The 8,000-character chunker backed up to the last whitespace in the second
half of the window and fell through to the target when it found none. A
minified tool result has none to find: valid minified JSON runs past 8,000
characters without a space, so the cut landed inside whatever word straddled
the target, and a search for `pericardium` matched neither `perica` nor
`rdium`. That is the shape most likely to be chunked in the first place,
because tool output is 80-97 % of a transcript's bytes.
The backoff now takes any character that cannot be inside a token — anything
outside a letter, digit or underscore — and still falls through to the target
when the window holds none, because 4,000 characters without one is not a
word. Surrogates are excluded from the class so a cut never lands between the
halves of an astral character.
A few of the tokenizer's own `tokenchars` (`. - / +`) are cut on even though
unicode61 keeps them inside a token. That costs the joined form of a path, and
only in a window with no whitespace anywhere, which is a far smaller loss than
the torn word this exists to prevent.
* fix(ai-vault-search): never chunk a read that cannot name its session
`updateProvisionalSession` is fed by the resumable readers alone. Claude and
Codex carry an `identity()` off their fold; `readWholeTranscript` supplies
none, because a format rewritten in place has no resumable state to ask. So a
Grok, Cursor, Gemini or OpenCode file large enough to pass the commit ceiling
published its chunks under a session with an empty id, an empty title and a
null cwd — rows that answer searches at once and that an interrupted read
leaves behind for good, since the re-read that heals them is the same read
starting over.
`add` now commits a chunk only while the read can name what it is writing. A
read with no identity, or one whose parser has decoded no id yet, keeps
buffering and commits whole at `finish`. The whole-file formats are the ones
with nothing to give and they are small — the largest on this machine is 5 MB
— so buffering one to the end costs nothing, and chunking stays reserved for
the readers that can say which session a prefix belongs to.
`updateProvisionalSession` takes a non-null identity now, so the invariant is
the type rather than a guard that silently wrote nothing.
* perf(ai-vault-search): cut a replaced generation loose instead of deleting it
The first transaction of a replace deleted every row of the old session before
inserting its own bounded chunk, so the transaction was sized by the history
being replaced rather than by the rows being written. On the synthetic 100 MB
transcript the longest replace transaction was 1,255-1,283 ms against
668-672 ms for the same file read fresh, and the gap grows with the session.
A replace now mints a new session row, points the `files` row at it and
deletes the one old `sessions` row, all in the same transaction. Every
retrieval joins `sessions`, so the old generation stops answering the moment
that commits — the same thing retention's first transaction does — and its
messages are reclaimed afterwards by `drainOrphanedMessages`, the bounded
batch loop that already exists for exactly this set. `sessions.id` is
AUTOINCREMENT, so the freed id is never handed to another session while those
rows still name it.
The longest replace transaction is now 746-848 ms, the same band as a fresh
read. The trade is stated plainly: the pass does more total work, 3.7 s
against 3.1 s, because the reclaim is 360 bounded transactions with an
incremental_vacuum step each instead of one large delete. It yields between
them, so none of it holds the process, and returning the pages per batch also
takes the index from 173.6 MB to 162.0 MB.
The store schedules the drain off the committing stack: an async function runs
synchronously to its first `await`, so calling it inline would put the first
batch back inside the transaction's own call. One drain at a time, with a
repeat flag for a replace that commits while one is running.
* fix(ai-vault-search): preserve search tokens and index Codex tools
* fix(ai-vault-search): match SQLite marks and Codex file-change records
* fix(ai-vault-search): preserve stat pairs and validate benchmark results