mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 00:02:31 +00:00
701dc2211c39ab002f89ae6a3b632bfd39c8c070
669
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Neil
|
701dc2211c | perf(mobile): precompute task sort keys and reuse repository collation (#20233) | ||
|
Neil
|
ef3b7e83b9 | perf(mobile): reuse numeric collators across source control sorts (#20224) | ||
  OrcaWinandm4air
|
9b2b02bb3b |
perf(mobile): reuse UTF-8 prefix truncation for diagnostics (#20358)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
e74075d351 |
perf(localization): prefilter absent brands before regex matching (#20354)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
a0d26cadd8 |
perf(computer): avoid rescanning fragmented native provider replies (#20348)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
91ecd32711 |
perf(terminal): reuse bounded native snapshot byte checks (#20333)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
Neil
|
30424810fe | perf: use native newline search in rich Markdown code stripping (#20320) | ||
|
OrcaWinandm4air
|
d57011fb34 |
perf(checks): bound log excerpt byte-limit checks (#20319)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
99cca036e7 |
perf(ssh): copy cleanup tab map only once (#20315)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
 OrcaWinandOrca Worker
|
382c281cfa |
perf: normalize manifest text files once (#20294)
Co-authored-by: Orca Worker <orca-worker@localhost> |
||
|
OrcaWinandm4air
|
51ff9c7ff3 |
perf(ssh): skip terminal source spans awaiting ACK publication (#20290)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
Neil
|
69c76b5ec7 | perf: bound rich Markdown comment detection and preservation scans (#20287) | ||
|
OrcaWinandm4air
|
f704374e2b |
perf(history): scan legacy alternate-screen markers linearly (#20267)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
Neil
|
57c7ef5835 | perf(plugins): stop collection traversal at message size limit (#20260) | ||
|
OrcaWinandm4air
|
1fe92719af |
perf(skills): index source ownership counts per discovery (#20259)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
ee1e94a623 |
perf(history): cache numeric transcript timeline bounds (#20244)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
56aefb542e |
perf(mobile): bound autocomplete substring retention (#20226)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
OrcaWinandm4air
|
baa1cb135c |
perf(mobile): avoid materializing input characters on backspace (#20220)
Co-authored-by: m4air <m4air@Mac.localdomain> |
||
|
Neil
|
20fb3e7d13 | perf(mobile): normalize history scope paths once per candidate (#20210) | ||
|
Neil
|
56fcb544e0 |
fix(browser): move cookie scoping off psl's stale suffix list (#20421)
* fix(browser): move cookie scoping off psl's stale suffix list psl@1.15.0 is its latest release and ships a Dec-2024 snapshot of the public suffix list. Measured against the current upstream list, it fails to recognise 600 of 10,030 suffixes; tldts misses 2. That gap is a cookie-isolation bug. psl does not know `api.br` is a suffix, so it falls back to the `br` rule and maps foo.api.br, bar.api.br and example.api.br all onto the single family `api.br`. Unrelated registrants then share a removal scope, and a replace-mode import for one clears the others' cookies. The same holds for seg.ar, co.az, gov.cz and ~597 more. tldts is called with allowPrivateDomains, without which the PSL's PRIVATE section is ignored and every *.github.io / *.s3.amazonaws.com / *.vercel.app tenant collapses into one family — 21 of 49 probed hosts changed family under the default. The new test pins that boundary. One deliberate behaviour change: hosts under `.local` (not in the PSL) were their own family under psl, which returned an all-null parse for them; they now resolve to the two-label boundary (app.orca.local -> orca.local), matching what Chromium treats as the registrable domain. * fix(build): bundle tldts into the main process like psl was psl sat in BUNDLED_MAIN_DEPENDENCIES, so it was inlined into the main bundle rather than externalized and copied into resources/node_modules. Swapping the dependency without moving that entry left a bare tldts import that afterPack's runtime-closure check rejects. * fix(build): point the output contract at tldts and drop the psl shim The contract test still asserted psl was in BUNDLED_MAIN_DEPENDENCIES, so it failed once the entry became tldts. src/types/psl.ts declared a module that no longer resolves; tldts ships its own types. * test(browser): pin the suffix boundaries the tldts swap moved Three semantic changes shipped untested: - `.local` is unlisted, and the libraries disagreed on what that means. psl returned an all-null parse so every `*.orca.local` host was its own family; tldts stops at `orca.local`. The consequence is wider than the family name — importDomainAncestors now yields the shared parent, so a replace-mode import of one host clears non-host-only cookies every sibling shares. - psl's snapshot had `compute.amazonaws.com` as a literal PRIVATE suffix; the current list only carries the wildcard, so the bare host is ICANN now. - The renderer's `psl.isValid` gate had no direct test at all — nothing imported the module from a test. Also drops comments that explained a boundary in terms of psl's internals. One was wrong under tldts: bracketed IPv6 does not reach an error branch, it parses with the brackets stripped and falls through the unlisted path. |
||
|
Neil
|
403b62a8d8 |
ci: balance existing unit and E2E shards using recorded timings (#20367)
* ci: balance existing unit and E2E shards using recorded timings * ci: fix timing refresh units and deferred-menu test traversal * ci: preserve isolated E2E window launch policy * ci: keep diagnostic artifact outages from failing tests |
||
|
Neil
|
4aa9329e99 |
ci: narrow pnpm cache keys and shallow development checkouts (#20370)
* ci: cache dependency downloads and shallow development checkouts * ci: defer mobile caches after measuring restore overhead * ci: retain existing release signing cache behavior |
||
|
Neil
|
57b0355f72 | perf: index Hermes session correlation instead of rescanning history (#20275) | ||
|
Neil
|
de8e5421ff | ci: reuse immutable package setup across shutdown checks (#20368) | ||
 Jinwoo Hong
|
eedd35645e |
feat(mobile): add typed RPC operations and fence raw requests (#20018)
* feat(mobile): add the RpcOperation descriptor, send, and barrier interpretation An operation family declares its method, compatible reader, acceptance policy and interpretation barrier once. The send classifies only a fulfilled envelope; transport rejection stays on the promise channel as the original error object, so the cutover and delivery-unknown predicates keep working and a Promise.all group still fails fast. Multi-request families go through a post-barrier combinator that awaits every raw request and then interprets in declared order. No production call site is migrated: this lands as self-contained machinery so runtime behaviour is provably untouched. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * fix(mobile): require a reader for RPC result variants * refactor(mobile): fence the raw RPC request port behind an inventoried boundary The raw sender takes an unchecked method string and returns an envelope whose result is `unknown`; 153 non-test files still reach it and each re-decides acceptance and decoding for itself. The type system cannot close that today — `RpcClient` structurally carries `sendRequest` and ~190 files hold a client — so move the port's declaration into its own module, name it unvalidated, and hold the boundary as a ratcheted inventory instead. `SendRequestOptions` is re-exported from rpc-client.ts so the move touches no call site, and rpc-operation.ts now asks for the port rather than the whole client: it is the one module allowed to cross it. Two ratchets, both AST-based: - the port inventory fails on an unlisted file, a stale entry, and a listed file whose reference count went up, so the list only shrinks; - the cast fence bans `as`, `any` and `@ts-` suppressions in the operation region, which is computed from the imports rather than listed, so step 4's operation modules land inside it automatically. Zero runtime change: no wire change, no call site touched. Claude-Session: https://claude.ai/code/session_01JNnE9qzUZMMnqpZWCqM3nb * merge: incorporate closed boundary and send-side types * fix(mobile): preserve RPC decoding invariants across the combined boundary * fix(mobile): consolidate RPC operation test imports * refactor(mobile): simplify RPC descriptors and fence the contract module * fix(mobile): baseline landed notification RPC callers |
||
|
Jinwoo Hong
|
fa3d29fa11 |
feat(ai-vault-search): session search query engine over the index (#20029)
* feat(ai-vault-search): give the index a generation a page cursor can be fenced by
A search page is a slice of one ranked list, so a cursor only means anything
against the snapshot that produced it. The store now keeps a monotone
generation in `meta`, bumped by every mutation that can change which rows a
read returns, and starts a new one on open so a write whose bump never landed
cannot leave a cursor pointing at content that is already gone.
Schema version 2 adds the two tables the query layer needs: `messages_vocab`
(fts5vocab over messages_fts, the typo repair's whole dictionary) and
`search_log`. Both are pure additions and the index is a cache, so a version-1
file is dropped and rebuilt exactly as any other mismatch is.
* feat(ai-vault-search): plan a query the way the index tokenized it
The planner unfolds the tokenizer contract instead of asking SQLite: same
boundaries as `unicode61 tokenchars '_.-/+'`, pinned against real fts5vocab
output so a query can be planned without a round trip. It decides the route
ladder's first rung (literal shape), strips stop words from prose but never
from a literal, and fans an identifier out into its pieces for the OR fallback.
Typo repair uses the index's own vocabulary as its dictionary, so it can never
suggest a term the index does not hold. Its exact-match probe now joins
`visible_messages`: `messages_vocab` is a view over the FTS b-tree and still
lists staged and tombstoned terms, and the PR 2 read ratchet is right to
demand the join.
`splitAiVaultSearchQuery` is the index's reading of repo: / path:. It keeps
operator case, which the panel folds and cwd_key must not; a census test pins
the two parsers to the same answer about what is an operator until PR 7 moves
the panel onto this one.
* feat(ai-vault-search): narrow a search the way the sidebar keys a folder
Every caller-supplied narrowing in one place, so retrieval, the operator-only
page and the session load cannot drift apart: agents, an updated-at floor, the
retention cutoff, scope paths, and the repo: / path: operators.
Scope keying goes through PR 2's `cwdKey`, which is the sidebar's
`folderGroupKey` without its prefix, rather than the original branch's second
spelling. That drops the branch's WSL distro qualification, which PR 2 removed
on purpose, and it makes the filesystem root a key that already ends in a
separator, so the child-prefix range is built from the key rather than by
appending one; `//` sorts below every real child and would scope the root to
nothing.
Engine types land here too, under src/main and not src/shared: nothing in this
PR is a wire type, and PR 5 lifts what a caller may receive.
* feat(ai-vault-search): rank, page and answer a session search
`SessionSearchEngine.search()` over the PR 2 store: route ladder (phrase, AND,
typo repair, OR), BM25 weights per corpus, one hit per session, fork folding,
and a page.
- `scope` picks the corpus and the engine never second-guesses it.
`conversation` is user and assistant turns; `all` adds tool output and the
identifier shadow column. Switching corpus while typing is PR 7's policy; an
engine that widened on a miss would make a result impossible to reproduce
from its own request.
- Pagination is an offset into one ranked list, fenced by the index generation
and by a hash of everything that changes the ranking. A cursor from another
generation or another query is refused with a typed error rather than
silently re-run. Ranking breaks every tie by session id, because a cursor
indexes into that order and retrieval does not promise one.
- Snippets and source presence are paid for by the page, not the list. A
snippet past the per-hit ceiling is cut on a code point, never between `[[`
and its `]]`, and flagged on the hit.
- Source presence is read from the `files` table. No stat on the query path,
and no `missing`: only a proven deletion may claim one, and this read cannot
prove it.
- `SESSION_SEARCH_CANDIDATE_LIMIT_DEFAULT` is an option, not a constant, and
the result says when the limit was what cut the answer.
- The engine is the only caller of `store.warm()`, on first search.
Library only: no IPC, no settings, no Electron, nothing constructs it in
production.
* perf(ai-vault-search): measure what a query costs and what the candidate limit buys
Two corpora, because they answer different questions. The 10.5 MB / 40-session
corpus is what the scope split costs a reader: conversation is about 1.6x
faster at p50 and 3.4x at p95 than the full corpus, which is the argument for
the second FTS table being the one a keystroke can afford.
The candidate limit needs more sessions than the limit before it costs
anything, so it is swept over 2,500 one-turn transcripts with every session
matching. Limits are interleaved sample by sample: run back to back, the first
configuration pays for every page the OS cache had not seen and the ordering
alone moved p95 further than the limit did.
The doc says plainly what these numbers do not cover. They are cost, not
relevance; the MRR figures quoted beside the BM25 weights and the identifier
shadow column come from a shoot-out over real transcripts and cannot be
reproduced from this repository.
* refactor(ai-vault-search): key a page once per search, and report reachable pages honestly
The page key was hashed twice per search, once to decode the incoming cursor
and once to mint the outgoing one. The benchmark's reachable-page figure was
clamped against a constant that could never bind; it is the limit over the page
size and nothing else.
* test(ai-vault-search): pin the two engine seams nothing was holding
The warm wiring and the query-length cap were both written and neither was
observable. A spy pins that a search is what warms the store, and the cap is
pinned through the one input the planner's own term limit does not already
bound: a single enormous token, where the cut is what decides whether the term
matches the indexed one at all.
* fix(ai-vault-search): fence the generation against writers this process cannot see
The generation was cached in memory and moved only on this store's own writes,
and the bump itself was a read-then-write outside any transaction. Two handles
on one file is the normal case once PR 3 lands: the indexer writes in the
scanner child while an engine reads elsewhere. A reader would see that writer's
deletions while its own generation stood still, honour a stale cursor, and skip
a session; two writers could mint one generation for two snapshots.
`generation` now reads `meta` on every call, and the bump is a single
`ON CONFLICT DO UPDATE ... + 1` inside the transaction that makes the change.
That closes the crash hole the bump-on-open existed for, so opening a store no
longer invalidates anyone's cursor.
Only a change to what a read returns counts. Retiring a path the index never
held hides nothing, and the row deletes that drain a tombstone take away rows
that were already invisible: bumping there would refuse a cursor every 256 rows
and leave pagination unusable for as long as indexing ran.
Also drops redaction from the query log, following PR 2's decision to store
transcript content as written; the module it called no longer exists.
* fix(ai-vault-search): answer from a version-1 index, and keep a repaired literal whole
Two smaller findings.
An engine can be handed a connection to a version-1 file that another handle is
still answering from, and this PR is what first makes that reachable. It now
probes once for the two tables version 2 added and names what it cannot serve
on every result, instead of throwing at the first query that reaches for a
vocabulary that is not there. The route ladder simply skips its repair rung.
Which process may unlink and rebuild the index is PR 3b's decision and is not
solved here.
Typo repair re-planned the corrected query from scratch, and a corrected
spelling can read as prose even when what was typed was a literal:
`parseJsonn(the, data)` has the punctuation, `parsejson the data` does not, so
the re-plan dropped `the` as a stop word. The repaired query searched for less
than was asked and `repairedTerms` reported a body nobody typed. The re-plan is
now told what the original decided, because a repair changes spellings, not the
query's character.
`repairedTerms` is documented as the whole body the repaired plan ran, with the
index's own case-folded spelling for the terms it corrected.
* fix(ai-vault-search): make repo: and path: mean one thing in the list and the index
The engine said `repo:` and `path:` a second time, in SQL, and SQL cannot say
them. LIKE folds ASCII and nothing else, so `path:CAFÉ` missed `café`; the
engine searched `cwd_key` while the panel searches the working directory and
the transcript path, so `path:jsonl` matched every session in the panel and
none in the index; and the engine compared one path segment where the panel
compares the last two, so `repo:orca/session-search` missed. All four
reproduce in both directions.
So there is one definition now, not two that resemble each other.
`matchesAiVaultQueryOperators` moves into the shared filter module beside the
panel that already owned the semantics, the panel calls it, and the engine
applies it over the rows it retrieved. SQL keeps only what it can express
exactly: the `cwd_key` prefix range for `scopePaths`.
`parseVaultQuery` now parses through `splitAiVaultSearchQuery`, so one parser
decides what an operator is. Its existing tests pass unchanged; four degenerate
shapes do answer differently and are pinned as decisions rather than left to be
discovered.
Two consequences worth stating. The operators are conjunctive now, because that
is what the panel has always done, where the engine had been ORing within a
key. And the operator-only page walks newest sessions in bounded pages applying
the predicate, rather than taking one cut of the newest N and filtering it,
which would have answered `repo:x` with nothing on a busy index.
* docs(ai-vault-search): re-measure the query benchmark after the operator change
repo: and path: moved out of SQL, so the operator-only row measures something
different now and the note that it is a range seek was wrong. The rest of the
table is re-measured on an idle machine: the previous run's p95 column was
mostly contention, which is why conversation looked 3.4x faster at p95 rather
than the 1.7x it actually is.
Adds what this PR does not settle: which process may unlink and rebuild the
index is PR 3b's, and PR 4 is only the first thing that makes reading it
reachable.
* fix(ai-vault-search): stop reporting a search that gave up as a search that finished
The operator-only walk stops at a scan ceiling as well as at a full candidate
set, and only the first of those reached the result. A query whose one match
sat past the ceiling came back with no hits and truncated.candidates false,
which is the engine claiming there is nothing to find when what happened is
that it stopped looking. Retrieval now says why it stopped, because it is the
only layer that knows, and the count it used to return could not distinguish
the two cases.
The cursor fence stays as it is: any published read moves the generation, so an
outstanding cursor is refused, and that is what F11 asked for. What was wrong
was the claim next to the row-delete skip that pagination stays usable through
indexing. It does not, and the engine now says so. The rejection carries the
generation the cursor was minted in and the one the index is at, so a caller
can tell a moved index from a bad cursor and re-issue page one without showing
anyone an error.
The capability probe was nearly dead code, since every store opens through a
function that rebuilds a stale file. It is not dead, because two handles can be
open on one file, so the claim is corrected rather than the probe deleted. It
now runs per search: a verdict cached in the constructor is wrong in both
directions once another handle rebuilds the index.
Also says why the row-delete loop may skip the bump: those messages keep
batch_id NULL and stay in visible_messages, so what makes them unreachable is
their session's tombstone, and the read ratchet is what keeps every reader
joining the view that applies it.
* fix(ai-vault-search): restore the panel's reading of a quote that does not end a word
Unifying the two parsers changed panel behaviour on nine of twenty probed
shapes, not the three previously pinned. Six of the nine were regressions, all
from one rule: the shared parser refused a quoted span whose closing quote was
not followed by a space, so `"a b"c` and `repo:"a"b` became single terms
carrying their own quote characters, which match nothing.
The rule was justified as what stops the apostrophes in `it's a repo:orca
thing's` from swallowing the operator between them. It is not: a span only ever
opens at a token start, and the quote in `it's` is not at one. Dropping the
rule restores all six shapes to what the panel has always done and leaves that
protection intact.
Three changes remain and are kept because the old answer was worse in each: an
operator with an empty quoted value is dropped rather than filtering on `""`
and silently emptying the list, and a bare pair of quotes reads as an empty
term rather than as the two characters. Each is pinned with a test that says
which behaviour it is and why.
* fix(ai-vault-search): trim operator values, and report a query the engine had to cut
Three lows.
`repo:" "` survived as a term and matched no label, silently emptying the
list, which is the exact defect the empty-value drop exists to prevent wearing
different clothes. Operator values are trimmed, and a whitespace-only one drops
like an empty one. The substring matcher's copy of a free-text term is trimmed
too, so `" "` reads as the empty term already does; the span kept for FTS is
still the query verbatim.
Two caps upstream of retrieval fired silently: the planner searches at most 48
terms, and the engine cuts the query at 512 characters. A 56-term query whose
only match was the 56th came back with no hits and nothing truncated, which
claims there is nothing to find. `truncated.query` now says when either fired,
alongside the candidate and snippet flags that already did.
Every cursor refusal now carries the generation the index is at, which the
engine knows before it looks at the cursor, and the generation the cursor
claimed wherever that survived parsing. The doc says exactly when each is
present instead of leaving absence unexplained.
* refactor(ai-vault-search): read the tables the simplified index writes, and own the fence
PR 2 deleted the visibility views, the staging tables and the store's
generation, so this reads `sessions` and `messages` directly and carries the
three schema objects only a query needs — the vocabulary, the query log, and
the triggers that move the generation — as its own extension over the store's
schema.
The fence is now three triggers on `files`, because every transaction the store
opens that can change what a search returns writes that table and nothing else
does; retention's orphan drain is the one write path that touches neither, and
it is the one that must not bump. The triggers live in the file, so a writer in
another process moves the generation without knowing a reader exists.
A message row can now outlive its session row until the drain reaches it, so
the snippet read joins `sessions` and the typo repair asks for a live posting
instead of trusting the vocabulary's document count.
The engine takes a connection rather than a store: PR 2's store keeps its
connection private, and which process may open or rebuild the index file is
PR 3b's decision, not a query engine's.
* perf(ai-vault-search): price the second FTS table, and re-measure without warmup
Open decision 3. A column filter over `messages_fts` returns the identical
rowid set as `conversation_fts` — checked here per query rather than assumed —
so the table exists for latency alone. On a 105 MB corpus at both ends of the
tool-output band, the column-filtered form costs 1.16-1.42x at p95, against a
bar of 2x, so the recommendation is to delete it.
The shoot-out writes its own corpus because the answer turns on the one
property the shared generator fixes: how much of a transcript is tool output.
Half the tokens in that output are words the conversation also uses, which is
deliberately generous to the table under question.
The doc records the number that argues the other way. PR 2 priced the table at
about a quarter of the index on a corpus whose tool output is 56% of its
message text; on a tool-heavy one it is 6.7-11%, because `messages_fts` grows
with the tool text and the second table does not.
Page warmup is not re-added. The measurement behind it was on a 4 GB index,
removing it moves this corpus by less than the run-to-run spread, and a
cancellable background pass needs a lifecycle a query library does not have.
* test(ai-vault-search): pin that an append moves the generation a cursor is fenced by
* refactor(ai-vault-search): answer the conversation scope with a column filter
PR 2 deleted `conversation_fts` on the strength of this PR's shoot-out, so the
scope is a column filter over the one FTS table now. `ftsTableFor` is gone; a
scope is a pair of `scopedExpression` and `scopedWeights`, and the table name
no longer travels through the engine, the snippet builder or a hit.
The filter is parenthesised, and that is the whole of it: `{cols}: (a AND b)`
binds both terms, while `{cols}: a AND b` binds only the first and searches
tool output for the rest. A test drives an AND whose second term lives only in
tool output through both scopes.
The snippet keeps one guard, not two. Its column list and its expression were
each hiding the other's mistakes — a tool-only row was unreachable through
either — so the list is the same four columns for every scope and the scoped
expression is what makes a conversation snippet impossible to draw out of tool
output. Dropping it now leaks that row, which a test catches.
One behaviour the deleted table did not have, pinned rather than wished away:
bm25 normalises by the whole row's length and has no per-column length, so two
rows with identical prose score differently when one also holds tool output.
The rowid set is unchanged; the order within it can move.
Re-measured on the shipping schema. The conversation scope is 1.2-1.4x faster
than `all` at every rung, and the index is 57 MB rather than about 150 MB at
93% tool output, because a tool row is now capped at 3,072 characters.
* fix(ai-vault-search): repair a spelling inside the scope that will answer it
Typo repair read `messages_vocab` and probed `messages_fts` with no column
filter, so tool output decided whether a conversation-scoped query was
repaired, in both directions. A tool row carrying the misspelling made the
query look correctly spelled and suppressed the repair; a tool row carrying a
rare word became the suggestion, naming in `repairedTerms` a string from a
column the scope will never show. Both reproduced against a control index that
differs by exactly that one row.
The vocabulary proposes and a scoped count disposes. fts5vocab is per table and
cannot be column-filtered, so every decision that reaches the plan — already
spelled right, eligible, and which of two equally close candidates wins — now
comes from a `messages_fts MATCH` under the same filter retrieval uses, joined
to `sessions`.
That also takes the vocabulary's `doc` out of the ranking, which is the half of
the drain defect that belongs here: `doc` counts rows whose session a purge has
already cut loose, so reclaiming them changed which word a query was repaired
to. Candidates are ordered by term now, because the ordering decides which of
them survive the scan limit, and ties on similarity go to the more common word
counted live rather than to the vocabulary's number.
The cost is one bounded count per candidate examined, at most eight per prefix,
and only for a term the scope has no posting for at all.
* fix(ai-vault-search): fence the rows a purge reclaims after it cuts a session loose
Retention's second half deletes from `messages` alone and touched neither
`files` nor `sessions`, so it moved no generation. The argument was that those
rows answer nothing, which was true of retrieval and not of the engine: the
typo repair's dictionary is a view over the FTS b-tree and listed them, so a
drain running between two pages swapped the repair under a cursor that was
still honoured, and a search that had answered stopped answering.
The commit before this one fixes that at its source by counting live rows. It
does not make the drain provably inert — the vocabulary still decides which
candidates survive its scan limit, and reclaiming a term's last row moves where
that limit cuts — so the fence is what covers the rest.
A fourth trigger, on `messages`, with a `WHEN` clause that is the whole reason
it is affordable: a replace and a `removeFile` delete a session's rows while
its `sessions` row still stands, so neither fires, and both already bump
through `files`. Only the drain deletes a row whose session is gone.
The price is named rather than avoided: a cursor outstanding while a purge runs
is now refused once per batch, which `SessionSearchCursorError` reports as
`stale-generation` so a caller re-issues page one. The test that pinned the old
contract is replaced by one for the new one, and by one proving a replace still
does not fire it.
* fix(ai-vault-search): tell a highlight from a transcript that contains brackets
The snippet builder asked each of a row's four columns for a marked snippet and
showed the first whose text contained `[[`. Transcripts contain `[[`: a bash
`if [[ -f … ]]`, numpy's `[[1, 2], [3, 4]]`. A row matching only in tool output
was shown its user turn instead, with nothing highlighted in it, and the
any-column fallback an identifier-only match depends on was unreachable behind
the same collision.
Whether a column matched is now the difference between two renderings of the
same text: `snippet(…, MARK, MARK, …)` beside `snippet(…, '', '', …)`. Content
cannot forge a difference between those two, because it is the same content
either way.
The marks FTS5 inserts are private-use code points, rewritten to the public
`[[` and `]]` once, at the end. That is for the other decision that has to tell
a mark from content: the truncation refuses to cut between an open mark and its
close, and a transcript's own bracket used to move that cut.
* fix(ai-vault-search): cut a query on a code point and bind ids in batches
Two small ones from the review's not-routed list.
`query.slice(0, 512)` can land between the halves of a surrogate pair, leaving
a lone half that matches nothing and that a caller cannot echo back. The reader
already has `sliceAtCodeUnitLimit` for exactly this.
`loadSessions` bound one parameter per candidate id in a single statement. The
list is as long as the candidate limit, the tuning doc invites a host to raise
that limit, and SQLite's `SQLITE_MAX_VARIABLE_NUMBER` is 999 on builds older
than 3.32 — so one settings change away from `too many SQL variables`. Read in
batches of 500, leaving room for the filter's own bound values.
* docs(ai-vault-search): price the repair rung, and record what is left open
Typo repair is the one rung whose cost tracks the vocabulary rather than the
result, and it only runs for a term the scope has no posting for. Measured over
1.6 M distinct terms: 10 ms for one unknown term, 387 ms for a 480-character
query of thirty-nine of them.
The scoped-count fix made that cheaper rather than dearer, from 737 ms, because
ordering the vocabulary scan by term drops the sort `doc DESC` needed and the
counts it adds are at most eight bounded probes per prefix. A cap on unknown
terms per query is a follow-up in the split plan, with the five other items the
final review raised and did not route.
* test(ai-vault-search): make each snippet mark mechanism answer for itself
Two mechanisms landed together and hid each other: choosing a column by
comparing a marked rendering against an unmarked one, and marking with
private-use code points instead of `[[`. Either alone fixed the bracket repro,
so neither had a mutation against it — the same masking the snippet's two
column guards had a round ago.
They do different jobs, so both stay and each gets the test that needs it. A
transcript holding a private-use code point of its own is what the comparison
is for; agent output carries Nerd Font glyphs from that block. A snippet past
the character ceiling with a bracket after its last real mark is what the
private-use marks are for, because the truncation has to find that mark by
searching the text.
The two one-line fixes get honest framing rather than a mutation neither can
have. A lone surrogate is not a token character, so the planner drops it either
way and the safe cut is hygiene. And no SQLite this stack runs refuses 1,100
bound ids — 32,766 has been the floor since 3.32 — so the batch is about
owning the ceiling here rather than rescuing a reachable failure.
* refactor(ai-vault-search): keep the scope's expression with the other expressions
Making the typo repair ask its questions in the search's own scope put an
import from retrieval into it, and retrieval already owns the repair — a cycle
the native audit catches. `scopedExpression` belongs beside `phraseExpression`,
`andExpression` and `orExpression` anyway: it builds a MATCH expression, and
two callers now need it. The bm25 weights stay in retrieval, where the SQL that
uses them is.
* docs(ai-vault-search): say which delete paths fire the orphan-reclaim trigger after a replace cuts loose
* test(ai-vault-search): make the trigger-restore test exercise the trigger it drops
* fix(ai-vault-search): keep a scope nothing could key from widening the search
* fix(ai-vault-search): read a fractional or negative cursor generation as malformed
* fix(ai-vault-search): highlight only the marks FTS5 inserted, not the text's own
* test(ai-vault-search): compare the whole query, so a quoted operator value survives
* fix(ai-vault-search): filter routes and fence page reads; simplify query engine
* refactor(ai-vault-search): narrow retrieval API and clarify page rejection
|
||
  Brennan BensonandMerge Sim
|
76c8e91d4a |
fix(e2e): run worktree first-paint probe on a mapped window (#20197)
Co-authored-by: Merge Sim <sim@local> |
||
|
Neil
|
9c918b93a6 | fix(deps): take Electron 43.7.0 for the glibc environ use-after-free (#20089) | ||
|
Neil
|
6bb2b0c6d7 |
test(runtime): capture real Antigravity transcripts — the detector is inverted on live output (#19983)
* test(runtime): capture real agent PTY transcripts before rewriting Antigravity readiness
Antigravity readiness has been written five times against a five-line screen
typed from memory. There is no Antigravity transcript in this repository, so
every attempt was a guess tested against another guess. This adds the recorder,
the protocol and the fixture-driven suite so the sixth attempt can be written
against evidence, and changes no detector logic.
- config/scripts/capture-agent-pty-transcript.mjs records a live agent session
through a real PTY, escapes and wrapping intact. Ctrl-] is consumed by the
recorder and never forwarded, which is the only way to end a capture while a
dialog still owns the screen.
- config/scripts/pty-transcript-secret-scan.mjs finds account identifiers and
credentials, redacts them with same-length placeholders so wrapping survives,
and recognises its own placeholders so a scrubbed file verifies clean.
- src/main/runtime/antigravity-readiness-transcripts.test.ts asserts a verdict
per transcript and skips by name until the transcripts land, with a
doc-coverage ratchet and a guard that a fixture contains escape bytes.
The escape-byte guard exists because the three cursor-agent fixtures carry a
comment claiming they were captured verbatim through Orca, yet contain zero ESC
bytes and zero carriage returns. That comment is corrected here to say what
those files are; the fixtures and the rules built on them are untouched.
* test(runtime): capture real Antigravity transcripts, and pin what they prove
`agy` 1.1.25 turned out to be installed, so the transcripts this scaffold was
built for now exist. Six are recorded from live sessions and committed; the
rest are named as skipped, because reaching them would mean signing the
operator out or deleting their config.
The captures invert the story. On real output the shipped detector refuses a
genuinely ready screen and accepts a live `/model` picker:
- Antigravity paints a block-glyph logo down the left, so the model row never
starts a line. `startsWith('gemini', trimmedStart)` cannot match a real ready
screen, on any account or model. Stripping the logo flips the same screen to
ready, which means a decorative glyph decides readiness today.
- The `/model` picker prints `Gemini 3.x Flash` one per line, at line start, and
a bare `>` composer sits earlier in the tail. Both halves of the rule are
satisfied while a dialog owns the screen.
- For an API-key user the identity row reads `Gemini API key` — no `@`, no
domain — and `AGY_CLI_HIDE_ACCOUNT_INFO=1` removes the row entirely. The
account-row requirement of attempts 4 and 5 can never pass for those users.
- The banner is printed once and never reprinted after a dialog is dismissed, so
`headerIndex` cannot be the ordering anchor.
Four suite cases are pinned as KNOWN DEFECT: they assert what the detector does
so CI stays honest instead of permanently red, and flip to failing the moment
someone fixes it. No detector logic changed.
The recorder gains `--send "<ms>:<text>"` because a dialog capture has to be
driven and an unattended run has no TTY, and the scrub scanner gains a UUID rule
because agy prints a resumable conversation id on exit.
* test(runtime): capture agy mid-turn, and make the scan file reviewable
Answers the busy-frame question a P1 review raised against attempt six, with
two new captures from a live turn.
At the frame level the review is right: a busy frame parks the caret with the
same bytes as an idle one, `CR ESC[2A ESC[2C`, and the only differing row —
`esc to cancel` versus `? for shortcuts` — is erased by that park.
At the retained-tail level it does not reproduce. Each spinner tick is its own
repaint with its own `CR ESC[2A`, two rows higher than the frame's, which
splices the composer away: a live turn's tail ends on `⣟ Generating...`, with
no bare caret to match. A constructed input that keeps the park and edits only
the status text is not faithful, because a live turn has a spinner row
repainting below the composer.
The residual is the gap between a frame park and the next tick, where the tail
does end on the bare caret. Quiescence-gated paths are safe there because ticks
keep arriving; text-only paths are not, and for those the capture supports one
clause: a braille glyph on the last visible line means working. That predicate
already exists here for cursor-agent and should be reused, scoped to the last
line — a first-run transcript prints `⠾ Signing in...` during startup.
Also in this commit, from the same review:
- pty-transcript-secret-scan.mjs held raw 0x00-0x1f bytes in a character class,
so the one file gating real PTY data into history was binary to git and
unreviewable in a diff. It now tests codepoints, which the formatter cannot
fold back into control bytes.
- Pin `src/main/runtime/__fixtures__/*.txt` as -text. A Windows checkout would
otherwise normalise line endings and rewrite the CR bytes that make these
files evidence.
The recorder now stops appending at the stop moment rather than through
shutdown: an agent repaints an idle frame on its way out, which was overwriting
the mid-turn state the capture existed to record.
* test(tooling): allowlist the transcript scan test in the batch-shim ratchet
pty-transcript-secret-scan.test.mjs asserts that the capture recorder routes
an 'agy.cmd' shim through cmd.exe, so the shim literal it names is the
assertion, not a spawn. Fits the existing assert-on-shim-files category.
|
||
|
Jinwoo Hong
|
f31bfa8fb7 |
Revert "feat(ai-vault-search): session search query engine over the index (#19750)" (#20023)
This reverts commit
|
||
|
Jinwoo Hong
|
cf7408a37f |
feat(ai-vault-search): session search query engine over the index (#19750)
* feat(ai-vault-search): give the index a generation a page cursor can be fenced by
A search page is a slice of one ranked list, so a cursor only means anything
against the snapshot that produced it. The store now keeps a monotone
generation in `meta`, bumped by every mutation that can change which rows a
read returns, and starts a new one on open so a write whose bump never landed
cannot leave a cursor pointing at content that is already gone.
Schema version 2 adds the two tables the query layer needs: `messages_vocab`
(fts5vocab over messages_fts, the typo repair's whole dictionary) and
`search_log`. Both are pure additions and the index is a cache, so a version-1
file is dropped and rebuilt exactly as any other mismatch is.
* feat(ai-vault-search): plan a query the way the index tokenized it
The planner unfolds the tokenizer contract instead of asking SQLite: same
boundaries as `unicode61 tokenchars '_.-/+'`, pinned against real fts5vocab
output so a query can be planned without a round trip. It decides the route
ladder's first rung (literal shape), strips stop words from prose but never
from a literal, and fans an identifier out into its pieces for the OR fallback.
Typo repair uses the index's own vocabulary as its dictionary, so it can never
suggest a term the index does not hold. Its exact-match probe now joins
`visible_messages`: `messages_vocab` is a view over the FTS b-tree and still
lists staged and tombstoned terms, and the PR 2 read ratchet is right to
demand the join.
`splitAiVaultSearchQuery` is the index's reading of repo: / path:. It keeps
operator case, which the panel folds and cwd_key must not; a census test pins
the two parsers to the same answer about what is an operator until PR 7 moves
the panel onto this one.
* feat(ai-vault-search): narrow a search the way the sidebar keys a folder
Every caller-supplied narrowing in one place, so retrieval, the operator-only
page and the session load cannot drift apart: agents, an updated-at floor, the
retention cutoff, scope paths, and the repo: / path: operators.
Scope keying goes through PR 2's `cwdKey`, which is the sidebar's
`folderGroupKey` without its prefix, rather than the original branch's second
spelling. That drops the branch's WSL distro qualification, which PR 2 removed
on purpose, and it makes the filesystem root a key that already ends in a
separator, so the child-prefix range is built from the key rather than by
appending one; `//` sorts below every real child and would scope the root to
nothing.
Engine types land here too, under src/main and not src/shared: nothing in this
PR is a wire type, and PR 5 lifts what a caller may receive.
* feat(ai-vault-search): rank, page and answer a session search
`SessionSearchEngine.search()` over the PR 2 store: route ladder (phrase, AND,
typo repair, OR), BM25 weights per corpus, one hit per session, fork folding,
and a page.
- `scope` picks the corpus and the engine never second-guesses it.
`conversation` is user and assistant turns; `all` adds tool output and the
identifier shadow column. Switching corpus while typing is PR 7's policy; an
engine that widened on a miss would make a result impossible to reproduce
from its own request.
- Pagination is an offset into one ranked list, fenced by the index generation
and by a hash of everything that changes the ranking. A cursor from another
generation or another query is refused with a typed error rather than
silently re-run. Ranking breaks every tie by session id, because a cursor
indexes into that order and retrieval does not promise one.
- Snippets and source presence are paid for by the page, not the list. A
snippet past the per-hit ceiling is cut on a code point, never between `[[`
and its `]]`, and flagged on the hit.
- Source presence is read from the `files` table. No stat on the query path,
and no `missing`: only a proven deletion may claim one, and this read cannot
prove it.
- `SESSION_SEARCH_CANDIDATE_LIMIT_DEFAULT` is an option, not a constant, and
the result says when the limit was what cut the answer.
- The engine is the only caller of `store.warm()`, on first search.
Library only: no IPC, no settings, no Electron, nothing constructs it in
production.
* perf(ai-vault-search): measure what a query costs and what the candidate limit buys
Two corpora, because they answer different questions. The 10.5 MB / 40-session
corpus is what the scope split costs a reader: conversation is about 1.6x
faster at p50 and 3.4x at p95 than the full corpus, which is the argument for
the second FTS table being the one a keystroke can afford.
The candidate limit needs more sessions than the limit before it costs
anything, so it is swept over 2,500 one-turn transcripts with every session
matching. Limits are interleaved sample by sample: run back to back, the first
configuration pays for every page the OS cache had not seen and the ordering
alone moved p95 further than the limit did.
The doc says plainly what these numbers do not cover. They are cost, not
relevance; the MRR figures quoted beside the BM25 weights and the identifier
shadow column come from a shoot-out over real transcripts and cannot be
reproduced from this repository.
* refactor(ai-vault-search): key a page once per search, and report reachable pages honestly
The page key was hashed twice per search, once to decode the incoming cursor
and once to mint the outgoing one. The benchmark's reachable-page figure was
clamped against a constant that could never bind; it is the limit over the page
size and nothing else.
* test(ai-vault-search): pin the two engine seams nothing was holding
The warm wiring and the query-length cap were both written and neither was
observable. A spy pins that a search is what warms the store, and the cap is
pinned through the one input the planner's own term limit does not already
bound: a single enormous token, where the cut is what decides whether the term
matches the indexed one at all.
* fix(ai-vault-search): fence the generation against writers this process cannot see
The generation was cached in memory and moved only on this store's own writes,
and the bump itself was a read-then-write outside any transaction. Two handles
on one file is the normal case once PR 3 lands: the indexer writes in the
scanner child while an engine reads elsewhere. A reader would see that writer's
deletions while its own generation stood still, honour a stale cursor, and skip
a session; two writers could mint one generation for two snapshots.
`generation` now reads `meta` on every call, and the bump is a single
`ON CONFLICT DO UPDATE ... + 1` inside the transaction that makes the change.
That closes the crash hole the bump-on-open existed for, so opening a store no
longer invalidates anyone's cursor.
Only a change to what a read returns counts. Retiring a path the index never
held hides nothing, and the row deletes that drain a tombstone take away rows
that were already invisible: bumping there would refuse a cursor every 256 rows
and leave pagination unusable for as long as indexing ran.
Also drops redaction from the query log, following PR 2's decision to store
transcript content as written; the module it called no longer exists.
* fix(ai-vault-search): answer from a version-1 index, and keep a repaired literal whole
Two smaller findings.
An engine can be handed a connection to a version-1 file that another handle is
still answering from, and this PR is what first makes that reachable. It now
probes once for the two tables version 2 added and names what it cannot serve
on every result, instead of throwing at the first query that reaches for a
vocabulary that is not there. The route ladder simply skips its repair rung.
Which process may unlink and rebuild the index is PR 3b's decision and is not
solved here.
Typo repair re-planned the corrected query from scratch, and a corrected
spelling can read as prose even when what was typed was a literal:
`parseJsonn(the, data)` has the punctuation, `parsejson the data` does not, so
the re-plan dropped `the` as a stop word. The repaired query searched for less
than was asked and `repairedTerms` reported a body nobody typed. The re-plan is
now told what the original decided, because a repair changes spellings, not the
query's character.
`repairedTerms` is documented as the whole body the repaired plan ran, with the
index's own case-folded spelling for the terms it corrected.
* fix(ai-vault-search): make repo: and path: mean one thing in the list and the index
The engine said `repo:` and `path:` a second time, in SQL, and SQL cannot say
them. LIKE folds ASCII and nothing else, so `path:CAFÉ` missed `café`; the
engine searched `cwd_key` while the panel searches the working directory and
the transcript path, so `path:jsonl` matched every session in the panel and
none in the index; and the engine compared one path segment where the panel
compares the last two, so `repo:orca/session-search` missed. All four
reproduce in both directions.
So there is one definition now, not two that resemble each other.
`matchesAiVaultQueryOperators` moves into the shared filter module beside the
panel that already owned the semantics, the panel calls it, and the engine
applies it over the rows it retrieved. SQL keeps only what it can express
exactly: the `cwd_key` prefix range for `scopePaths`.
`parseVaultQuery` now parses through `splitAiVaultSearchQuery`, so one parser
decides what an operator is. Its existing tests pass unchanged; four degenerate
shapes do answer differently and are pinned as decisions rather than left to be
discovered.
Two consequences worth stating. The operators are conjunctive now, because that
is what the panel has always done, where the engine had been ORing within a
key. And the operator-only page walks newest sessions in bounded pages applying
the predicate, rather than taking one cut of the newest N and filtering it,
which would have answered `repo:x` with nothing on a busy index.
* docs(ai-vault-search): re-measure the query benchmark after the operator change
repo: and path: moved out of SQL, so the operator-only row measures something
different now and the note that it is a range seek was wrong. The rest of the
table is re-measured on an idle machine: the previous run's p95 column was
mostly contention, which is why conversation looked 3.4x faster at p95 rather
than the 1.7x it actually is.
Adds what this PR does not settle: which process may unlink and rebuild the
index is PR 3b's, and PR 4 is only the first thing that makes reading it
reachable.
* fix(ai-vault-search): stop reporting a search that gave up as a search that finished
The operator-only walk stops at a scan ceiling as well as at a full candidate
set, and only the first of those reached the result. A query whose one match
sat past the ceiling came back with no hits and truncated.candidates false,
which is the engine claiming there is nothing to find when what happened is
that it stopped looking. Retrieval now says why it stopped, because it is the
only layer that knows, and the count it used to return could not distinguish
the two cases.
The cursor fence stays as it is: any published read moves the generation, so an
outstanding cursor is refused, and that is what F11 asked for. What was wrong
was the claim next to the row-delete skip that pagination stays usable through
indexing. It does not, and the engine now says so. The rejection carries the
generation the cursor was minted in and the one the index is at, so a caller
can tell a moved index from a bad cursor and re-issue page one without showing
anyone an error.
The capability probe was nearly dead code, since every store opens through a
function that rebuilds a stale file. It is not dead, because two handles can be
open on one file, so the claim is corrected rather than the probe deleted. It
now runs per search: a verdict cached in the constructor is wrong in both
directions once another handle rebuilds the index.
Also says why the row-delete loop may skip the bump: those messages keep
batch_id NULL and stay in visible_messages, so what makes them unreachable is
their session's tombstone, and the read ratchet is what keeps every reader
joining the view that applies it.
* fix(ai-vault-search): restore the panel's reading of a quote that does not end a word
Unifying the two parsers changed panel behaviour on nine of twenty probed
shapes, not the three previously pinned. Six of the nine were regressions, all
from one rule: the shared parser refused a quoted span whose closing quote was
not followed by a space, so `"a b"c` and `repo:"a"b` became single terms
carrying their own quote characters, which match nothing.
The rule was justified as what stops the apostrophes in `it's a repo:orca
thing's` from swallowing the operator between them. It is not: a span only ever
opens at a token start, and the quote in `it's` is not at one. Dropping the
rule restores all six shapes to what the panel has always done and leaves that
protection intact.
Three changes remain and are kept because the old answer was worse in each: an
operator with an empty quoted value is dropped rather than filtering on `""`
and silently emptying the list, and a bare pair of quotes reads as an empty
term rather than as the two characters. Each is pinned with a test that says
which behaviour it is and why.
* fix(ai-vault-search): trim operator values, and report a query the engine had to cut
Three lows.
`repo:" "` survived as a term and matched no label, silently emptying the
list, which is the exact defect the empty-value drop exists to prevent wearing
different clothes. Operator values are trimmed, and a whitespace-only one drops
like an empty one. The substring matcher's copy of a free-text term is trimmed
too, so `" "` reads as the empty term already does; the span kept for FTS is
still the query verbatim.
Two caps upstream of retrieval fired silently: the planner searches at most 48
terms, and the engine cuts the query at 512 characters. A 56-term query whose
only match was the 56th came back with no hits and nothing truncated, which
claims there is nothing to find. `truncated.query` now says when either fired,
alongside the candidate and snippet flags that already did.
Every cursor refusal now carries the generation the index is at, which the
engine knows before it looks at the cursor, and the generation the cursor
claimed wherever that survived parsing. The doc says exactly when each is
present instead of leaving absence unexplained.
* refactor(ai-vault-search): read the tables the simplified index writes, and own the fence
PR 2 deleted the visibility views, the staging tables and the store's
generation, so this reads `sessions` and `messages` directly and carries the
three schema objects only a query needs — the vocabulary, the query log, and
the triggers that move the generation — as its own extension over the store's
schema.
The fence is now three triggers on `files`, because every transaction the store
opens that can change what a search returns writes that table and nothing else
does; retention's orphan drain is the one write path that touches neither, and
it is the one that must not bump. The triggers live in the file, so a writer in
another process moves the generation without knowing a reader exists.
A message row can now outlive its session row until the drain reaches it, so
the snippet read joins `sessions` and the typo repair asks for a live posting
instead of trusting the vocabulary's document count.
The engine takes a connection rather than a store: PR 2's store keeps its
connection private, and which process may open or rebuild the index file is
PR 3b's decision, not a query engine's.
* perf(ai-vault-search): price the second FTS table, and re-measure without warmup
Open decision 3. A column filter over `messages_fts` returns the identical
rowid set as `conversation_fts` — checked here per query rather than assumed —
so the table exists for latency alone. On a 105 MB corpus at both ends of the
tool-output band, the column-filtered form costs 1.16-1.42x at p95, against a
bar of 2x, so the recommendation is to delete it.
The shoot-out writes its own corpus because the answer turns on the one
property the shared generator fixes: how much of a transcript is tool output.
Half the tokens in that output are words the conversation also uses, which is
deliberately generous to the table under question.
The doc records the number that argues the other way. PR 2 priced the table at
about a quarter of the index on a corpus whose tool output is 56% of its
message text; on a tool-heavy one it is 6.7-11%, because `messages_fts` grows
with the tool text and the second table does not.
Page warmup is not re-added. The measurement behind it was on a 4 GB index,
removing it moves this corpus by less than the run-to-run spread, and a
cancellable background pass needs a lifecycle a query library does not have.
* test(ai-vault-search): pin that an append moves the generation a cursor is fenced by
* refactor(ai-vault-search): answer the conversation scope with a column filter
PR 2 deleted `conversation_fts` on the strength of this PR's shoot-out, so the
scope is a column filter over the one FTS table now. `ftsTableFor` is gone; a
scope is a pair of `scopedExpression` and `scopedWeights`, and the table name
no longer travels through the engine, the snippet builder or a hit.
The filter is parenthesised, and that is the whole of it: `{cols}: (a AND b)`
binds both terms, while `{cols}: a AND b` binds only the first and searches
tool output for the rest. A test drives an AND whose second term lives only in
tool output through both scopes.
The snippet keeps one guard, not two. Its column list and its expression were
each hiding the other's mistakes — a tool-only row was unreachable through
either — so the list is the same four columns for every scope and the scoped
expression is what makes a conversation snippet impossible to draw out of tool
output. Dropping it now leaks that row, which a test catches.
One behaviour the deleted table did not have, pinned rather than wished away:
bm25 normalises by the whole row's length and has no per-column length, so two
rows with identical prose score differently when one also holds tool output.
The rowid set is unchanged; the order within it can move.
Re-measured on the shipping schema. The conversation scope is 1.2-1.4x faster
than `all` at every rung, and the index is 57 MB rather than about 150 MB at
93% tool output, because a tool row is now capped at 3,072 characters.
* fix(ai-vault-search): repair a spelling inside the scope that will answer it
Typo repair read `messages_vocab` and probed `messages_fts` with no column
filter, so tool output decided whether a conversation-scoped query was
repaired, in both directions. A tool row carrying the misspelling made the
query look correctly spelled and suppressed the repair; a tool row carrying a
rare word became the suggestion, naming in `repairedTerms` a string from a
column the scope will never show. Both reproduced against a control index that
differs by exactly that one row.
The vocabulary proposes and a scoped count disposes. fts5vocab is per table and
cannot be column-filtered, so every decision that reaches the plan — already
spelled right, eligible, and which of two equally close candidates wins — now
comes from a `messages_fts MATCH` under the same filter retrieval uses, joined
to `sessions`.
That also takes the vocabulary's `doc` out of the ranking, which is the half of
the drain defect that belongs here: `doc` counts rows whose session a purge has
already cut loose, so reclaiming them changed which word a query was repaired
to. Candidates are ordered by term now, because the ordering decides which of
them survive the scan limit, and ties on similarity go to the more common word
counted live rather than to the vocabulary's number.
The cost is one bounded count per candidate examined, at most eight per prefix,
and only for a term the scope has no posting for at all.
* fix(ai-vault-search): fence the rows a purge reclaims after it cuts a session loose
Retention's second half deletes from `messages` alone and touched neither
`files` nor `sessions`, so it moved no generation. The argument was that those
rows answer nothing, which was true of retrieval and not of the engine: the
typo repair's dictionary is a view over the FTS b-tree and listed them, so a
drain running between two pages swapped the repair under a cursor that was
still honoured, and a search that had answered stopped answering.
The commit before this one fixes that at its source by counting live rows. It
does not make the drain provably inert — the vocabulary still decides which
candidates survive its scan limit, and reclaiming a term's last row moves where
that limit cuts — so the fence is what covers the rest.
A fourth trigger, on `messages`, with a `WHEN` clause that is the whole reason
it is affordable: a replace and a `removeFile` delete a session's rows while
its `sessions` row still stands, so neither fires, and both already bump
through `files`. Only the drain deletes a row whose session is gone.
The price is named rather than avoided: a cursor outstanding while a purge runs
is now refused once per batch, which `SessionSearchCursorError` reports as
`stale-generation` so a caller re-issues page one. The test that pinned the old
contract is replaced by one for the new one, and by one proving a replace still
does not fire it.
* fix(ai-vault-search): tell a highlight from a transcript that contains brackets
The snippet builder asked each of a row's four columns for a marked snippet and
showed the first whose text contained `[[`. Transcripts contain `[[`: a bash
`if [[ -f … ]]`, numpy's `[[1, 2], [3, 4]]`. A row matching only in tool output
was shown its user turn instead, with nothing highlighted in it, and the
any-column fallback an identifier-only match depends on was unreachable behind
the same collision.
Whether a column matched is now the difference between two renderings of the
same text: `snippet(…, MARK, MARK, …)` beside `snippet(…, '', '', …)`. Content
cannot forge a difference between those two, because it is the same content
either way.
The marks FTS5 inserts are private-use code points, rewritten to the public
`[[` and `]]` once, at the end. That is for the other decision that has to tell
a mark from content: the truncation refuses to cut between an open mark and its
close, and a transcript's own bracket used to move that cut.
* fix(ai-vault-search): cut a query on a code point and bind ids in batches
Two small ones from the review's not-routed list.
`query.slice(0, 512)` can land between the halves of a surrogate pair, leaving
a lone half that matches nothing and that a caller cannot echo back. The reader
already has `sliceAtCodeUnitLimit` for exactly this.
`loadSessions` bound one parameter per candidate id in a single statement. The
list is as long as the candidate limit, the tuning doc invites a host to raise
that limit, and SQLite's `SQLITE_MAX_VARIABLE_NUMBER` is 999 on builds older
than 3.32 — so one settings change away from `too many SQL variables`. Read in
batches of 500, leaving room for the filter's own bound values.
* docs(ai-vault-search): price the repair rung, and record what is left open
Typo repair is the one rung whose cost tracks the vocabulary rather than the
result, and it only runs for a term the scope has no posting for. Measured over
1.6 M distinct terms: 10 ms for one unknown term, 387 ms for a 480-character
query of thirty-nine of them.
The scoped-count fix made that cheaper rather than dearer, from 737 ms, because
ordering the vocabulary scan by term drops the sort `doc DESC` needed and the
counts it adds are at most eight bounded probes per prefix. A cap on unknown
terms per query is a follow-up in the split plan, with the five other items the
final review raised and did not route.
* test(ai-vault-search): make each snippet mark mechanism answer for itself
Two mechanisms landed together and hid each other: choosing a column by
comparing a marked rendering against an unmarked one, and marking with
private-use code points instead of `[[`. Either alone fixed the bracket repro,
so neither had a mutation against it — the same masking the snippet's two
column guards had a round ago.
They do different jobs, so both stay and each gets the test that needs it. A
transcript holding a private-use code point of its own is what the comparison
is for; agent output carries Nerd Font glyphs from that block. A snippet past
the character ceiling with a bracket after its last real mark is what the
private-use marks are for, because the truncation has to find that mark by
searching the text.
The two one-line fixes get honest framing rather than a mutation neither can
have. A lone surrogate is not a token character, so the planner drops it either
way and the safe cut is hygiene. And no SQLite this stack runs refuses 1,100
bound ids — 32,766 has been the floor since 3.32 — so the batch is about
owning the ceiling here rather than rescuing a reachable failure.
* refactor(ai-vault-search): keep the scope's expression with the other expressions
Making the typo repair ask its questions in the search's own scope put an
import from retrieval into it, and retrieval already owns the repair — a cycle
the native audit catches. `scopedExpression` belongs beside `phraseExpression`,
`andExpression` and `orExpression` anyway: it builds a MATCH expression, and
two callers now need it. The bm25 weights stay in retrieval, where the SQL that
uses them is.
* docs(ai-vault-search): say which delete paths fire the orphan-reclaim trigger after a replace cuts loose
|
||
|
Jinwoo Hong
|
c84007c541 | feat(rpc): generate a shared params catalog from the host registry, gated on parse parity (#19961) | ||
|
Jinwoo Hong
|
2c984bcdaa |
feat(ai-vault-search): session search index as a transcript reader consumer (#19687)
* feat(ai-vault-search): add the session search index schema and row modules
The FTS5 index that PR 2 folds the transcript reader's message stream into:
two FTS tables behind visibility views, the identifier shadow column, the
resumable fork digest, redaction at the row-insert choke point, and the
bounded compaction, warm-up and retention lanes.
Schema version starts at 1: this branch drops the query log and the fts5vocab
table, which the query engine reintroduces with its own bump.
* feat(ai-vault-search): stage index writes and publish them atomically
The writer stages a read's rows against an unpublished session row and a batch
id, then flips the whole read visible in one transaction: both FTS tables are
written together per row, and publish nulls the batch pointer before dropping
the batch so a recycled rowid can never name a later in-flight write.
Buffering replaces the branch's streamed producer. The transcript reader pushes
messages synchronously, so a staged write cannot make it wait; rows are held to
128 of them or 256 KB and then flushed in one transaction, which bounds both the
retained bytes and one stall.
The store owns the database and the set of files the index is behind on. It
carries no query, coverage or scheduling: draining that set is the indexer's.
* feat(ai-vault-search): register the index as a transcript reader consumer
The index keeps its own per-file cursor in the `files` table and never consults
the parse cache; the branch's AsyncLocalStorage capture scope is gone.
Three refusals, each leaving the cursor where it was and recording the file for
a later whole re-read: an append whose predecessor offset is not this index's
own cursor (or whose dev/ino changed) is declined in `beginRead`, a staging
failure ends the read's rows without throwing back at the reader, and an
`incomplete` outcome never publishes. A null session drops the file's rows and
still advances the cursor, because the file was read through.
Nothing in production registers it: the indexer decides when the index is live.
* test(ai-vault-search): measure the index's disk and latency cost
The write benchmark indexes a synthetic corpus through the real reader and
reports rows/s, per-table bytes per transcript MB, write amplification and
rebuild time; the retention benchmark compares the batched purge against a
whole-file delete. The corpus generator ships with them, so the numbers are
reproducible on any host and no real transcript is ever read.
* fix(ai-vault-search): decline a source the message channel cannot reach
An OpenCode SQLite candidate decodes inside a worker, so the reader reports
every read of it as incomplete. Staging a batch first meant one staging session
and one tombstone per scan of every such session, forever. Declining in
`beginRead` also keeps it out of the re-read set, because no re-read helps: the
index cannot cover that source until its capture path lands.
* fix(ai-vault-search): redact a message before it is cut into rows
A credential is a shape, and a shape split across two chunks matches neither
half: an 8000-char boundary landing inside a PEM block or a JWT indexed the key
material in full, identifier shadow terms included. Redaction moves ahead of
chunking, and the row type is branded so only `searchMessageRows` can mint a row
an insert site accepts.
`upsertFile` stops erasing dev/ino when a candidate carries none. A host that
cannot stat identity (SSH, WSL, a degraded Windows stat) was overwriting a
proven identity with NULL, which made the next rename-replace of that path
undetectable.
* test(ai-vault-search): pin each index guard on its own
Three guards were each shadowed by another, so mutating any one of them left
every test green. Driving the store directly separates the writer's own
predecessor-offset check from the consumer's cursor check, and a stubbed store
proves `beginRead` refuses before the writer is ever asked.
`publishable` gets the case it never had: a second read of the same path
publishes first, and the stale stage is tombstoned rather than resurrected. It
covers the overlap the parse file lane normally prevents, so the invalidation
slot being per-path is not load-bearing on its own.
Two ratchets: every provider fingerprint needs a fixture that reaches past
`SECRET_ANCHOR`, and every FTS read in the index modules must subtract staged
rows through a visibility view.
* fix(ai-vault-search): rebuild an index that cannot be trusted, and index retention
Opening self-heals. A file too torn to open, or one whose recovery fails, is
unlinked with its sidecars and reopened once; a second failure throws. Before
this a torn index broke open permanently, even though the index is a cache over
the transcripts and throwing it away costs only a re-scan.
A `meta` table with no readable version row is now `stale`, not `fresh`. Seeding
the current version over it would have kept whatever rows the old schema left.
`fresh` means no `meta` table at all, and no longer triggers a rebuild: the
first open of a new profile was doing create-remove-create.
Recovery moves inside open, because retiring a batch that outlived its writer is
part of opening the index rather than of using it. Its append case gets the test
it never had: 200 rows staged onto a live session and abandoned leave the
published generation intact and nothing else.
Also an index on files(mtime_ms), so the retention pass seeks the expiring end
of the list instead of scanning and sorting it, a pragma read-back test, and a
sessions.file_path comment that says what is actually true of OpenCode.
* fix(ai-vault-search): key a session's cwd the way the sidebar already does
`cwd_key` had its own normalizer, which qualified a WSL cwd with its distro:
`/home/me/repo` was stored as `//wsl/ubuntu/home/me/repo` while the sidebar's
`folderGroupKey` stored `/home/me/repo`. Any later join between an indexed hit
and a sidebar group would have returned nothing for exactly the WSL users the
qualification was meant to help. The key is now the shared normalizer verbatim,
pinned against `folderGroupKey` for POSIX, Windows drive, /mnt/c, both WSL UNC
aliases, a Linux path, and the root.
The collision it guarded against is real: two distros both spell
`/home/me/repo`. It is also a collision every SSH host has, neither key
qualifies for SSH, and the honest fix is a column naming the execution host
rather than a path key that only some hosts spell differently.
`/` now keys as `/` instead of the empty string. An empty key cannot be told
apart from a session with no cwd, and the scope filter builds its child prefix
as `key + '/'`, which would have been `//`.
* test(ai-vault-search): widen the FTS visibility ratchet past whole literals
Concatenating or interpolating a table name hid it from the per-literal scan,
so `'SELECT rowid FROM ' + table` passed. The unit is now the file, the scan
covers src/main and src/relay rather than one directory, and both bypass shapes
are checker unit tests. Exemptions carry a reason and the census fails on one
that has stopped being needed, which is how the schema module lost its.
* fix(ai-vault-search): keep a failed rebuild's real cause, and read per statement
Closing the stale handle before the unlink left `db` dangling: if the unlink or
the reopen threw, the failure path closed it a second time and node:sqlite's
ERR_INVALID_STATE replaced the real cause. Nothing classifies that as worth
retrying, so an index a virus scanner or a second Orca was holding would never
rebuild. The handle is nulled while none is open.
The visibility ratchet moves from the file to the statement. A file is far too
coarse for what it exists to guard: the query module will name both views
somewhere, and that whitelisted every raw read in it. Statements come from
literals and their concatenation chains, split on `;`, and a table name
assembled at runtime counts as a read in any file that names an FTS table. The
roots now cover cli, shared and preload as well.
Half a recorded identity is now stated to be no identity. `remote-session-file-stat`
spreads dev and ino independently and the COALESCE preserves whichever half a
host could prove, so one number cannot tell a rename-replace from a same-file
re-read; comparing it would decline healthy resumes on a coincidence.
`discard` clearing the staging slot gets a test: without it the map grew one
entry per path for the store's life.
* fix(ai-vault-search): give a behind consumer a way to get the read it needs
`markStale` promised a whole re-read that nothing could deliver. The reader
picks append or replace from the session list's resume point, so with an empty
index and a warm parse cache every read arrives as `append`, the consumer
declines every one, and nothing is ever indexed. That is the state on first
enablement inside a running app.
`requestWholeTranscriptRead` in the reader drops that path's resume point, which
is the one lever that changes the next read's mode, and it lives with the cache
that owns it rather than with the consumer that wants it. `takeStale` documents
that its paths need it before a scan is re-dispatched.
Pausing no longer empties the re-read set. `acceptsCandidate` answers whether a
write may start now and was being used for both jobs, so reconfiguring retention
while paused dropped every entry and a declined read during a pause was never
recorded at all. Retention alone prunes; a paused decline is recorded, bounded,
with the oldest dropped and the drops counted, because a set that silently
forgets is worse than one that says it is incomplete.
A read that decodes no session now tombstones its staged rows before the store
schedules cleanup. The cleanup lane reads the tombstone table the moment it is
scheduled, so the old order left that batch on disk until some later write, and
for the last read before a shutdown that is never.
* fix(ai-vault-search): create the directory the index lives in
Nothing made `<userData>/ai-vault-search/`, and SQLite's failure for a missing
parent is `unable to open database file`, which is correctly not classified as
corruption — so the retry never fired and the feature stranded on any profile
that had never held an index.
* refactor(ai-vault-search): store transcript content as written
Decision: the index does not redact. A secret in a transcript is already
plaintext under the user's home directory and is treated as compromised, so the
index is a second copy of content the user already holds, not a new exposure.
The reference agent-session-search products do not redact either. What a snippet
may carry once it leaves this machine is a transport policy and belongs where
the wire is, not in the write path.
Removes the redaction module and its census, the branded row type that existed
to prove redaction had run before an insert, and the two chunk-boundary tests.
`insertSearchMessage` takes a plain chunk again and identifier shadow terms come
off the raw text. `src/main/observability/redactor.ts` goes back to what main
has, so this PR no longer touches it at all.
Chunking and the fork digest are unchanged; the digest always folded raw message
text, ahead of chunking, so its tests hold as written.
Cost: redaction was 14 ms per 10.5 MB of transcript, about 3% of a rebuild and
below the benchmark's run-to-run spread, so the write numbers are unchanged at
roughly 22-25k rows/s and 22-27 MB/s.
* fix(ai-vault-search): hide a tombstoned session's messages, not only its row
`visible_sessions` already subtracted session-keyed tombstones; the message
half filtered on the batch pointer alone. A published row outlives its session
row until the cleanup lane reaches it, so between a `replace` publish and that
drain both generations answered, and a removed file kept answering after its
session was gone.
Both views now subtract the same set, over a partial index so neither read
scans the tombstone table.
* fix(ai-vault-search): drain the rows a read that never published staged
Only `writePublished` and `removeFile` scheduled the cleanup lane. A read that
staged rows and then declined to publish them tombstoned its batch and told
nobody, so 256 rows of a 300-message incomplete read sat in `messages` and both
FTS tables until some unrelated write happened to schedule a pass. Open-time
recovery had the same hole: it wrote the tombstones and drained none.
The abandoned branch now schedules the drain the published branch already got,
and opening schedules one pass for what recovery just tombstoned. Recovery no
longer adds a batch tombstone when the session-keyed one already covers those
rows, which it did once more on every reopen.
* fix(ai-vault-search): continue the cursor of a file that decoded no session
A read that decodes no session — an excluded Codex worker transcript — advances
the cursor and leaves `files.session_row_id` null. Every later append was then
declined, so a file that only ever grows would be re-read whole on every pass
for the rest of its life.
An append now only has to continue this index's own cursor; it creates the
session row when there is none to resume. That also collapses the `append` flag,
which meant both "resume this session" and "do not own it", into the resumed row
id itself.
* refactor(ai-vault-search): drop the index path module nothing calls
No caller in this PR or the two that follow it: the composition root that would
capture the userData path is PR 3b's.
* refactor(ai-vault-search): read the schema version as stale or not
Only the stale answer was ever acted on; the fresh/current split named two ways
of being fine.
* fix(ai-vault-search): read the resumed session id off an optional row
* refactor(ai-vault-search): leave page warmup and the freshness check to PR 4
`warm()` and `session-search-page-warmup.ts` have their first caller in the
query engine, which is what knows which pages are worth warming; the module
itself went with the previous commit. `isSessionSearchFileCurrent` has its first
caller in the reconciler. `session-search-file-cursor.ts` stays, because
`fileIdentity` and both its types are load-bearing here.
* refactor(ai-vault-search): stop pruning the re-read set on a retention change
The prune walked the whole set to drop what the next `beginRead` would refuse
anyway: `acceptsCandidate` applies the window when the re-read is dispatched,
and `markStale` applies it again before recording anything. The bound stays.
* fix(ai-vault-search): keep a batch tombstone from outliving its batch row
Draining a session-keyed tombstone deletes the session's batch rows, but left
any batch-keyed tombstone naming them in place. `search_write_batches` empties
on every publish, so ids restart low and the next read takes the freed rowid;
the stale tombstone then deleted that live batch's staged rows and the session
published missing messages, with no re-read to fill the gap.
The session drain now clears those tombstones in the same transaction.
* refactor(ai-vault-search): commit a file's rows and its cursor in one transaction
The staged-publish model is replaced by one SQLite transaction per file in WAL
mode. A read buffers its decoded rows and writes them, its session and its
cursor together; a reader on another handle sees the last committed state, which
is the "never a torn session" guarantee the staging machinery was built to
provide. A crash rolls the whole file back and it is re-read.
Gone with it: `search_write_batches`, `search_pending_deletes`,
`messages.batch_id`, `sessions.index_ready`, both `visible_*` views, the
open-time recovery pass, the cleanup lane and `settled()`, and the ratchet test
that made every query site read through a view. Rounds 2 through 6 were all
seams between those pieces.
A file whose rows exceed `SESSION_SEARCH_COMMIT_CHARS` is cut into chunks. The
reader only hands out a byte offset when a read finishes, so a chunk records one
no append can continue from: its rows answer searches as a coherent prefix of
the session, and the next whole read replaces them.
Retention keeps no record of unfinished work. It cuts a session loose from its
file in one small transaction, which is what stops it answering, then reclaims
its rows in bounded batches and hands the freed pages back as it goes. Rows
whose session row is gone are the record of what an interrupted purge left.
Deleted for want of a caller in PR 2, 3 or 4: the WAL budget (a buffered write
has no staging window to grow one across), the compaction module (folded into
the drain), the `sessions_content_hash` index (fork folding runs in JS over rows
PR 4 already holds), `lastWriteAt`, `failures`, `SessionSearchStoreOptions`,
`openStageCount` and `chunkMessageText`.
* test(ai-vault-search): price a per-file commit and the ceiling that bounds it
The write benchmark now times every transaction, because with one per file that
is the whole stall a file costs. A second phase indexes a single synthetic
100 MB transcript, which is what the commit ceiling is chosen against.
* fix(ai-vault-search): stop a fenced write reopening a transaction per message
A chunk write that finds the file record moved under it — a `removeFile`, or an
overlapping read of the same path — can never land anything afterwards. It kept
buffering, so every remaining message re-entered `BEGIN IMMEDIATE` only to roll
back, once per message for the rest of a file that may be a hundred megabytes.
It now stops at the first refusal and drops what it holds.
* fix(ai-vault-search): never hand a live session the id of a purged one
`sessions.id` was a plain rowid alias, so SQLite reissued it as max+1. That id
names rows in `messages` for far longer than the row itself lives: retention
cuts a session loose in one transaction and reclaims its messages over many, and
a session created inside that window was handed a freed id and adopted whatever
of the purged conversation the drain had not reached. The drain then skipped
those rows for good, because their session row exists again, and a search
answered for a purged transcript under a live session's name.
Reachable with no crash at all: an append of a growing transcript the parser
decoded no session from creates its session row mid-purge. AUTOINCREMENT is the
class fix; the schema version goes to 3 so a file written by an earlier commit
of this branch rebuilds rather than keeping a plain-rowid table under a
`CREATE TABLE IF NOT EXISTS`. `messages.id` was checked and is not exposed to
the same window: a row and its two FTS entries always go in one transaction.
Two smaller ones in the same pass. `removeFile` did not fence a read of a path
this index had never written: `current()` compared a cursor, and on an unknown
path the absent row and the absent expectation are both undefined, so the write
recreated the source after its owner had proven it gone. The writer now counts
removals per path and a write compares that count, which is a positive fence
rather than an inferred one. And `drainOrphanedMessages`,
`CONTENT_HASH_MESSAGE_LIMIT` and `CONTENT_HASH_MIN_MESSAGES` are no longer
exported: nothing outside their own modules reads them.
* fix(ai-vault-search): report a half-written file as held, not as unknown
`indexedFile` returned null for a file a chunked read left partway through, the
same answer it gives for a path the index has never seen. A caller asking "do
you hold this file" therefore read a half-written one as new, asked for whatever
read the parse cache offered, and the reader picked append — which the consumer
then declined. Only the stale set healed it, a cycle later.
It now reports the record with a null `byteOffset`, and
`requiresWholeRead(indexed)` names that state. Null rather than an added flag
because the mtime and size on that record are the file's real ones: a freshness
check comparing only those would call a half-written file current, and a boolean
is easy to not read, while every site that does arithmetic on the offset has to
say what null means at compile time.
The test also found the writer accepting an append that passed the partial
sentinel back as its predecessor offset. Nothing in the reader produces a
negative offset, but the sentinel is not a byte offset and no caller should be
able to continue it; `beginWrite` now refuses it outright.
* fix(ai-vault-search): cut a chunk at whitespace, never mid-token
The 8,000-char chunker backed up only to a newline, and only when one sat in
the second half of the window. A wrapped paragraph, a CJK transcript or a
minified log has no newline there, so the cut landed inside whatever word
straddled the target: the user's term was filed as two halves and matched
neither. It now backs up to the last Unicode whitespace in the second half,
and falls through to the target when there is none, because 4,000 characters
without a space is not a word.
A phrase that straddles a chunk boundary is still not matched. Chunks are
separate FTS rows and FTS5 cannot span them; that is stated at the chunker.
* feat(ai-vault-search): cap an indexed tool row at 3,072 characters
Tool output is 80-97 % of a transcript's bytes and a single message may be a
quarter of a megabyte, so without a cap the index, the buffer a read holds and
the transaction it commits are all sized by how much a tool printed rather
than by how much is worth searching. A `tool` message now becomes one row of
at most 3,072 characters, kept from the head, where the command and the first
lines of its output are. User and assistant text is never capped.
Measured on the write benchmark, 40 sessions with tool output at 95 % of
message text (the real band), per transcript MB: index 1,334,126 -> 353,973
bytes, write amplification 1.27x -> 0.34x, rows 18,804 -> 9,600, rebuild
1,676 -> 379 ms, largest transaction 49.7 -> 12.1 ms. On the default corpus,
whose tool results are 1.4 KB and so under the cap, nothing changes at all:
2,167,887 bytes per transcript MB on both arms.
The corpus generator takes `toolResultWords` and the benchmark reads
ORCA_SEARCH_BENCH_TOOL_WORDS so both arms are the same harness.
* fix(ai-vault-search): check the commit ceiling per row, not per message
The buffered-chars check ran after a whole message had been folded into rows,
so a single message could carry a transaction as far past the ceiling as it
was large. A conversation turn is one message and can be megabytes; the
ceiling exists to bound how long one commit holds the process and how large a
WAL it produces, and neither bound survived a message that overshot it.
* style(ai-vault-search): undo a stray reformat of untouched assertions
Three assertions in the file-write test were expanded by a formatter run that
was not the repo's, and the expansion survived because a multi-line object
literal is preserved once it exists. Restored to what they were.
* feat(ai-vault-search): write a session's identity with its first commit
A chunked read created its session row with an empty session id, an empty
title, an empty resume command and null cwd and timestamps, and only filled
them in on the final commit. Those chunks answer searches the moment they
land, so until the read ended every hit they produced named a session nothing
could identify — and a crash between chunks left it that way for good, since
the re-read that heals it is the same read starting over.
The reader already knows the id, cwd and timestamps from a transcript's
opening lines; it just had nowhere to put them. `TranscriptReadStart` now
carries an optional `identity()` the consumer calls during the read, backed by
an optional `identity()` on `ResumableSessionParseState`: one line in the
shared accumulator fold (which covers cursor, copilot, droid, gemini,
antigravity and the graph parsers) and one each in the Claude and Codex folds,
which keep their own state. A chunk commit writes what it returns; the final
commit overwrites it from the decoded session, so the mid-read title stays
provisional.
Two `cwd` guards in the Codex fold collapse into the `?? ` form the `branch`
line beside one of them already used. `extractString` never returns an empty
string, so it is the same assignment in one line instead of four, and it is
what keeps the file under the 300-line cap with the identity accessor added.
* refactor(ai-vault-search): drop conversation_fts for a column filter
The second FTS table existed for query latency alone: a column-filtered
`messages_fts MATCH '{user_text assistant_text}: q'` returns the identical
rowid set, which PR 2's round 5 review proved and PR 4's benchmark re-checked
per query. PR 4 then measured the filter at 1.16-1.36x the p95 of the second
table on a 105 MB corpus across two points in the tool-output band, under the
2x bar the decision was set at, and closes the stack's open decision 3.
It cost a quarter of the index on a corpus whose tool output is half the
message text. With the tool-row cap it is a smaller saving than the decision
was framed around, but it is still a table, a write per conversational row and
a delete per reclaimed one.
Schema version 4: `CREATE TABLE IF NOT EXISTS` is a no-op over an existing
index, so only the bump makes a file that still carries the table go.
* feat(ai-vault-search): expose the index handle a composed reader queries
PR 4's engine reads the index this store owns. One getter lets it compose over
the store's handle instead of opening a second connection to the same file,
and it carries the two rules this PR measured: never hold a read transaction
across an await, and no `.iterate()` outliving its statement. Either pins a
read snapshot, and a checkpoint cannot pass one, so the WAL grows without
bound for as long as it is held (10 MB to 266 MB on the write benchmark).
* fix(ai-vault-search): cut a chunk at punctuation, not only whitespace
The 8,000-character chunker backed up to the last whitespace in the second
half of the window and fell through to the target when it found none. A
minified tool result has none to find: valid minified JSON runs past 8,000
characters without a space, so the cut landed inside whatever word straddled
the target, and a search for `pericardium` matched neither `perica` nor
`rdium`. That is the shape most likely to be chunked in the first place,
because tool output is 80-97 % of a transcript's bytes.
The backoff now takes any character that cannot be inside a token — anything
outside a letter, digit or underscore — and still falls through to the target
when the window holds none, because 4,000 characters without one is not a
word. Surrogates are excluded from the class so a cut never lands between the
halves of an astral character.
A few of the tokenizer's own `tokenchars` (`. - / +`) are cut on even though
unicode61 keeps them inside a token. That costs the joined form of a path, and
only in a window with no whitespace anywhere, which is a far smaller loss than
the torn word this exists to prevent.
* fix(ai-vault-search): never chunk a read that cannot name its session
`updateProvisionalSession` is fed by the resumable readers alone. Claude and
Codex carry an `identity()` off their fold; `readWholeTranscript` supplies
none, because a format rewritten in place has no resumable state to ask. So a
Grok, Cursor, Gemini or OpenCode file large enough to pass the commit ceiling
published its chunks under a session with an empty id, an empty title and a
null cwd — rows that answer searches at once and that an interrupted read
leaves behind for good, since the re-read that heals them is the same read
starting over.
`add` now commits a chunk only while the read can name what it is writing. A
read with no identity, or one whose parser has decoded no id yet, keeps
buffering and commits whole at `finish`. The whole-file formats are the ones
with nothing to give and they are small — the largest on this machine is 5 MB
— so buffering one to the end costs nothing, and chunking stays reserved for
the readers that can say which session a prefix belongs to.
`updateProvisionalSession` takes a non-null identity now, so the invariant is
the type rather than a guard that silently wrote nothing.
* perf(ai-vault-search): cut a replaced generation loose instead of deleting it
The first transaction of a replace deleted every row of the old session before
inserting its own bounded chunk, so the transaction was sized by the history
being replaced rather than by the rows being written. On the synthetic 100 MB
transcript the longest replace transaction was 1,255-1,283 ms against
668-672 ms for the same file read fresh, and the gap grows with the session.
A replace now mints a new session row, points the `files` row at it and
deletes the one old `sessions` row, all in the same transaction. Every
retrieval joins `sessions`, so the old generation stops answering the moment
that commits — the same thing retention's first transaction does — and its
messages are reclaimed afterwards by `drainOrphanedMessages`, the bounded
batch loop that already exists for exactly this set. `sessions.id` is
AUTOINCREMENT, so the freed id is never handed to another session while those
rows still name it.
The longest replace transaction is now 746-848 ms, the same band as a fresh
read. The trade is stated plainly: the pass does more total work, 3.7 s
against 3.1 s, because the reclaim is 360 bounded transactions with an
incremental_vacuum step each instead of one large delete. It yields between
them, so none of it holds the process, and returning the pages per batch also
takes the index from 173.6 MB to 162.0 MB.
The store schedules the drain off the committing stack: an async function runs
synchronously to its first `await`, so calling it inline would put the first
batch back inside the transaction's own call. One drain at a time, with a
repeat flag for a replace that commits while one is running.
* fix(ai-vault-search): preserve search tokens and index Codex tools
* fix(ai-vault-search): match SQLite marks and Codex file-change records
* fix(ai-vault-search): preserve stat pairs and validate benchmark results
|
||
|
254a07bb05 |
fix(release): port three release-gate fixes to main so cuts stop re-inheriting them (#19945)
* fix(release): prune optional natives before the linux arch floor check
The linux-arm64 release build failed packaging, and retried three times:
[verify-linux-glibc-floor] 1 bundled native binary is built for the
wrong architecture (target arm64):
resources/node_modules/@parcel/watcher-linux-x64-glibc/watcher.node
is x64, expected arm64 (from its own path)
`afterPack` ran `verifyLinuxGlibcFloor` at its top, before
`prunePackagedRuntimeNodeModules`. A cross-build intentionally installs
every optional native variant, so at that point the arm64 slice still
carries the x64 `@parcel/watcher` package that the prune exists to drop.
The check was reading a file that was never going to ship.
Move the floor check below the prune so it inspects the binaries actually
packed. Same ordering as
|
||
|
Jinwoo Hong
|
4b1b7178ad |
fix(orchestration): scope @ group addresses to the sender's Run (#19783)
* fix(orchestration): scope @ group addresses to the sender's Run `@all`, `@idle`, and the agent-name groups (`@claude`, `@codex`, ...) resolved against every terminal on the host. A coordinator meaning "my three reviewers" reached 126 agents across every open project, twice in one day, and every unrelated agent burned a turn discarding mail that was never for it. Every group except `@worktree:<id>` now means the live Dispatches of the sender's own Run, each addressed as `dispatch:<id>` so delivery is durable even when the worker terminal is not attached yet. A sender bound to no Run is refused with `invalid_argument` naming `run:<id>` / `dispatch:<id>`; there is no host-wide fallback and the host's terminals are never enumerated for it. `@idle` and the agent-name groups filter within that set by the same terminal status and host-resolved identity as before. `ask --to @group` returns the same code and points at the owning Run mailbox. Federated Dispatches read relayed control mail rather than a local mailbox, so a Run-scoped fan-out skips them with a `recipient_unreachable` warning naming the direct `dispatch:<id>` address. Group addresses are resolved host-side, so no RPC or stream shape changes; an older CLI sending `@all` to a new host gets the Run-scoped meaning. Claude-Session: run-scoped-group-addresses * fix(orchestration): revalidate legacy takeover before the recipient verdict A legacy coordinator taken over while `listTerminals` was in flight reported `runtime_error` instead of `legacy_read_only`: Run scoping made "no live Dispatch in this Run" the first thing the group send could fail on, and that threw before the takeover check ran. Takeover is a precondition, not a commit-time detail — the sender must be told it is read-only whatever else is wrong with its recipient set. Revalidation moves to immediately after the only `await` in the path. Everything below it is synchronous, so the commit-time window it used to guard is unchanged; only the error paths now see it. The legacy partition test gave `term_current_worker` no Dispatch, so under Run scoping it is correctly not a recipient. It now holds a real current-contract Dispatch in the same adopted Run, which is what the test is named for: one `legacy_direct` and one `current_delivery` recipient in one fan-out. Claude-Session: run-scoped-group-addresses * fix(orchestration): address the Run a nested coordinator created, not its parent A nested coordinator is both a worker of its parent Run and the coordinator of the Run it created. `resolveMessageRun` answers with the parent, correctly, because that is where its own `worker_done` belongs — but audience is a different question. Scoping `@all` to that Run sent a nested coordinator's "shared context" to the siblings it was started beside instead of the workers it started, and reported success, so it never learned its sub-workers heard nothing. Before Run scoping the host-wide fan-out reached the sub-workers by accident; this turned an over-broad delivery into a wrong-audience one, the exact failure class the change exists to remove. Group audience now resolves off the Run the sender coordinates, falling back to its Dispatch's Run. A leaf worker coordinates nothing and is unaffected. This is a separate question from `routing.run`, not a second answer to the same one, so `resolveMessageRun` keeps its meaning for point-to-point mail. Also: when every live Dispatch in a Run is federated, the fan-out skipped them all and threw a bare `Error` that discarded the warnings naming those remote workers and how to address each one. The sender was told "no recipients" while three remote workers existed. That throw now carries a code and the skip explanations. Claude-Session: run-scoped-group-addresses * docs(orchestration): say that no group address reaches a coordinator A coordinator is not a Dispatch, so Run-scoped groups never include one. That follows from the rule, but nothing said it, and the old host-wide meaning did include the coordinator — a worker sending `@all` to raise a blocker would be heard by its siblings and by nobody who can act. The guide, the CLI note, and the docs page now say to use `run:<id>` for that, and that a worker which created its own Run addresses that Run's workers. Also restores the `@cursor` case dropped when the group tests moved: a Claude pane titled "Fix the text cursor blink" must not receive Cursor's mail. That hazard was recorded from real titles and `@droid` alone did not cover it. Claude-Session: run-scoped-group-addresses * fix(orchestration): preserve group audience and mailbox identity * fix(orchestration): validate group scope before dispatch routing * fix(orchestration): preserve pane identity and exclude coordinator dispatches |
||
|
Brennan BensonandMerge Sim
|
7197593e31 |
fix(lint): preserve deliberate collator benchmark baselines (#19686)
Co-authored-by: Merge Sim <sim@local> |
||
|
Brennan BensonandMerge Sim
|
72befaf360 |
feat(native-chat): show Claude subagent activity on the shared carrier (#18806)
* feat(native-chat): show Claude subagent activity on the shared carrier Claude's `message:system:task_*` frames are classified `status-chrome` and reach the transcript as nothing at all, so a turn that spawns subagents renders as an idle turn. The journal translator now reads them into the shared subagent-group carrier — no new UI, and the frames stay `status-chrome` so nothing prints a raw opcode row. `local_agent`, `local_workflow` and `local_bash` tasks share that channel and all carry a `tool_use_id`, so `task_type` is the discriminator and a backgrounded `sleep 20` stays out of the roster; `subagent_type` covers releases that predate `task_type`. `skip_transcript` tasks never render, `is_backgrounded` children survive the turn-end sweep, and a resumed task re-announced under a fresh tool id is aliased onto its `task_id` rather than duplicated. A child still reported as working when the turn — or the session — ends becomes `unverifiable`: contact was lost, which is not evidence it exited. * fix(native-chat): stop the Claude subagent roster dropping its own rows The roster published under the same coalescing key it appends the row with, and the sink queue replaces any queued operation sharing a key regardless of kind: once a write was in flight, each new append evicted the pending publish and the next publish evicted that append, so the body never reached the journal and `lastSerialized` had already moved past it. Publish now takes the sink's own slot, as the Codex streams do. A tombstoned row could never come back: the non-batch item-row builder derived its revision from `items` alone, so a re-add was built at revision 1 against a tombstone at 2 and the reducer discarded it forever. It now takes the same `max(items, tombstones)` the batch builder already used — reachable here because an announcement that reveals a `local_bash` task empties and tombstones the group row that a genuine subagent later in the turn reuses. `settleTurn` swept whatever group the key named at the time it ran, so children rostered before any turn key existed were never swept, and a turn whose result never arrives was left working forever. The ending turn's key is now an argument, a superseding turn start settles the turn it replaces, and every turn end also sweeps the outside-turn group. Teardown without an `ended` event, and eviction past the group bound, both lose contact instead of stranding a row at `working`. Label ordinals are a high-water mark now: releasing one on a re-label handed the next child an ordinal that was already on screen. * fix(native-chat): bound subagent-group blocks on every wire that carries one Adding a fifth arm to `NativeChatBlock` made every consumer that assumed four wrong. Two of them ended in `return block`, so they compiled while handing a roster straight through: the mobile RPC sanitizer shipped it unclipped past both mobile char caps, and the legacy transcript import stored an untrusted roster unbounded. Both now clip each label and cap the entry count the way they bound their other blocks. The remaining three sites did not compile at all. The worker transcript payload and the live-session benchmark get real arms rather than casts — a cast would have turned the transcript one into a third silent passthrough inside the wire byte budget — and the CLI worker output renders a roster with its shared summary instead of `[image omitted]`. The mobile sanitizer moves to a sibling module beside the image-block one: the file sat exactly on the max-lines bound, and the block bounds are a self-contained concern with their own caps. Also caps the roster's `subagent_type` label fallback, which reached the journal uncapped, and covers the new block type in the schema audit. * test(native-chat): cover the capped subagent_type label The roster stores the frame's label verbatim, so the cap on the `subagent_type` fallback is the only thing bounding it. * fix(native-chat): type the roster fixture so the suite typechecks The mobile-cap test built its entries with an inferred `state: string`, which is not a `NativeChatSubagentState` — the only typecheck failure on the branch. * fix(native-chat): stop child traffic rostering an id Claude never announced `observeChildActivity` minted a provisional row for any `parent_tool_use_id` outside the excluded set. An id that was never announced is never excluded, so a nested Task, a workflow child, or a grandchild parented to a tool id inside the sidechain each produced a permanently unlabelled `subagent` row that could only ever end `unverifiable`. The bounded exclusion set cannot cover an id no frame ever declared, and in a long session it can forget a genuine exclusion. Track instead whether this CLI announces tasks at all — set by ANY `task_started`, including one the subagent filter rejects. Once it has, an undeclared child is provably not a new subagent, so no row is created. The provisional path now serves only releases that announce no task frames, which is what its comment already said it was for. The label-ordinal test moves to an announcement-driven removal, the scenario that path now actually reaches; it still fails if `remove` releases the ordinal. * fix(native-chat): outrank the tombstone when building one too `buildJournalTombstoneRow` still built its revision from `items` alone, leaving it asymmetric with the item builder. It is correct today only because `upsertItem` clears the tombstone whenever a re-add wins — an invariant that lives in the reducer and was not pinned. Apply the same `Math.max`, and pin the invariant so the reducer cannot drop it silently. * fix(native-chat): stop an unrelated turn end settling an outside-turn child `settleTurn` swept the `outside-turn` group on every turn end, so a child Claude announced while no turn was live — a frame trailing the previous turn's result, or one that arrives before the first turn starts — was marked `unverifiable` by the next, unrelated turn ending. That state is terminal and latches, so the `task_updated: completed` that followed was discarded: loss of contact was recorded as the child's outcome on evidence that was never about that child. A turn end now sweeps exactly the group its key names. `outside-turn` belongs to no turn, so only an end with no key of its own reaches it, and what no turn end reaches `settleSession` does — reliably, since teardown without an `ended` event also routes through it. The cost is a child outside every turn showing `working` a little longer; the alternative prints a wrong outcome that nothing can revise. Also pins that a subagent announced after a task the filter rejected still rosters: the announcement path was never what the child-traffic gate closes. * fix(native-chat): bound a subagent entry's id, not just its label Every site that bounds a `subagent-group` block clipped the label and handed the id through whole. From the Claude producer the id is bounded upstream, but the legacy transcript import reads an untrusted file, so an oversized id survived into the journal and then out to every wire that replays it — 64 entries of it, since only the entry count was capped. Each site now clips the id with the helper it already uses for its other bounded fields: the journal's inline-text bound on import, the transcript payload's metadata clip, and the mobile char cap (renamed, since it is no longer a label-only cap). * fix(native-chat): surface an adverse subagent outcome in the fallback sentence The roster row's plain-text stand-in counted only `working`, so a fan-out whose children all latched `unverifiable` (or `failed`, or `stopped`) rendered as "Ran 3 subagents" — a completion claim. Mobile and paired web have no roster renderer, so that write-time-frozen sentence is the entire row there, and collapsing `unverifiable` into something that reads like success is exactly what the SSH execution boundary forbids. It now appends the worst adverse count, worst-first across failed/stopped/ unverifiable, and shows it even while siblings still work — matching the Codex lane's shared `subagentGroupFallbackText` verbatim so collapsing the two copies later is a deletion, not a behaviour change. Also bounds the provisional entry id. `observeChildActivity` wrote the `parent_tool_use_id` straight into the entry's durable id with no length cap, while the announced path already rejects an over-long id via `claudeTaskId`. Both now share `isBoundedClaudeTaskId`, and the provisional path rejects rather than truncates, as the announced one does. * fix(native-chat): stop a subagent label ordinal and a clipped roster key colliding - claimLabel probes the labels the group actually rendered instead of a per-base counter, so a generated `Audit 2` cannot duplicate a provider's own `Audit 2`. - Bound `NativeChatSubagentEntry.id` with a head plus a digest of the whole id at every site that bounds it. The id is the roster key: a prefix clip merged two distinct children onto one entry. - Correct a stale journal-reducer test comment: tombstone cleanup is a map-state invariant, no longer load-bearing for revision ordering. * fix(claude): merge duplicate unhandled-provider-frame imports * fix(claude): preserve subagent lifecycle and bounded invocation identity --------- Co-authored-by: Merge Sim <sim@local> |
||
|
Jinwoo Hong
|
98fdbc4ade | fix(orchestration): file federated worker mail under the coordinator Run (#19542) | ||
|
Neil
|
e182930670 |
test: cover input in five simultaneously flooding SSH panes (#19071)
* test: cover keyboard input in five simultaneously flooding SSH panes * test: capture pane focus and buffers on flood input failure * test: capture pane focus and buffers on flood input failure * test: capture pane focus and buffers on flood input failure * test: record replay input loss and application fix dependency * test: record merged replay-input fix in the five-pane flood gate |
||
 
|
aeddfa463d |
perf(renderer): avoid per-second spinner animation events (#19407)
* perf(renderer): avoid per-second spinner animation events * fix(bench): ensure the Electron runtime before bench:spinners The script launches Electron via Playwright but skipped ensure:electron-runtime, which every other Electron-launching bench script runs first. * docs(renderer): scope spinner pixel-tolerance claim to paused-animation checks --------- Co-authored-by: m4air <m4air@m4airs-MacBook-Air.local> Co-authored-by: pullfrog[bot] <226033991+pullfrog[bot]@users.noreply.github.com> |
||
|
Neil
|
cbc7bb418c |
fix(ci): read the changed-path list past the first pipe buffer (#19409)
`pr-code-change-scope.mjs` read stdin with `readFileSync(0, 'utf8')`, a single read of fd 0. Once the writer outgrows the 64 KB pipe buffer that read returns early or throws EAGAIN, the script exits 0 having emitted no `name=value` pairs, and `tee -a "$GITHUB_OUTPUT"` records nothing -- so every lane the classifier gates is silently skipped rather than failing loudly. A PR opened long ago carries a stale `pull_request.base.sha`, so the gate's merge-base diff spans the whole base branch. PR #13178 diffed 13,294 files (773 KB) against a base 1,592 commits behind main and lost typecheck, test, static analysis, xterm patch sync, package and e2e to this. Stream stdin instead, matching how the sibling `pr-e2e-source-routing.mjs` already reads the same list in the same workflow. |
||
|
Jinwoo Hong
|
c37413271e |
perf(mobile): open a session with parallel startup RPCs and a pre-warmed terminal engine (#19260)
Startup RPCs now fan out in parallel and the xterm engine pre-warms inside the real terminal frame while they are in flight, so the first pane inherits a warm WebView and an already-measured viewport instead of paying a round trip for it. The pre-warm opens its engine before measuring: web-ready only reports that the bundle loaded, and the WebView answers a measure with null until a terminal exists. It also pre-warms at the user's saved text size, because cell size is what the frame height gets divided by. Host writes such as worktree.activate wait for an evaluated status.get reply. Navigation still fails open when a host cannot answer one, but that fallback no longer reads as a passing compatibility verdict. |
||
|
a899f92402 |
feat(windows): enable structured Codex chat on native Windows (#18519)
* feat(native-chat): enable Windows structured sessions
* fix(codex): prove native Windows process identity
* style(codex): format Windows session seam
* fix Windows structured Codex admission
* fix(windows): reprobe missing process identity capability
* fix(windows): decide folder-workspace WSL routing before the click
Review found pathUsesWslUnc exported but unused, and the folder composer
hardcoding worktreeUsesWslPath:false. Together those meant a folder picked
under a \\wsl.localhost\ parent routed to structured chat, then got refused
by the host and fell back AFTER the click -- which defeats the lane's own
design goal that create cannot fail after the click.
The group's parentPath is in scope at submit and the workspace is created
under it, so the parent decides WSL-ness pre-click. Wires pathUsesWslUnc
there and adds tests for the helper, including the unhydrated-store case
that previously threw.
* fix(windows): collapse the gate derivation to one call, restoring max-lines
CI static analysis failed: launch-agent-in-new-tab.ts crossed the 300-line
oxlint ceiling. Adding a max-lines disable is forbidden, so the two gate
derivations collapse into one readWindowsStructuredGateInputs() call --
a store-backed site now adds one line and one import name instead of two.
Better shape anyway: one derivation entry point rather than two reads a
call site must remember to pair.
* fix(windows): engage the legacy fallback when the host THROWS a refusal
Review found a P1 this merge composes: neither parent could reach it. At the
lane head the only structured entry was launch-agent-in-new-tab (full
store-backed WSL check); on main all win32 was refused. The merge enables
win32 in creation flows that pass no projectRuntime, so a WSL folder
workspace, a WSL-configured repo, or a repair-required runtime now routes
structured -- and the host refuses correctly, but by THROWING rather than
returning {ok:false, refusal}.
Callers engage their legacy-terminal fallback on the refusal CLASS, so an
unmapped throw arrives as a generic RPC rejection: no fallback, empty
workspace, error toast, prompt stranded in the launch outbox. Pre-merge the
same action opened a legacy terminal agent.
Map the host's thrown definitive refusals onto the refusal class at the
launch boundary, so every creation flow -- present and future -- degrades to
the legacy terminal instead of stranding. Narrow predicate: unrelated
failures (ECONNRESET, empty message, non-Error) still propagate untouched.
Ablation-proven: removing the mapping reddens the fallback test.
* fix(windows): teach the mobile RPC double the status probe the lane added
CI's first-ever run on this lane caught a pre-existing lane defect. The lane
changed status.get to resolve through
runtime.getStatusAfterWindowsProcessStartTimeProbe(), but never taught the
mobile-surface runtime double about it, so status.get failed for mobile
clients with "not a function". The lane's own test list did not include this
file and the lane had zero CI, so nothing ever ran it.
The real runtime always implements the method; the double omitted it.
* chore: merge current main and regenerate the localization runtime catalog
CI static analysis failed on a stale en-runtime-required.json: main added
onboarding integration-capability keys, and the generated catalog is checked
against the PR MERGE result, not the branch alone -- so it read clean locally
while failing in CI. Merging current main (
|
||
|
Neil
|
314506003a |
fix: retain MSYS shell descendants in their terminal job (#19068)
* fix: retain MSYS shell descendants in their terminal job * test: complete MSYS regression CI registration and teardown contract * fix(windows): deny job breakaway for the whole Cygwin/MSYS shell family The per-PTY job probed only msys-2.0.dll, and only for bash.exe/sh.exe. Cygwin ships the same spawn.cc breakaway logic under cygwin1.dll, and an MSYS2 zsh escapes exactly like its bash does, so both kept the orphan bug. Probe the runtime DLL on the shell's own search path instead of matching shell names: that is the property that decides whether the runtime will ask for CREATE_BREAKAWAY_FROM_JOB, and it drops the name special-casing. * chore(patch): restore the conpty.cc index line The earlier hand-edit dropped it while every sibling section kept one. Recomputed against the real blobs: applying this patch to 7b286d3d yields exactly 4b06d185, so git apply -3 has its fallback back. |
||
|
Jinwoo Hong
|
fb322046e8 |
skills: rewrite and trim the seven non-orchestration guides (#19128)
* skills: rewrite the seven non-orchestration guides to one outcome-first standard
Every guide leads with Result / Done / Safe failure, states conditions instead of case lists, keeps one done bar and one autonomy envelope, and loads references at the point of use via `skills get <topic> --full`. orca-cli drops from 424 to 260 always-loaded lines with three references; orca-per-workspace-env from 794 to 397 with five.
Defects fixed in shipped guides: `emulator camera` (no such command), iOS `permissions` (backend refuses it), Android pane described as in development, `relayGracePeriodSeconds: 0` documented as immediate teardown (it is unbounded), doctor `ok: true` hiding `warn`, an SSH exemplar setting both `jumpHost` and `proxyCommand`, a provisioned-root fetch from `origin`, and the Linear unconfirmed-write rule keyed on four verbs when ten emit it.
The resolver ladder, placeholder rule, and older-binary fallback shared by every installable SKILL.md now come from one skill-stubs/_shared/cli-resolution.md fragment composed by the generator, which also bundles per-guide references into --full. New guards: every ORCA invocation and flag resolves against COMMAND_SPECS, descriptions carry no angle-bracket tokens, reference routing is checked both ways, and an always-loaded size ratchet (300 lines) that guides may leave but never join.
* skills: address review on the SSH recipe and the parity guard
- ssh-host create script: route the bootstrap ssh through the chosen jump host or proxy command, refuse both at once, use StrictHostKeyChecking=accept-new instead of a blind ssh-keyscan append, and pass gh_token/project_root/repo_url/repo_ref to the remote bash via printf %q so a quote in a value cannot break out of the command.
- per-workspace-env envelope: the step-10 workspace test the user asked for is no longer forbidden by the same paragraph.
- linear guides: name the full verb, ORCA linear list-issues.
- parity guard: a prefix reference such as ORCA linear --help or ORCA emulator --webcam now has its flags checked against every command under that prefix; only an exact path or an explicit ... was checked before.
* skills: tighten prose in the seven rewritten guides
Shorter outcome spines, one idea per sentence, no restated rationale after a rule. No rule, command, or pinned phrase changes; 47 net lines fewer across the guides and references.
* skills: route orca-cli and per-workspace-env gates through --reference
Both guides told agents to load --full at a gate because the per-reference
selector did not exist when they were written. Now that main serves
`skills get <topic> --reference references/<file>.md`, load only the
named file and keep --full as the fallback for an older CLI, matching the
orchestration kernel.
* skills: drop outcome-spine boilerplate from the CLI-wrapper guides
The Result/Done/Safe-failure preambles and Next Action closers restated
rules the body already carries. Agents stop fine without them, and for
a CLI wrapper the command surface is the guide. Keeps the one substantive
rule computer-use's Done block added (never report unverified as success)
inside Action Rules. orchestration and per-workspace-env keep theirs:
those are multi-step workflows where the done bar is load-bearing.
(cherry picked from commit
|
||
|
Neil
|
1848855515 |
test: enable software WebGL for Linux CI headful specs (#19001)
* test: enable CI WebGL and route GPU-dependent regressions * test: retain headful atlas cases in terminal rendering goldens * test: reuse golden command in project coverage assertions |
||
|
Neil
|
1e301ab1df |
test: cover native Wayland Hangul in isolated CI (#19174)
* test: exercise native Wayland Hangul in isolated CI session * test: wait for nested compositor socket before selecting IBus * test: align Wayland IBus discovery with GNOME environment filtering * test: assert Wayland launch and register native Hangul evidence |
||
|
Neil
|
2e8fa3fe9b |
test: exercise packaged browser compatibility in scheduled CI (#19157)
* test: exercise packaged browser compatibility in scheduled CI * test: record final packaged workflow participation evidence * test: expose manual packaged revision and simplify executable check * test: reject missing package checksum assertion |
||
|
Neil
|
4120501979 |
perf(store): detect Zustand rerender churn the current audit cannot see (#19059)
* perf(store): detect Zustand rerender churn the current audit cannot see The app-store-performance audit only understood inline selectors passed to a hook imported literally as `useAppStore`, so three shapes went unlinted: - a selector referenced by name (`useAppStore(selectRows)`), including one hoisted below its call site — resolved now via a Program:exit pass - the sibling store hooks (`usePluginPanelsStore` and friends), matched by the use<Name>Store convention on local imports; React's `useSyncExternalStore` matches that shape and is excluded - a fresh reference nested inside a `useShallow` projection, which is the worst case of the three: the comparator runs on every write and can never match, so the memo silently buys nothing `no-nested-fresh-under-shallow` covers the last one. `src` is clean against all four rules today, so this is a ratchet rather than a cleanup. The write side stays undecidable statically — whether a `set()` reallocated for nothing depends on the payload — so it gets a runtime probe instead. withStoreIdentityChurnProbe counts writes that replace a field's reference while its value stays equal, and can name the calling site. Cost when disarmed is one boolean load per write, matching react-commit-cascade-write-probe. * perf(store): scope the churn probe's scan to the write's own keys recordWrite iterated Object.keys of the full post-write state, so the armed cost scaled with the store's top-level field count (hundreds) rather than the size of the write. `set(partial)` merges, so no field outside the partial can have changed. The wrapper now resolves a functional updater itself and iterates the resolved partial's keys. Same function, same argument, called once — there is a test pinning that, since calling it twice would double any work a slice does inside its own updater. A replace write drops absent fields, so that path still scans every field. Disarmed cost is unchanged: one boolean load. * perf(store): follow a selector one hop into its helper Review feedback: both the lint rule and the manual sweep it was checked against only looked at the inline selector body, so neither could see a fresh allocation made inside a helper the selector calls — and delegating to a module-scope helper is the idiomatic shape here. Two methods sharing a blind spot is not corroboration. The two fresh-reference rules now resolve a single hop into a module-scope helper. The predicate used across that hop is deliberately stricter than the inline one: it requires EVERY returned expression to allocate unconditionally, so the common `cache.get(k) ?? buildFresh(state)` identity-caching shape is not flagged. An unresolvable helper is left alone rather than guessed at. Still zero hits across 20,330 files, so this stays a ratchet. * perf(store): keep the churn probe off the shipped write path Review hardening for the churn probe and the widened lint rules. Probe: it no longer resolves a functional updater itself. Zustand keeps sole ownership of when and with what argument an updater runs, so the middleware cannot double-invoke it or hand it a stale state. Object partials still scope the scan to the write's own keys; updater and replace writes fall back to the full field list, which costs one Object.is per untouched field and nothing more, since the deep compare only runs on replaced references. store/index.ts installs the probe only when import.meta.env.DEV or e2eConfig.exposeStore is set, the same gate as __store exposure. Nothing in the app arms it, so a shipped build was paying a wrapper frame per write for a diagnostic it could never read. The cascade probe stays unconditional because crash telemetry arms it in the field. Site capture now skips any *-probe.ts frame; under the real composition the first non-node_modules frame was the cascade probe's wrapper, so every churn was attributed to react-commit-cascade-write-probe.ts:32 instead of the caller. Plugin: named-selector recording is restricted to module scope. A component-local `const selectRows = ...` used to overwrite the entry for a same-named imported selector and flag an unrelated useAppStore(selectRows). The any-branch and every-branch allocation predicates are one function with a flag, the Object.* static list is a Set, and import recording is a single pass. Tests: updater called once with live state, identical-state writes ignored, disarmed path forwards exact arguments without calling get(), full composition with the cascade probe (no drop, no double, correct site), and the module-scope shadowing case for the plugin. |
||
|
Jinwoo Hong
|
b8311d509a |
Revert "skills: rewrite the seven non-orchestration guides to one outcome-first standard (#18724)" (#19126)
This reverts commit
|