From 5b739473e554cf1a50aa6da2ea89e32876446fbc Mon Sep 17 00:00:00 2001 From: Jinwoo-H Date: Wed, 9 Sep 2026 12:47:50 -0400 Subject: [PATCH] docs(ai-vault-search): re-measure the query benchmark after the operator change repo: and path: moved out of SQL, so the operator-only row measures something different now and the note that it is a range seek was wrong. The rest of the table is re-measured on an idle machine: the previous run's p95 column was mostly contention, which is why conversation looked 3.4x faster at p95 rather than the 1.7x it actually is. Adds what this PR does not settle: which process may unlink and rebuild the index is PR 3b's, and PR 4 is only the first thing that makes reading it reachable. --- .../agent-session-search-query-tuning.md | 67 ++++++++++++------- 1 file changed, 41 insertions(+), 26 deletions(-) diff --git a/docs/reference/agent-session-search-query-tuning.md b/docs/reference/agent-session-search-query-tuning.md index 97bffe96619..284377b1d03 100644 --- a/docs/reference/agent-session-search-query-tuning.md +++ b/docs/reference/agent-session-search-query-tuning.md @@ -30,37 +30,42 @@ to that path as well as printed. Corpus: 40 synthetic Claude transcripts, 10.5 MB, 9,600 messages, indexed through the real store. Eight queries, one per rung of the route ladder plus the two shapes that skip it; 5 warm-up runs and 25 samples each. Apple silicon, warm page -cache. Milliseconds. +cache, machine otherwise idle. Milliseconds, and p95 over 25 samples moves +several milliseconds run to run if anything else is competing for the disk. -| Scope | p50 | p95 | -| -------------- | ---- | ----- | -| `all` | 8.46 | 41.74 | -| `conversation` | 5.32 | 12.20 | +| Scope | p50 | p95 | +| -------------- | ---- | ---- | +| `all` | 6.08 | 8.82 | +| `conversation` | 3.74 | 5.09 | Per query, `all` then `conversation` (p50 / p95): -| Query | `all` | `conversation` | -| ------------------------------------------------ | ------------- | -------------- | -| `"terminal reattach"` (phrase) | 5.03 / 5.86 | 2.77 / 3.78 | -| `resolveTerminalPath` (identifier) | 8.47 / 9.54 | 5.39 / 17.89 | -| `src/main/…/session-transcript-reader.ts` (path) | 11.96 / 39.31 | 5.55 / 17.27 | -| `why is the daemon snapshot stale` (prose) | 10.78 / 25.23 | 7.11 / 12.34 | -| `reattahc worktre` (typo repair) | 10.41 / 115.05| 6.73 / 9.41 | -| `index` (common term) | 7.25 / 29.99 | 4.98 / 11.51 | -| `repo:app-3` (operator only) | 0.10 / 0.60 | 0.10 / 0.20 | -| `worktree` scoped to one cwd | 2.10 / 2.60 | 1.48 / 2.06 | +| Query | `all` | `conversation` | +| ------------------------------------------------ | ----------- | -------------- | +| `"terminal reattach"` (phrase) | 3.61 / 3.76 | 1.94 / 2.09 | +| `resolveTerminalPath` (identifier) | 6.16 / 6.40 | 3.81 / 3.97 | +| `src/main/…/session-transcript-reader.ts` (path) | 8.76 / 11.14| 3.95 / 3.98 | +| `why is the daemon snapshot stale` (prose) | 7.61 / 7.81 | 5.07 / 5.19 | +| `reattahc worktre` (typo repair) | 7.32 / 7.51 | 4.93 / 5.09 | +| `index` (common term) | 5.34 / 5.58 | 3.54 / 3.61 | +| `repo:app-3` (operator only) | 0.13 / 0.16 | 0.12 / 0.13 | +| `worktree` scoped to one cwd | 1.48 / 1.52 | 1.05 / 1.13 | Reading it: -- `conversation` is about 1.6x faster at p50 and 3.4x at p95. That gap is the +- `conversation` is about 1.6x faster at p50 and 1.7x at p95. That gap is the answer to "what is the second table for": it is the corpus a keystroke can afford, and it holds no tool output, so it is also the corpus where a match is something a person wrote. -- An operator-only query never touches FTS at all. It is a range seek on - `sessions_cwd_key`, and it costs a tenth of a millisecond. -- Typo repair's p95 in `all` is the worst number on the page. The repair walks - `messages_vocab` per prefix, and the first walk after a cold statement cache - pays for the b-tree pages. It is a first-query cost, not a per-query one. +- A `scopePaths` query is the cheapest real search on the page. It is the one + narrowing SQL can express exactly, so it seeks `sessions_cwd_key` and hands + ranking a small candidate set. +- The operator-only figure is a floor, not a typical cost. `repo:` and `path:` + are applied in JS over retrieved rows (see `session-search-row-filter` for why + they cannot be pushed into SQL), so their cost tracks how many sessions the + walk has to read before it fills a candidate set. This corpus has 40 sessions, + which is one page of that walk; an index where few sessions match the operator + will read up to the ceiling in `session-search-retrieval` instead. ## `sessionCandidateLimit` @@ -76,12 +81,12 @@ seen and the ordering alone moves p95 further than the limit does. | Limit | p50 | p95 | Pages of 20 a caller can reach | | ----- | ----- | ----- | ------------------------------ | -| 200 | 9.50 | 13.18 | 10 | -| 600 | 10.86 | 20.30 | 30 | -| 1200 | 12.65 | 15.20 | 60 | -| 2400 | 17.68 | 27.08 | 120 | +| 200 | 7.08 | 7.23 | 10 | +| 600 | 8.18 | 8.73 | 30 | +| 1200 | 9.72 | 10.44 | 60 | +| 2400 | 12.60 | 13.68 | 120 | -600 is the default: it costs about 14% over 200 at p50 and buys three times the +600 is the default: it costs about 16% over 200 at p50 and buys three times the reachable depth, and the curve only turns steep past 1200. A host with a much larger index can raise it; the result's `truncated.candidates` says when the limit was the thing that cut the answer, so a caller never has to guess. @@ -92,3 +97,13 @@ not what it retrieves. The MRR figures quoted in the BM25 weights (`session-search-identifier-split.ts`) come from the original retrieval shoot-out on real transcripts and are not reproducible from this repository. Any change to the limit justified on relevance grounds needs an eval set, not this benchmark. + +## Not settled here + +Which process may open, unlink and rebuild the index is PR 3b's decision. A +second handle that finds an older schema version replaces the file while a live +store keeps answering from the unlinked inode, and this PR is what first makes +that reachable, because it is the first thing that reads. What PR 4 does is +refuse to make it worse: an engine over an index it does not fully recognise +answers from the tables that are there and names the feature it cannot serve, +rather than throwing on the first query that reaches for one.