feat(query): dedup hybrid search by primary key on MemWAL tables

Hybrid search fuses its two legs on `_rowid`, so both sub-queries set
`with_row_id` -- which a MemWAL table rejects before it plans, because the
fresh tier has no stable row id. Every hybrid shape and every reranker
therefore 400s on a WAL-backed table.

Join on the primary key instead. Every WAL-managed table declares an
unenforced primary key and the WAL union already dedups each leg by it, so
the two legs share a key space by construction -- and because they are
separate round trips that can land on different snapshots, joining on the
key is strictly more correct than joining on row ids, which would emit the
same logical row twice at two versions.

The key is dictionary-encoded into a dense u64 written under the `_rowid`
name. The fusion only ever compares that column for equality -- dedup,
grouping, score restoration -- so any injective mapping serves, and writing
it under that name is what lets every reranker, third-party implementations
included, run unmodified. It never reaches the caller.

Routing is decided by `lsm_enabled()`, added to `BaseTable` with a `false`
default and implemented on the remote table over the existing
`get_lsm_write_spec` endpoint, cached beside the schema. No server change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UozJQ9kQxntWarST7hC971
This commit is contained in:
Daniel Rammer
2026-09-16 15:47:06 -05:00
co-authored by Claude Opus 5
parent bc4497b21a
commit 959645f1c7
11 changed files with 1163 additions and 26 deletions
+9
View File
@@ -1481,6 +1481,15 @@ impl Table {
})
}
/// Whether reads on this table route through the MemWAL. Cached by the
/// remote table, so the hybrid query path can ask on every call.
pub fn lsm_enabled(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
let inner = self_.inner_ref()?.clone();
future_into_py(self_.py(), async move {
inner.lsm_enabled().await.infer_error()
})
}
/// Converge the table's LSM write path into its base table.
///
/// Best-effort: with writes flowing, new rows may land after the last