mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-04 09:46:52 +00:00
feat(query): dedup hybrid search by primary key on MemWAL tables
Hybrid search fuses its two legs on `_rowid`, so both sub-queries set `with_row_id` -- which a MemWAL table rejects before it plans, because the fresh tier has no stable row id. Every hybrid shape and every reranker therefore 400s on a WAL-backed table. Join on the primary key instead. Every WAL-managed table declares an unenforced primary key and the WAL union already dedups each leg by it, so the two legs share a key space by construction -- and because they are separate round trips that can land on different snapshots, joining on the key is strictly more correct than joining on row ids, which would emit the same logical row twice at two versions. The key is dictionary-encoded into a dense u64 written under the `_rowid` name. The fusion only ever compares that column for equality -- dedup, grouping, score restoration -- so any injective mapping serves, and writing it under that name is what lets every reranker, third-party implementations included, run unmodified. It never reaches the caller. Routing is decided by `lsm_enabled()`, added to `BaseTable` with a `false` default and implemented on the remote table over the existing `get_lsm_write_spec` endpoint, cached beside the schema. No server change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UozJQ9kQxntWarST7hC971
This commit is contained in:
co-authored by
Claude Opus 5
parent
bc4497b21a
commit
959645f1c7
@@ -1481,6 +1481,15 @@ impl Table {
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether reads on this table route through the MemWAL. Cached by the
|
||||
/// remote table, so the hybrid query path can ask on every call.
|
||||
pub fn lsm_enabled(self_: PyRef<'_, Self>) -> PyResult<Bound<'_, PyAny>> {
|
||||
let inner = self_.inner_ref()?.clone();
|
||||
future_into_py(self_.py(), async move {
|
||||
inner.lsm_enabled().await.infer_error()
|
||||
})
|
||||
}
|
||||
|
||||
/// Converge the table's LSM write path into its base table.
|
||||
///
|
||||
/// Best-effort: with writes flowing, new rows may land after the last
|
||||
|
||||
Reference in New Issue
Block a user