mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-13 23:32:27 +00:00
XY ZhanandXYZhan
25fcfc696a
## Why lance-format/lance#9803 makes the MemWAL's memtable index kinds pluggable: a `MemIndexRegistry` on the writer config decides which base-table index kinds are maintained in memory, so a kind Lance does not build in can be added by registering a plugin. LanceDB opens the MemWAL writer itself for `merge_insert`, and checks a named maintained set itself, both against Lance's built-ins only. So a kind registered outside Lance could not be named in `LsmWriteSpec::with_maintained_indexes`, and under "maintain every index" it was skipped. ## What changes - `NativeTable::set_mem_index_registry(registry)` sets the memtable index kinds a table maintains. The default is Lance's built-ins, so nothing changes unless a caller sets one. The registry checks a named maintained set (`validate_maintained_indexes_with`), is recorded on `initialize_mem_wal`, and goes on the writer config `merge_insert` opens. `lancedb::table` re-exports `MemIndexRegistry` and Lance's plugin traits (`mem_index`), so a caller needs no direct `lance` dependency. - The LSM vector read no longer checks resident memtables at all, and the `carries_vector_index` helper goes with it. Lance already compares every row of a memtable whose index can't answer the search, so no rows are missed. It still requires the base vector index to be maintained. ## Limits - The registry belongs to the table handle. Clones of a handle and its branch handles share it; a handle opened separately starts from the built-ins. Set it before the handle's first LSM write: a writer already open keeps the kinds it opened with. - The maintained set is stored with the table but the registry is not. Once a spec names an index only a registered plugin maintains, an LSM write from a handle without that registry is refused with Lance's "no registered plugin maintains" error rather than skipping the index. That includes the Python and Node bindings, which have no way to set a registry yet. - Rust only. A registry holds Rust plugin objects, which the Python, TypeScript and Java bindings cannot construct, so they get no setter; their documented behaviour (built-in kinds only) is unchanged. - Per-index plugin settings (`ShardWriterConfig::index_overrides`, from lance-format/lance#9805) are not exposed; writers use each plugin's defaults. ## Open decision Whether the registry should also be settable on the connection, so every table a connection opens carries it. This PR keeps it per table; a connection-level option would sit on top of `set_mem_index_registry` without changing it. ## Depends on Lance v14.0.0-beta.13, which `main` already uses. It includes lance-format/lance#9803 and #9805 (pluggable memtable index kinds) and #9857 (flushed generations keep the table's field ids). ## Testing - `lsm_maintains_an_index_kind_from_the_tables_registry`: a stand-in plugin maintains the base table's bitmap index as a B-tree. Naming the index is refused without the registry and accepted with it; a separately opened handle without the registry is refused an LSM write. Rows written through `merge_insert` are found by id before and after the flush. Each generation the shard manifests publish carries a bitmap index on `region`, and a filter on it is answered through that index. - `lsm_default_set_maintains_a_registry_kind_created_while_writing`: under "maintain every index", a bitmap index created after the writer opened is picked up by that writer and flushed. A clone of the table handle shares the registry. - `lsm_vector_search_compares_every_row_when_no_resident_index_answers`: with a vector plugin that answers no search, LSM vector search still returns the right rows, including a second page. - `cargo test -p lancedb --features remote --lib` passes; `cargo clippy -p lancedb --features remote --tests -- -D warnings` and `cargo fmt --all -- --check` are clean. --------- Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>