mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
XY ZhanandXYZhan
25fcfc696a
## Why lance-format/lance#9803 makes the MemWAL's memtable index kinds pluggable: a `MemIndexRegistry` on the writer config decides which base-table index kinds are maintained in memory, so a kind Lance does not build in can be added by registering a plugin. LanceDB opens the MemWAL writer itself for `merge_insert`, and checks a named maintained set itself, both against Lance's built-ins only. So a kind registered outside Lance could not be named in `LsmWriteSpec::with_maintained_indexes`, and under "maintain every index" it was skipped. ## What changes - `NativeTable::set_mem_index_registry(registry)` sets the memtable index kinds a table maintains. The default is Lance's built-ins, so nothing changes unless a caller sets one. The registry checks a named maintained set (`validate_maintained_indexes_with`), is recorded on `initialize_mem_wal`, and goes on the writer config `merge_insert` opens. `lancedb::table` re-exports `MemIndexRegistry` and Lance's plugin traits (`mem_index`), so a caller needs no direct `lance` dependency. - The LSM vector read no longer checks resident memtables at all, and the `carries_vector_index` helper goes with it. Lance already compares every row of a memtable whose index can't answer the search, so no rows are missed. It still requires the base vector index to be maintained. ## Limits - The registry belongs to the table handle. Clones of a handle and its branch handles share it; a handle opened separately starts from the built-ins. Set it before the handle's first LSM write: a writer already open keeps the kinds it opened with. - The maintained set is stored with the table but the registry is not. Once a spec names an index only a registered plugin maintains, an LSM write from a handle without that registry is refused with Lance's "no registered plugin maintains" error rather than skipping the index. That includes the Python and Node bindings, which have no way to set a registry yet. - Rust only. A registry holds Rust plugin objects, which the Python, TypeScript and Java bindings cannot construct, so they get no setter; their documented behaviour (built-in kinds only) is unchanged. - Per-index plugin settings (`ShardWriterConfig::index_overrides`, from lance-format/lance#9805) are not exposed; writers use each plugin's defaults. ## Open decision Whether the registry should also be settable on the connection, so every table a connection opens carries it. This PR keeps it per table; a connection-level option would sit on top of `set_mem_index_registry` without changing it. ## Depends on Lance v14.0.0-beta.13, which `main` already uses. It includes lance-format/lance#9803 and #9805 (pluggable memtable index kinds) and #9857 (flushed generations keep the table's field ids). ## Testing - `lsm_maintains_an_index_kind_from_the_tables_registry`: a stand-in plugin maintains the base table's bitmap index as a B-tree. Naming the index is refused without the registry and accepted with it; a separately opened handle without the registry is refused an LSM write. Rows written through `merge_insert` are found by id before and after the flush. Each generation the shard manifests publish carries a bitmap index on `region`, and a filter on it is answered through that index. - `lsm_default_set_maintains_a_registry_kind_created_while_writing`: under "maintain every index", a bitmap index created after the writer opened is picked up by that writer and flushed. A clone of the table handle shares the registry. - `lsm_vector_search_compares_every_row_when_no_resident_index_answers`: with a vector plugin that answers no search, LSM vector search still returns the right rows, including a second page. - `cargo test -p lancedb --features remote --lib` passes; `cargo clippy -p lancedb --features remote --tests -- -D warnings` and `cargo fmt --all -- --check` are clean. --------- Co-authored-by: XYZhan <xuanyu.zhan@lancedb.com>
The Multimodal Lakehouse for AI
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
- Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us