mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-14 06:36:54 +00:00
Yang Cen
69ac2158a1
`take_row_ids` used an `IN` filter, so the backend could reorder rows and collapse duplicate IDs. That can silently misalign training rows with labels or sample weights. Restore the requested order and duplicate occurrences in the shared Rust take path for native and remote tables, and retain that information through Python query serialization. Missing or deleted IDs are omitted. Explicit Rust `order_by` overrides request order, and pagination applies after restoring occurrences. Empty projections and `_rowid` projection aliases retain their output schema. RTE tables reject unsupported takes with the existing base-table guidance before checking the internal row-ID projection used for ordering. An explicit `use_rte(False)` reads the base table with the same requested order and occurrences. A pre-existing fixture defect also blocked CI: `test_table_add_in_threadpool` responded without reading uploads, allowing a connection reset during response decoding. This PR includes a fix confined to that test's handler: consume each request body before responding and verify one create row plus ten inserted rows. This was not a regression in the take implementation. Production write retries, concurrency and HTTP client code are unchanged. The selective sampling guide scopes the row-ID guarantees to SDK releases containing this fix, preserves the distinct offset-take semantics, and explains that ordered takes collect and combine matching rows before output batching. It recommends smaller ID requests and narrow projections because `max_batch_length` does not cap working memory. This uses the existing REST query protocol and requires a new SDK release; no deployed service was changed. Validation: - Docs follow-up: the changed page compiles with the Mint 4.2.888 MDX compiler; Typos 1.26.0 passes and Vale 3.24.0 has no findings on changed lines. Full-site Mint validation reports four existing snippet-import warnings on the reranking/vector-search pages, identical with the original page restored in a baseline comparison. - The applied threadpool test passed locally after the fixture change. The identical candidate previously passed 10 trials with the ordinary reader and 10 with an unbuffered reader; each trial verifies all uploaded rows. Those checks used macOS arm64/Python 3.14/default features, not the full failing CI environment. The normal new-head macOS Python 3.13/experimental-functions job remains the required platform check. - Python package Ruff format/check passed. Repository-root Ruff still reports 59 existing errors and six formatting differences in unchanged `docs/web-tests` files; none are in changed files. - At `a569d3f0`, the RTE diagnostic regression was reproduced and fixed; 5 targeted cases and strict workspace Clippy passed. The fixture-only follow-up changes no production Rust/Python code, so that evidence is reused. - Earlier validation at `6bcc0878`: 439 Rust query/remote-table tests and 24 Python take/getitems tests passed; the focused 49-test suite also covered offset dataloaders. A local 9,216-row, four-fragment reproduction returned 200 seeded random batches of 64 rows with zero ordering or label-alignment mismatches. The ordering and projection implementation is unchanged by the follow-ups. - No live Enterprise run was performed. Fixes [ENT-3469](https://linear.app/lancedb/issue/ENT-3469).
The Multimodal Lakehouse for AI
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- LanceDB Embedded (open source): 100% open source, runs in your application's process; your data can live on local disk or in object storage. No vendor lock-in.
- LanceDB Enterprise: Production-scale distributed search, indexing, and maintenance. Plus, complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us