mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-14 06:36:54 +00:00
Xuanwo
6d24446a30
Existing Rust and Python blob range calls must keep working when only the SDK is upgraded. Add negotiated batched range reads across Rust, Python sync/async and TypeScript, preserving ordered NULLs, duplicates and branch/version reads with splitting and reassembly at 1,024 ranges or 64 MiB per request. Only a successful describe response advertising `x-lancedb-blob-ranges: 1` selects the batched endpoint. Missing or unknown capability values select the existing descriptor-query plus BlobFile Range GET route before the operation. Endpoint failures never switch protocols. HTTP 400 preserves the server message and request ID as Rust `InvalidInput` / Python `ValueError`; other errors retain their existing semantics. The operation binds both numeric version and immutable snapshot identity from its producing query or first range response. Every later partition and oversized-range piece sends that original identity and requires an equal response identity before decoding. Same-name table recreation therefore fails explicitly instead of returning mixed old/new payloads; no partial array is returned on later failure. Targets `main`. The read-buffering prerequisite #4979, safe-write prerequisite #4977 and deferred-query identity prerequisite #4973 are merged into main; #5000 also supplies the merged SQL timeout-test prerequisite. Server support is [Sophon #10219](https://github.com/lancedb/sophon/pull/10219), including #10211. Exact SDK `1497b508324662180086a4d1e0a6eb9e0b3970db` and server `f39712c2c150b220eb831e00f6663c9eef32dce5` passed a real local SDK → REST → PE experiment: one Rust harness test, 14 assertion groups, 38 REST range requests and 34 PE RPCs. All four sync/async × warm/cold table-recreation cases failed explicitly before the second descriptor lookup or PE dispatch, with no mixed payload or partial result. The other groups retained 1,025-range success, 70 MiB reassembly, branch/version reads, overwrite snapshot consistency, error behavior and TypeScript buffer ownership. Bindings were rebuilt and their loaded hashes verified. This is local interoperability evidence, not Cloud, mixed-fleet or deployed header-forwarding acceptance. The server dependency upgrade, External/cache/URI-policy acceptance and validated rollout remain required. Enforcing PE support must precede QN and SDK enablement; identity does not retain versions or freeze External objects. This PR does not claim deployed support or complete ENT-3100/GA acceptance. Published documentation retains the Early access badge and released Enterprise instructions, including explicit fetches for copying. The new Enterprise `blob_mode="bytes"` guidance, snapshot-bound copy example, future streaming restrictions and GA wording remain source-only MDX drafts until release. The correction distinguishing released Enterprise SQL bytes from SDK descriptors remains visible. Documentation validation checked comment boundaries, visible instructions, Python example syntax and whitespace; the site was not built. The main integration keeps these release gates unchanged. Main integration at `0491d9bdcc65ce5d52ea93e1083c89e951ea4f9d` normally merges main `9b13e8f72530c8b1950c09685f6920711154a539`, including its Lance beta.18 dependency. Focused current-source validation passed: 493 remote-table/query/Blob/IPC/write-progress Rust tests, 23 shared create/write-body tests, 59 native Blob integration tests and 7 public HTTP range tests. Rebuilt Python passed 468 selected tests (one optional PyAV test skipped), and rebuilt TypeScript passed 194 selected tests across four suites; lint/format, TypeScript compilation and API documentation generation passed. Shared tests include actual HTTP multipart failures, acknowledgment boundaries, dictionary compaction and native Blob compaction. The 598 unrelated Jest cases excluded by the focused shared-suite selector were not run. The retained 14-group joint REST/PE experiment used the earlier exact candidates stated above. Operation-identity/range/IPC code is unchanged, but dependencies and shared write interfaces changed with main; the new local checks validate the SDK combination and do not constitute a fresh final-head SDK/server joint, Cloud or rollout run. Early access and every remaining server/External/cache/policy/dependency/rollout condition remain in force. CI snapshot on this head (2026-11-12 18:30 UTC): `build-no-lock` failed in the standalone `ci/tests/test_build_no_lock.py` fixture at the retained-rlib assertion, before the actual SDK no-lock build. The fixture, helper and Rust workflow are byte-identical to main; the fixture does not load SDK source. The underlying CI/environment cause is not diagnosed here. Other jobs remained pending/running at the snapshot, so CI is not reported as passing. Failure: https://github.com/lancedb/lancedb/actions/runs/43280277756/job/131399418284.
The Multimodal Lakehouse for AI
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- LanceDB Embedded (open source): 100% open source, runs in your application's process; your data can live on local disk or in object storage. No vendor lock-in.
- LanceDB Enterprise: Production-scale distributed search, indexing, and maintenance. Plus, complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us