mirror of
https://github.com/lancedb/lancedb.git
synced 2026-08-18 03:58:26 +00:00
Yang Cen
1d75638dea
## What is the bug? #3731 tries to distinguish a missing table from a corrupt table after Lance returns `DatasetNotFound`. It does that by listing the database parent and treating a physical `<name>.lance` entry as evidence that the table exists. That premise is not sound for a listing database. Table creation writes data before atomically committing the first manifest, so the same physical prefix can represent a live concurrent create, abandoned uncommitted data, or an old empty directory. It is not evidence of a committed table. The parent listing also makes every missing-table open, including the create-on-miss path, perform work proportional to the number of sibling tables. Cloud `list_with_delimiter` exhausts all pages before returning. ## How does this PR fix the problem? This PR makes the committed Lance manifest the sole table-existence authority for listing-database opens: - `DatasetNotFound` maps directly to `TableNotFound`; no parent or target storage probe runs. - Other Lance load errors continue to propagate unchanged. - A physical directory, object prefix, or uncommitted data file alone does not block `Create`. - Concurrent `Create` requests are arbitrated by the conditional version-1 manifest commit: one succeeds and the loser receives `TableAlreadyExists`. - `table_names` is documented as physical discovery, not an atomic table-existence check. Its snapshot can contain an entry that is still being created, has only uncommitted storage, or is concurrently dropped. This removes the need for a new Lance object-store capability. LanceDB remains on the official Lance `v11.0.0-beta.6` dependency from `main`; the merge commit for lance-format/lance#7722 is an ancestor of that tag, so the ambiguous-GCS-500 corruption-prevention fix is retained. ## Performance evidence Lower is better. The benchmark uses real `.lance` directories with marker objects on the local filesystem; fixture creation and teardown are outside the timed region. Baseline is `origin/main` at `6fb976cf`, candidate is `e1240751`. Both were built from the same lockfile on the same macOS arm64 machine with the repository's `release` profile (fat LTO), then executed in alternating baseline/candidate order for three pairs. Each run used 10 warmups and 100 distinct missing-table opens per scale. The table reports the median of the three run-level percentiles. | Scenario / metric | Baseline | This PR | Benefit | | --- | ---: | ---: | ---: | | 1,000 real sibling directories, p50 | 11.905 ms | 21.042 us | 566x speedup | | 10,000 real sibling directories, p50 | 143.630 ms | 18.375 us | 7,817x speedup | | 100,000 real sibling directories, p50 | 1.991 s | 19.917 us | 99,984x speedup | | 100,000 real sibling directories, p95 | 2.346 s | 25.792 us | 90,965x speedup | These results validate removal of the sibling-cardinality dependency in this local-filesystem workload; they are not an extrapolation to production GCS latency. A structural object-store regression test separately asserts that opening one missing table performs zero parent-scoped `list`, `list_with_offset`, or `list_with_delimiter` calls. Run with: ```bash BENCH_SIBLINGS=1000,10000,100000 BENCH_WARMUPS=10 BENCH_TRIALS=100 \ cargo run --locked --release --quiet -p lancedb --example bench_open_missing_table ``` ## Correctness and compatibility boundaries - An empty `.lance` directory or orphan data without a committed manifest now opens as `TableNotFound` and may be replaced by a successful `Create`. - Two synchronized creators sharing one object store deterministically produce one success and one conditional-manifest conflict mapped to `TableAlreadyExists`. - A readable manifest remains authoritative; non-`DatasetNotFound` corruption, external-manifest, authorization, and object-store errors are not folded into `TableNotFound`. - `TableCorrupted` remains in the public error enum for compatibility, but this listing-database fallback no longer synthesizes it from an ambiguous physical footprint. - Reliably distinguishing `Missing`, `Creating`, and `Corrupt` would require explicit authoritative lifecycle/catalog metadata (for example a leased creation record). It cannot be inferred from a directory or prefix, and is outside this incident fix. ## Validation - `cargo fmt --all -- --check` - `cargo check --quiet --locked -p lancedb --features remote --tests --examples` - `cargo clippy --quiet --locked -p lancedb --features remote --tests --examples -- -D warnings` - `cargo test --quiet --locked -p lancedb --features remote --tests` - library: 843 passed, 1 ignored - integration groups: 39 passed, 6 passed, 5 passed - focused coverage for empty directories, orphan data, physical listing snapshots, zero parent listings, and concurrent manifest arbitration
The Multimodal AI Lakehouse
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
- Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us