mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-23 12:12:27 +00:00
Avanish Garg
bc4c32b779
`optimize()` fails once a tagged version is older than the cleanup cutoff, although the docs say tagged versions are exempt from cleanup, and the failed prune also skips the index-optimize step (repro and numbers in the issue). Change: `OptimizeAction::Prune` already exposes `error_if_tagged_old_versions: Option<bool>`, so no new API. `cleanup_old_versions` and `cleanup_old_versions_before` in `rust/lancedb/src/table/optimize.rs` now use `unwrap_or(false)`, which covers `OptimizeAction::Prune`, `OptimizeAction::All` and `optimize_prune_before`. The Python and Node bindings pass `None`, which now means "retain tagged versions"; I left `python/src/table.rs` and `nodejs/src/table.rs` untouched (this also avoids a conflict with #3870, which edits that region). `Some(true)` still opts in to the error. Doc comments updated; the sentence in `docs/web/management/table.mdx` is accurate once this lands. Tests: Rust `test_optimize_prune_retains_tagged_versions` (`Some(true)` still errors; the default retains the tag; `optimize_prune_before`; `checkout_tag` plus a data read), a Python sync test with a scalar index (index step runs, a second optimize works, untagged versions are pruned, `checkout("release-1")` works), a Python async test and a Node test. **What I could and could not verify (please read):** - I could NOT compile Rust here (not enough disk), so `cargo check`, `cargo test`, the Python/Node binding builds and therefore the fixed-path assertions of the new Rust, Python and Node tests have not been run. This is why the PR is a draft; I will mark it ready once CI is green and fix whatever it finds. - Ran: both new Python tests fail on the released 0.40.0 wheel with the exact error above (so they do catch the bug); pylance 13.0.0 confirms the semantics (`cleanup_old_versions(error_if_tagged_old_versions=False)` removes the untagged versions and keeps the tagged one, still readable with `lance.dataset(version="keep")`); in Lance v14.0.0-beta.7 `cleanup.rs` the policy field is `bool`, the builder default is `true`, and Lance's own `auto_cleanup_hook` uses `Some(false)`; `rustfmt --edition 2024 --check` passes on `optimize.rs`. - The Rust and Node tests rely on timestamp ordering (`nextMillisecond()`, `days(0)`/`now()`), so they could be flaky: tell me if you prefer another approach. Behaviour change: anyone who relied on the error to learn that a tag blocks cleanup now gets a silent retain. Not changed: the deprecated sync `LanceTable.cleanup_old_versions` calls pylance directly and still errors on tagged versions (the keyword may not exist in the oldest supported pylance); `freshness.rs:1377` and `refresh.rs:1077` still pass `None` (I did not check what those paths do with tagged versions). Fixes #4543
The Multimodal Lakehouse for AI
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
- Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us