mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-13 22:26:53 +00:00
ChilePiquinandClaude Sonnet 5
2dec579641
The Python bindings route every async call through a single, lazily created Tokio runtime, stored behind a raw pointer so it can survive being reset (not dropped) across fork() -- dropping it there would hang joining now-dead worker threads. That leak-on-fork path is deliberate and documented, but normal process exit had no equivalent handling at all: nothing ever tells the runtime to shut down, so its worker threads keep running, completely uncoordinated with the interpreter, until the process ends. If a worker thread is mid-task exactly as Py_Finalize begins tearing down interpreter state, it can panic on state that's already gone. That panic happens on a background thread with no PyO3-wrapped call frame to catch it -- and if it lands while another unwind is already in progress, Rust aborts the whole process outright rather than merely failing that one call. This reproduces as an intermittent SIGABRT on otherwise-successful short-lived processes that open a connection, do a handful of operations, and exit normally. Give the runtime a coordinated, bounded shutdown at the one point it was missing one: register a Python atexit callback that calls runtime::shutdown(), which takes the runtime through the same atomic slot the fork handler uses and calls shutdown_timeout() rather than a bare drop -- bounded, so a wedged background task can never hang process exit, but real in-flight work gets a genuine chance to finish first. Runs with the GIL released (Python::detach), since the wait would otherwise be able to deadlock against any in-flight task that itself needs the GIL to complete. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The Multimodal AI Lakehouse
How to Install ✦ Detailed Documentation ✦ Tutorials and Recipes ✦ Contributors
The ultimate multimodal data platform for AI/ML applications.
LanceDB is designed for fast, scalable, and production-ready vector search. It is built on top of the Lance columnar format. You can store, index, and search over petabytes of multimodal data and vectors with ease. LanceDB is a central location where developers can build, train and analyze their AI workloads.
Demo: Multimodal Search by Keyword, Vector or with SQL
Star LanceDB to get updates!
⭐ Click here ⭐ to see how fast we're growing!
Key Features:
- Fast Vector Search: Search billions of vectors in milliseconds with state-of-the-art indexing.
- Comprehensive Search: Support for vector similarity search, full-text search and SQL.
- Multimodal Support: Store, query and filter vectors, metadata and multimodal data (text, images, videos, point clouds, and more).
- Advanced Features: Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure. GPU support in building vector index.
Products:
- Open Source & Local: 100% open source, runs locally or in your cloud. No vendor lock-in.
- Cloud and Enterprise: Production-scale vector search with no servers to manage. Complete data sovereignty and security.
Ecosystem:
- Columnar Storage: Built on the Lance columnar format for efficient storage and analytics.
- Seamless Integration: Python, Node.js, Rust, and REST APIs for easy integration. Native Python and Javascript/Typescript support.
- Rich Ecosystem: Integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
How to Install:
Follow the Quickstart doc to set up LanceDB locally.
API & SDK: We also support Python, Typescript and Rust SDKs
| Interface | Documentation |
|---|---|
| Python SDK | https://lancedb.github.io/lancedb/python/python/ |
| Typescript SDK | https://lancedb.github.io/lancedb/js/globals/ |
| Rust SDK | https://docs.rs/lancedb/latest/lancedb/index.html |
| REST API | https://docs.lancedb.com/api-reference/rest |
Join Us and Contribute
We welcome contributions from everyone! Whether you're a developer, researcher, or just someone who wants to help out.
If you have any suggestions or feature requests, please feel free to open an issue on GitHub or discuss it on our Discord server.
Check out the GitHub Issues if you would like to work on the features that are planned for the future. If you have any suggestions or feature requests, please feel free to open an issue on GitHub.
Contributors
Stay in Touch With Us