f485378ea4
This is v1 of integrating full text search index into LanceDB.
# API
The query API is roughly the same as before, except if the input is text
instead of a vector we assume that its fts search.
## Example
If `table` is a LanceDB LanceTable, then:
Build index: `table.create_fts_index("text")`
Query: `df = table.search("puppy").limit(10).select(["text"]).to_df()`
# Implementation
Here we use the tantivy-py package to build the index. We then use the
row id's as the full-text-search index's doc id then we just do a Take
operation to fetch the rows.
# Limitations
1. don't support incremental row appends yet. New data won't show up in
search
2. local filesystem only
3. requires building tantivy explicitly
---------
Co-authored-by: Chang She <chang@lancedb.com>
2.0 KiB
Welcome to LanceDB's Documentation
LanceDB is an open-source database for vector-search built with persistent storage, which greatly simplifies retrivial, filtering and management of embeddings.
The key features of LanceDB include:
-
Production-scale vector search with no servers to manage.
-
Store, query and filter vectors, metadata and multi-modal data (text, images, videos, point clouds, and more).
-
Native Python and Javascript/Typescript support (coming soon).
-
Zero-copy, automatic versioning, manage versions of your data without needing extra infrastructure.
-
Ecosystem integrations with LangChain 🦜️🔗, LlamaIndex 🦙, Apache-Arrow, Pandas, Polars, DuckDB and more on the way.
LanceDB's core is written in Rust 🦀 and is built using Lance, an open-source columnar format designed for performant ML workloads.
Installation
pip install lancedb
Quickstart
import lancedb
db = lancedb.connect(".")
table = db.create_table("my_table",
data=[{"vector": [3.1, 4.1], "item": "foo", "price": 10.0},
{"vector": [5.9, 26.5], "item": "bar", "price": 20.0}])
result = table.search([100, 100]).limit(2).to_df()
Complete Demos
We will be adding completed demo apps built using LanceDB.
Documentation Quick Links
Basic Operations- basic functionality of LanceDB.Embedding Functions- functions for working with embeddings.Indexing- create vector indexes to speed up queries.Full text search- [EXPERIMENTAL] full-text search APIEcosystem Integrations- integrating LanceDB with python data tooling ecosystem.API Reference- detailed documentation for the LanceDB Python SDK.