mirror of
https://github.com/lancedb/lancedb.git
synced 2025-12-25 22:29:58 +00:00
f9dd7a5d8a
This forces the user to replace the whole FTS directory when re-creating the index, prevent duplicate data from being created. Previously, the whole dataset was re-added to the existing index, duplicating existing rows in the index. This (in combination with lancedb/lance#1707) caused #726, since the duplicate data emitted duplicate indices for `take()` and an upstream issue caused those queries to fail. This solution isn't ideal, since it makes the FTS index temporarily unavailable while the index is built. In the future, we should have multiple FTS index directories, which would allow atomic commits of new indexes (as well as multiple indexes for different columns). Fixes #498. Fixes #726. --------- Co-authored-by: Chang She <759245+changhiskhan@users.noreply.github.com>
LanceDB
A Python library for LanceDB.
Installation
pip install lancedb
Usage
Basic Example
import lancedb
db = lancedb.connect('<PATH_TO_LANCEDB_DATASET>')
table = db.open_table('my_table')
results = table.search([0.1, 0.3]).limit(20).to_list()
print(results)
Development
Create a virtual environment and activate it:
python -m venv venv
. ./venv/bin/activate
Install the necessary packages:
python -m pip install .
To run the unit tests:
pytest
To run linter and automatically fix all errors:
black .
isort .
If any packages are missing, install them with:
pip install <PACKAGE_NAME>
For Windows users, there may be errors when installing packages, so these commands may be helpful:
Activate the virtual environment:
. .\venv\Scripts\activate
You may need to run the installs separately:
pip install -e .[tests]
pip install -e .[dev]
tantivy requires rust to be installed, so install it with conda, as it doesn't support windows installation:
pip install wheel
pip install cargo
conda install rust
pip install tantivy
To run the unit tests:
pytest