rust/lancedb
1
0
Fork 0
mirror of https://github.com/lancedb/lancedb.git synced 2025-12-26 14:49:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: enable stemming (#1356)

Browse Source 
Added the ability to specify tokenizer_name, when creating a full text
search index using tantivy. This enables the use of language specific
stemming.

Also updated the [guide on full text
search](https://lancedb.github.io/lancedb/fts/) with a short section on
choosing tokenizer.

Fixes #1315
This commit is contained in:
josca42
2024-06-20 23:23:55 +02:00
committed by GitHub
parent f41eb899dc
commit 0fe844034d
4 changed files with 35 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
python/python/tests/test_fts.py
View File

@@ -66,6 +66,17 @@ def test_create_index(tmp_path):
assert os.path.exists(str(tmp_path / "index"))
def test_create_index_with_stemming(tmp_path, table):
index = ldb.fts.create_index(
str(tmp_path / "index"), ["text"], tokenizer_name="en_stem"
)
assert isinstance(index, tantivy.Index)
assert os.path.exists(str(tmp_path / "index"))
# Check stemming by running tokenizer on non empty table
table.create_fts_index("text", tokenizer_name="en_stem")
def test_populate_index(tmp_path, table):
index = ldb.fts.create_index(str(tmp_path / "index"), ["text"])
assert ldb.fts.populate_index(index, table, ["text"]) == len(table)