mirror of
https://github.com/lancedb/lancedb.git
synced 2026-11-14 06:36:54 +00:00
feat(python): expose the NGram scalar index
## What kind of change does this PR introduce?
Feature. Exposes the NGram scalar index in the Python SDK.
## What is the current behavior?
The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.
## What is the new behavior?
Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:
```python
from lancedb.index import NGram
await table.create_index("text", config=NGram())
```
The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.
Changes:
- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.
This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
This commit is contained in:
@@ -26,6 +26,7 @@ patterns. LanceDB supports three main index families:
|
||||
| `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
|
||||
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_has_all` or `array_has_any` filters. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
|
||||
| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |
|
||||
| `NGram` (Scalar) | String columns that need substring search using character trigrams. | Trigram index over `Utf8` and `LargeUtf8` columns that narrows `contains(col, 'needle')` and some `LIKE` and regular expression predicates to rows sharing the query's trigrams, then rechecks the predicate. A query shorter than the trigram length cannot use it; use `FM` for arbitrary length substrings or `FTS` for tokenized word search. |
|
||||
|
||||
A deployment builds the vector indexes `IVF_FLAT`, `IVF_PQ`, `IVF_RQ` and
|
||||
`IVF_HNSW_SQ`. `IVF_SQ`, `IVF_HNSW_FLAT` and `IVF_HNSW_PQ` are built only by the
|
||||
|
||||
@@ -26,12 +26,13 @@ Scalar indexes accelerate filters on values such as numbers, categories and
|
||||
strings. Use them for selective table queries or metadata filters combined
|
||||
with vector and full-text search.
|
||||
|
||||
LanceDB supports four types of scalar indexes:
|
||||
LanceDB supports five types of scalar indexes:
|
||||
|
||||
- `BTREE`: Stores column data in sorted order for binary search. Best for columns with many unique values.
|
||||
- `BITMAP`: Uses bitmaps to track value presence. Ideal for columns with few unique values (e.g., categories, tags).
|
||||
- `LABEL_LIST`: Special index for `List<T>` and `LargeList<T>` columns of primitive values supporting `array_has_all` and `array_has_any` queries.
|
||||
- `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.
|
||||
- `NGRAM`: Trigram index over UTF-8 string columns that accelerates substring search via `contains(col, 'needle')`, along with some `LIKE` and regular-expression filters.
|
||||
|
||||
<Tip title="Working with scalar indices">
|
||||
Use scalar indexes with:
|
||||
@@ -50,6 +51,7 @@ Use scalar indexes with:
|
||||
| Boolean, numbers or strings with fewer than 1,000 unique values | `<`, `=`, `>`, `in`, `between`, `is null` | `BITMAP` |
|
||||
| List of low cardinality of numbers or strings | `array_has_any`, `array_has_all` | `LABEL_LIST` |
|
||||
| String or binary (`Utf8`, `LargeUtf8`, `Binary`, `LargeBinary`) | `contains(col, 'needle')` | `FM` |
|
||||
| String (`Utf8`, `LargeUtf8`) | `contains(col, 'needle')`, `LIKE`, regex | `NGRAM` |
|
||||
|
||||
## Scalar Index Operations
|
||||
|
||||
@@ -167,6 +169,45 @@ table.search().where("contains(text, 'needle')").limit(10).to_pandas()
|
||||
|
||||
`list_indices()` reports the index type as `"Fm"`.
|
||||
|
||||
## NGram index for substring search
|
||||
|
||||
The `NGram` index is a scalar index over UTF-8 string columns. It tokenizes
|
||||
values into overlapping character n-grams (using Lance's default trigram
|
||||
parameters) and uses them to accelerate `contains(col, 'needle')`, along with
|
||||
some `LIKE` and regular-expression filters. Unlike the `FM` index, which matches
|
||||
arbitrary substrings of the raw bytes, the `NGram` index is trigram based, so a
|
||||
query shorter than the n-gram length cannot be served by it.
|
||||
|
||||
### Create an NGram index
|
||||
|
||||
Build an NGram index with the async `create_index` API by passing the `NGram`
|
||||
config in Python. In Rust, use `Index::NGram(NGramIndexBuilder::default())`.
|
||||
|
||||
<CodeGroup>
|
||||
```python Python icon="python"
|
||||
from lancedb.index import NGram
|
||||
|
||||
await tbl.create_index("text", config=NGram())
|
||||
```
|
||||
|
||||
```rust Rust icon="rust"
|
||||
use lancedb::index::Index;
|
||||
use lancedb::index::scalar::NGramIndexBuilder;
|
||||
|
||||
tbl.create_index(&["text"], Index::NGram(NGramIndexBuilder::default()))
|
||||
.execute()
|
||||
.await?;
|
||||
```
|
||||
</CodeGroup>
|
||||
|
||||
After the index is built, substring filters use it automatically:
|
||||
|
||||
```python
|
||||
table.search().where("contains(text, 'needle')").limit(10).to_pandas()
|
||||
```
|
||||
|
||||
`list_indices()` reports the index type as `"NGram"`.
|
||||
|
||||
## Index UUID Columns
|
||||
|
||||
LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`), enabling efficient lookups and filtering on UUID-based primary keys.
|
||||
|
||||
Reference in New Issue
Block a user