feat(python): expose the NGram scalar index

## What kind of change does this PR introduce?

Feature. Exposes the NGram scalar index in the Python SDK.

## What is the current behavior?

The NGram scalar index already exists in the Rust core (`Index::NGram`) and is accepted over the remote protocol (the `"NGRAM"` type), but the Python SDK has no way to create it. `lancedb.index` offers `BTree`, `Bitmap`, `LabelList`, `Fm`, and the vector indexes, but not `NGram`, so it cannot be built through `create_index(config=...)`.

## What is the new behavior?

Adds an `NGram` config to `lancedb.index` and wires it through the pyo3 binding:

```python
from lancedb.index import NGram

await table.create_index("text", config=NGram())
```

The NGram index accelerates substring filters such as `contains(col, 'needle')`, along with some `LIKE` and regular-expression predicates, on UTF-8 string columns. It is parameter free and uses Lance's default trigram parameters.

Changes:

- `lancedb.index`: new `NGram` dataclass, added to `__all__`.
- pyo3 binding (`python/src/index.rs`): a match arm mapping `NGram` to `LanceDbIndex::NGram`, plus the import and the error message.
- Type hints: `NGram` added to the `create_index` config unions and the isinstance check in `table.py`, the `IndexConfigType` union, and the `_lancedb.pyi` stubs; `"NGRAM"` added to the `IndexType` and `IndexStatistics.index_type` literals.
- Test: `test_create_ngram_index` covers creation, `list_indices`, `index_stats`, and that a substring filter uses the index.
- Docs: an NGram section in the scalar index guide.

This mirrors the approach used for the RTree index in https://github.com/lancedb/lancedb/pull/4555 (that PR keeps the RTree scope; this one adds the remaining parameter free string index).
This commit is contained in:
BHAWESHBHASKAR
2026-11-01 11:54:27 +08:00
committed by GitHub
parent e5d4e9bf6e
commit a9146d536a
8 changed files with 116 additions and 6 deletions
+1
View File
@@ -26,6 +26,7 @@ patterns. LanceDB supports three main index families:
| `Bitmap` (Scalar) | Low-cardinality columns with few thousand or fewer distinct values. Accelerates equality and range filters. | Stores a bitmap for each distinct value in the column, with one bit per row indicating presence. Memory-efficient for low-cardinality data. |
| `LabelList` (Scalar) | List columns (e.g., tags, categories, keywords) requiring `array_has_all` or `array_has_any` filters. | Scalar index for `List<T>` and `LargeList<T>` columns of primitive values, using an underlying bitmap index structure to enable fast array membership lookups. |
| `FM` (Scalar) | String or binary columns that need raw substring search. | FM-Index over `Utf8`, `LargeUtf8`, `Binary`, or `LargeBinary` data for filters such as `contains(path, 'needle')`. Use FTS instead for tokenized word search and BM25 ranking. |
| `NGram` (Scalar) | String columns that need substring search using character trigrams. | Trigram index over `Utf8` and `LargeUtf8` columns that narrows `contains(col, 'needle')` and some `LIKE` and regular expression predicates to rows sharing the query's trigrams, then rechecks the predicate. A query shorter than the trigram length cannot use it; use `FM` for arbitrary length substrings or `FTS` for tokenized word search. |
A deployment builds the vector indexes `IVF_FLAT`, `IVF_PQ`, `IVF_RQ` and
`IVF_HNSW_SQ`. `IVF_SQ`, `IVF_HNSW_FLAT` and `IVF_HNSW_PQ` are built only by the
+42 -1
View File
@@ -26,12 +26,13 @@ Scalar indexes accelerate filters on values such as numbers, categories and
strings. Use them for selective table queries or metadata filters combined
with vector and full-text search.
LanceDB supports four types of scalar indexes:
LanceDB supports five types of scalar indexes:
- `BTREE`: Stores column data in sorted order for binary search. Best for columns with many unique values.
- `BITMAP`: Uses bitmaps to track value presence. Ideal for columns with few unique values (e.g., categories, tags).
- `LABEL_LIST`: Special index for `List<T>` and `LargeList<T>` columns of primitive values supporting `array_has_all` and `array_has_any` queries.
- `FM`: FM-Index over string or binary columns that accelerates substring search via `contains(col, 'needle')`.
- `NGRAM`: Trigram index over UTF-8 string columns that accelerates substring search via `contains(col, 'needle')`, along with some `LIKE` and regular-expression filters.
<Tip title="Working with scalar indices">
Use scalar indexes with:
@@ -50,6 +51,7 @@ Use scalar indexes with:
| Boolean, numbers or strings with fewer than 1,000 unique values | `<`, `=`, `>`, `in`, `between`, `is null` | `BITMAP` |
| List of low cardinality of numbers or strings | `array_has_any`, `array_has_all` | `LABEL_LIST` |
| String or binary (`Utf8`, `LargeUtf8`, `Binary`, `LargeBinary`) | `contains(col, 'needle')` | `FM` |
| String (`Utf8`, `LargeUtf8`) | `contains(col, 'needle')`, `LIKE`, regex | `NGRAM` |
## Scalar Index Operations
@@ -167,6 +169,45 @@ table.search().where("contains(text, 'needle')").limit(10).to_pandas()
`list_indices()` reports the index type as `"Fm"`.
## NGram index for substring search
The `NGram` index is a scalar index over UTF-8 string columns. It tokenizes
values into overlapping character n-grams (using Lance's default trigram
parameters) and uses them to accelerate `contains(col, 'needle')`, along with
some `LIKE` and regular-expression filters. Unlike the `FM` index, which matches
arbitrary substrings of the raw bytes, the `NGram` index is trigram based, so a
query shorter than the n-gram length cannot be served by it.
### Create an NGram index
Build an NGram index with the async `create_index` API by passing the `NGram`
config in Python. In Rust, use `Index::NGram(NGramIndexBuilder::default())`.
<CodeGroup>
```python Python icon="python"
from lancedb.index import NGram
await tbl.create_index("text", config=NGram())
```
```rust Rust icon="rust"
use lancedb::index::Index;
use lancedb::index::scalar::NGramIndexBuilder;
tbl.create_index(&["text"], Index::NGram(NGramIndexBuilder::default()))
.execute()
.await?;
```
</CodeGroup>
After the index is built, substring filters use it automatically:
```python
table.search().where("contains(text, 'needle')").limit(10).to_pandas()
```
`list_indices()` reports the index type as `"NGram"`.
## Index UUID Columns
LanceDB supports scalar indexes on UUID columns (stored as `FixedSizeBinary(16)`), enabling efficient lookups and filtering on UUID-based primary keys.