mirror of https://github.com/lancedb/lancedb.git synced 2026-08-26 07:58:31 +00:00

Files

T

Yang Cen f7feed48c3 feat(fts): support custom stop-word lists (#3734 )

## What

Expose custom FTS stop-word lists in the Python and TypeScript public
APIs, including their standalone tokenize helpers and remote index
creation.

This PR supports concrete string lists only. It does not add file or
LanceDB-table stop-word sources.

## Why

Rust already exposes Lance's custom stop-word list option. The Python
and TypeScript APIs did not pass it through, and local index details did
not retain the full tokenizer parameters needed by index-backed
tokenization after reopening a table.

## How

- Add `custom_stop_words` / `customStopWords` to the Python and
TypeScript FTS and tokenize options.
- Preserve `None` / `undefined`, empty lists, and list contents without
normalization.
- Load the persisted FTS segment parameters when returning local index
details.
- Serialize the concrete list in remote create-index requests.
- Keep Python and TypeScript tests thin; behavior, persistence, query
tokenization, and remote JSON coverage live primarily in Rust.

## Validation

- `cargo check --quiet --features remote --tests --examples`
- `cargo clippy --quiet --features remote --tests --examples`
- `cargo test --quiet --features remote --tests`
- Python extension rebuild with `uv` and `maturin`
- Targeted Python tests: 4 passed
- Python `ruff format --check` and `ruff check`
- TypeScript build, typecheck, Biome lint, generated docs, and targeted
tests

---------

Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>

2026-07-29 17:40:12 +08:00

1.5 KiB

Raw Blame History

@lancedb/lancedb • Docs

@lancedb/lancedb / TokenizeOptions

Interface: TokenizeOptions

Options for tokenizing a full-text search query without a table index.

Properties

asciiFolding?

optional asciiFolding: boolean;

Whether to fold ASCII characters.

baseTokenizer?

optional baseTokenizer: BaseTokenizer;

The tokenizer to use. The default is "simple".

customStopWords?

optional customStopWords: string[];

Custom stop words that replace the built-in list for language.

This option only affects tokenization when removeStopWords is true.

undefined keeps the built-in language list. An empty array explicitly replaces it with no stop words.

language?

optional language: string;

Language for stemming and stop words.

lowercase?

optional lowercase: boolean;

Whether to lowercase tokens.

maxTokenLength?

optional maxTokenLength: number;

Maximum token length; tokens longer than this are ignored.

ngramMaxLength?

optional ngramMaxLength: number;

N-gram maximum length.

ngramMinLength?

optional ngramMinLength: number;

N-gram minimum length.

prefixOnly?

optional prefixOnly: boolean;

Whether to only emit token prefixes for the n-gram tokenizer.

removeStopWords?

optional removeStopWords: boolean;

Whether to remove stop words.

stem?

optional stem: boolean;

Whether to stem tokens.

1.5 KiB Raw Blame History