Yang Cen
f7feed48c3
## What Expose custom FTS stop-word lists in the Python and TypeScript public APIs, including their standalone tokenize helpers and remote index creation. This PR supports concrete string lists only. It does not add file or LanceDB-table stop-word sources. ## Why Rust already exposes Lance's custom stop-word list option. The Python and TypeScript APIs did not pass it through, and local index details did not retain the full tokenizer parameters needed by index-backed tokenization after reopening a table. ## How - Add `custom_stop_words` / `customStopWords` to the Python and TypeScript FTS and tokenize options. - Preserve `None` / `undefined`, empty lists, and list contents without normalization. - Load the persisted FTS segment parameters when returning local index details. - Serialize the concrete list in remote create-index requests. - Keep Python and TypeScript tests thin; behavior, persistence, query tokenization, and remote JSON coverage live primarily in Rust. ## Validation - `cargo check --quiet --features remote --tests --examples` - `cargo clippy --quiet --features remote --tests --examples` - `cargo test --quiet --features remote --tests` - Python extension rebuild with `uv` and `maturin` - Targeted Python tests: 4 passed - Python `ruff format --check` and `ruff check` - TypeScript build, typecheck, Biome lint, generated docs, and targeted tests --------- Co-authored-by: Yang Cen <yangcen@Yangs-Mac-mini.local>
2.6 KiB
@lancedb/lancedb • Docs
@lancedb/lancedb / FtsOptions
Interface: FtsOptions
Options to create a full text search index
Properties
asciiFolding?
optional asciiFolding: boolean;
whether to remove punctuation
baseTokenizer?
optional baseTokenizer: BaseTokenizer;
The tokenizer to use when building the index. The default is "simple".
The following tokenizers are available:
"simple" - Simple tokenizer. This tokenizer splits the text into tokens using whitespace and punctuation as a delimiter.
"whitespace" - Whitespace tokenizer. This tokenizer splits the text into tokens using whitespace as a delimiter.
"raw" - Raw tokenizer. This tokenizer does not split the text into tokens and indexes the entire text as a single token.
"icu" - ICU dictionary-based word segmentation.
"icu/split" - ICU segmentation with simple-style delimiter splitting.
blockSize?
optional blockSize: 128 | 256;
Number of documents per compressed posting block.
The default is 128. Supported values are 128 and 256. A value of 256 uses the experimental FTS V3 format and may introduce breaking changes.
customStopWords?
optional customStopWords: string[];
Custom stop words that replace the built-in list for language.
This option only affects tokenization when removeStopWords is true.
undefined keeps the built-in language list. An empty array explicitly
replaces it with no stop words.
language?
optional language: string;
language for stemming and stop words
this is only used when stem or remove_stop_words is true
lowercase?
optional lowercase: boolean;
whether to lowercase tokens
maxTokenLength?
optional maxTokenLength: number;
maximum token length tokens longer than this length will be ignored
ngramMaxLength?
optional ngramMaxLength: number;
ngram max length
ngramMinLength?
optional ngramMinLength: number;
ngram min length
prefixOnly?
optional prefixOnly: boolean;
whether to only index the prefix of the token for ngram tokenizer
removeStopWords?
optional removeStopWords: boolean;
whether to remove stop words
stem?
optional stem: boolean;
whether to stem tokens
withPosition?
optional withPosition: boolean;
Whether to build the index with positions. True by default. If set to false, the index will not store the positions of the tokens in the text, which will make the index smaller and faster to build, but will not support phrase queries.