mirror of https://github.com/lancedb/lancedb.git synced 2026-11-14 22:56:54 +00:00

Files

T

Jack Ye 06b53c97d6 feat: add table FTS query tokenization (#3659 )

## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage

2026-07-14 10:59:33 -07:00

1.2 KiB

Raw Blame History

@lancedb/lancedb • Docs

@lancedb/lancedb / TokenizeOptions

Interface: TokenizeOptions

Options for tokenizing a full-text search query without a table index.

Properties

asciiFolding?

optional asciiFolding: boolean;

Whether to fold ASCII characters.

baseTokenizer?

optional baseTokenizer: BaseTokenizer;

The tokenizer to use. The default is "simple".

language?

optional language: string;

Language for stemming and stop words.

lowercase?

optional lowercase: boolean;

Whether to lowercase tokens.

maxTokenLength?

optional maxTokenLength: number;

Maximum token length; tokens longer than this are ignored.

ngramMaxLength?

optional ngramMaxLength: number;

N-gram maximum length.

ngramMinLength?

optional ngramMinLength: number;

N-gram minimum length.

prefixOnly?

optional prefixOnly: boolean;

Whether to only emit token prefixes for the n-gram tokenizer.

removeStopWords?

optional removeStopWords: boolean;

Whether to remove stop words.

stem?

optional stem: boolean;

Whether to stem tokens.

1.2 KiB Raw Blame History