mirror of https://github.com/lancedb/lancedb.git synced 2026-08-26 07:58:31 +00:00

Files

T

Jack Ye 06b53c97d6 feat: add table FTS query tokenization (#3659 )

## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage

2026-07-14 10:59:33 -07:00

565 B

Raw Blame History

@lancedb/lancedb • Docs

@lancedb/lancedb / tokenize

Function: tokenize()

function tokenize(query, options?): Promise<FtsToken[]>

Tokenize a full-text search query using an explicit tokenizer.

This does not require a table or FTS index. The tokenizer options match Index.fts.

Parameters

query: string
options?: Partial<TokenizeOptions>

Returns

Promise<FtsToken[]>

565 B Raw Blame History

Function: tokenize()

Parameters

Returns

565 B

Raw Blame History