feat: add table FTS query tokenization (#3659)

## Summary
- add table-level FTS query tokenization returning token text and
position
- use the native index tokenizer for local tables and remote index
metadata for remote tables
- expose sync and async Python table wrappers with focused coverage

docs/src/js/classes/Table.md

@@ -934,6 +934,32 @@ Return the table as an arrow table
 
 ***
 
+### tokenize()
+
+```ts
+abstract tokenize(query, options): Promise<FtsToken[]>
+```
+
+Tokenize a full-text search query using the tokenizer configured on an FTS index.
+
+Specify exactly one of `column` or `indexName`.
+
+Model-backed tokenizers such as `jieba/*` and `lindera/*` are rebuilt in
+the client process from index metadata. For remote tables, this means the
+same tokenizer model files must also exist locally.
+
+#### Parameters
+
+* **query**: `string`
+
+* **options**: [`TokenizeTableOptions`](../type-aliases/TokenizeTableOptions.md)
+
+#### Returns
+
+`Promise`&lt;[`FtsToken`](../interfaces/FtsToken.md)[]&gt;
+
+***
+
 ### unsetLsmWriteSpec()
 
 ```ts

docs/src/js/functions/tokenize.md

@@ -0,0 +1,26 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / tokenize
+
+# Function: tokenize()
+
+```ts
+function tokenize(query, options?): Promise<FtsToken[]>
+```
+
+Tokenize a full-text search query using an explicit tokenizer.
+
+This does not require a table or FTS index. The tokenizer options match
+[Index.fts](../classes/Index.md#fts).
+
+## Parameters
+
+* **query**: `string`
+
+* **options?**: `Partial`&lt;[`TokenizeOptions`](../interfaces/TokenizeOptions.md)&gt;
+
+## Returns
+
+`Promise`&lt;[`FtsToken`](../interfaces/FtsToken.md)[]&gt;

docs/src/js/globals.md

@@ -72,6 +72,7 @@
 - [FragmentStatistics](interfaces/FragmentStatistics.md)
 - [FragmentSummaryStats](interfaces/FragmentSummaryStats.md)
 - [FtsOptions](interfaces/FtsOptions.md)
+- [FtsToken](interfaces/FtsToken.md)
 - [FullTextQuery](interfaces/FullTextQuery.md)
 - [FullTextSearchOptions](interfaces/FullTextSearchOptions.md)
 - [HnswPqOptions](interfaces/HnswPqOptions.md)
@@ -107,6 +108,7 @@
 - [TimeoutConfig](interfaces/TimeoutConfig.md)
 - [TlsConfig](interfaces/TlsConfig.md)
 - [TokenResponse](interfaces/TokenResponse.md)
+- [TokenizeOptions](interfaces/TokenizeOptions.md)
 - [UpdateFieldMetadataResult](interfaces/UpdateFieldMetadataResult.md)
 - [UpdateOptions](interfaces/UpdateOptions.md)
 - [UpdateResult](interfaces/UpdateResult.md)
@@ -116,6 +118,7 @@
 
 ## Type Aliases
 
+- [BaseTokenizer](type-aliases/BaseTokenizer.md)
 - [Data](type-aliases/Data.md)
 - [DataLike](type-aliases/DataLike.md)
 - [FieldLike](type-aliases/FieldLike.md)
@@ -125,6 +128,7 @@
 - [RecordBatchLike](type-aliases/RecordBatchLike.md)
 - [SchemaLike](type-aliases/SchemaLike.md)
 - [TableLike](type-aliases/TableLike.md)
+- [TokenizeTableOptions](type-aliases/TokenizeTableOptions.md)
 
 ## Functions
 
@@ -135,3 +139,4 @@
 - [makeArrowTable](functions/makeArrowTable.md)
 - [packBits](functions/packBits.md)
 - [permutationBuilder](functions/permutationBuilder.md)
+- [tokenize](functions/tokenize.md)

docs/src/js/interfaces/FtsOptions.md

@@ -23,7 +23,7 @@ whether to remove punctuation
 ### baseTokenizer?
 
 ```ts
-optional baseTokenizer: "raw" | "simple" | "whitespace" | "ngram";
+optional baseTokenizer: BaseTokenizer;
 ```
 
 The tokenizer to use when building the index.
@@ -37,6 +37,10 @@ The following tokenizers are available:
 
 "raw" - Raw tokenizer. This tokenizer does not split the text into tokens and indexes the entire text as a single token.
 
+"icu" - ICU dictionary-based word segmentation.
+
+"icu/split" - ICU segmentation with simple-style delimiter splitting.
+
 ***
 
 ### language?

docs/src/js/interfaces/FtsToken.md

@@ -0,0 +1,29 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / FtsToken
+
+# Interface: FtsToken
+
+Token produced by the tokenizer configured on a full-text search index.
+
+## Properties
+
+### position
+
+```ts
+position: number;
+```
+
+Token position used by full-text query matching.
+
+***
+
+### text
+
+```ts
+text: string;
+```
+
+Token text after tokenizer filters have been applied.

docs/src/js/interfaces/TokenizeOptions.md

@@ -0,0 +1,109 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / TokenizeOptions
+
+# Interface: TokenizeOptions
+
+Options for tokenizing a full-text search query without a table index.
+
+## Properties
+
+### asciiFolding?
+
+```ts
+optional asciiFolding: boolean;
+```
+
+Whether to fold ASCII characters.
+
+***
+
+### baseTokenizer?
+
+```ts
+optional baseTokenizer: BaseTokenizer;
+```
+
+The tokenizer to use. The default is "simple".
+
+***
+
+### language?
+
+```ts
+optional language: string;
+```
+
+Language for stemming and stop words.
+
+***
+
+### lowercase?
+
+```ts
+optional lowercase: boolean;
+```
+
+Whether to lowercase tokens.
+
+***
+
+### maxTokenLength?
+
+```ts
+optional maxTokenLength: number;
+```
+
+Maximum token length; tokens longer than this are ignored.
+
+***
+
+### ngramMaxLength?
+
+```ts
+optional ngramMaxLength: number;
+```
+
+N-gram maximum length.
+
+***
+
+### ngramMinLength?
+
+```ts
+optional ngramMinLength: number;
+```
+
+N-gram minimum length.
+
+***
+
+### prefixOnly?
+
+```ts
+optional prefixOnly: boolean;
+```
+
+Whether to only emit token prefixes for the n-gram tokenizer.
+
+***
+
+### removeStopWords?
+
+```ts
+optional removeStopWords: boolean;
+```
+
+Whether to remove stop words.
+
+***
+
+### stem?
+
+```ts
+optional stem: boolean;
+```
+
+Whether to stem tokens.

docs/src/js/type-aliases/BaseTokenizer.md

@@ -0,0 +1,19 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / BaseTokenizer
+
+# Type Alias: BaseTokenizer
+
+```ts
+type BaseTokenizer:
+  | "simple"
+  | "whitespace"
+  | "raw"
+  | "ngram"
+  | "icu"
+  | "icu/split"
+  | `jieba/${string}`
+  | `lindera/${string}`;
+```

docs/src/js/type-aliases/TokenizeTableOptions.md

@@ -0,0 +1,11 @@
+[**@lancedb/lancedb**](../README.md) • **Docs**
+
+***
+
+[@lancedb/lancedb](../globals.md) / TokenizeTableOptions
+
+# Type Alias: TokenizeTableOptions
+
+```ts
+type TokenizeTableOptions: object | object;
+```