mirror of
https://github.com/lancedb/lancedb.git
synced 2026-10-12 23:02:28 +00:00
lancedb-gatefixer[bot]
3348087d09
TypeScript rejected `Index.fts({ baseTokenizer: "code" })` because the
public `BaseTokenizer` union omitted `"code"`, although the native
tokenizer already supports it.
Add `"code"` to the shared union, document its identifier behavior, and
regenerate the API reference. The regression test creates and searches a
code-tokenized FTS index with a typed configuration across Arrow
versions 15–18.
Validation:
- The regression failed before the fix with TS2322: `"code"` was not
assignable to `BaseTokenizer`.
- After the fix, all four cases pass: `pnpm test __test__/table.test.ts
--runInBand -t 'full text search code tokenizer' --verbose=false`.
- `pnpm build`, `pnpm lint`, `pnpm run docs`, and formatting of the
changed TypeScript files pass.
Fixes #4366
<!-- lance-gatekeeper-fix:v1 agent=8c37ac529cec8862bd5dbc218c5aaa8d
generation=1 -->
Co-authored-by: Gatefixer <313497061+lancedb-gatefixer[bot]@users.noreply.github.com>
168 lines
2.7 KiB
Markdown
168 lines
2.7 KiB
Markdown
[**@lancedb/lancedb**](../README.md) • **Docs**
|
|
|
|
***
|
|
|
|
[@lancedb/lancedb](../globals.md) / FtsOptions
|
|
|
|
# Interface: FtsOptions
|
|
|
|
Options to create a full text search index
|
|
|
|
## Properties
|
|
|
|
### asciiFolding?
|
|
|
|
```ts
|
|
optional asciiFolding: boolean;
|
|
```
|
|
|
|
whether to remove punctuation
|
|
|
|
***
|
|
|
|
### baseTokenizer?
|
|
|
|
```ts
|
|
optional baseTokenizer: BaseTokenizer;
|
|
```
|
|
|
|
The tokenizer to use when building the index.
|
|
The default is "simple".
|
|
|
|
The following tokenizers are available:
|
|
|
|
"simple" - Simple tokenizer. This tokenizer splits the text into tokens using whitespace and punctuation as a delimiter.
|
|
|
|
"whitespace" - Whitespace tokenizer. This tokenizer splits the text into tokens using whitespace as a delimiter.
|
|
|
|
"raw" - Raw tokenizer. This tokenizer does not split the text into tokens and indexes the entire text as a single token.
|
|
|
|
"code" - Code tokenizer. This tokenizer keeps snake_case and camelCase identifiers as single tokens.
|
|
|
|
"icu" - ICU dictionary-based word segmentation.
|
|
|
|
"icu/split" - ICU segmentation with simple-style delimiter splitting.
|
|
|
|
***
|
|
|
|
### blockSize?
|
|
|
|
```ts
|
|
optional blockSize: 128 | 256;
|
|
```
|
|
|
|
Number of documents per compressed posting block.
|
|
|
|
The default is 128. Supported values are 128 and 256. A value of 256 uses
|
|
the experimental FTS V3 format and may introduce breaking changes.
|
|
|
|
***
|
|
|
|
### customStopWords?
|
|
|
|
```ts
|
|
optional customStopWords: string[];
|
|
```
|
|
|
|
Custom stop words that replace the built-in list for `language`.
|
|
|
|
This option only affects tokenization when `removeStopWords` is true.
|
|
|
|
`undefined` keeps the built-in language list. An empty array explicitly
|
|
replaces it with no stop words.
|
|
|
|
***
|
|
|
|
### language?
|
|
|
|
```ts
|
|
optional language: string;
|
|
```
|
|
|
|
language for stemming and stop words
|
|
this is only used when `stem` or `remove_stop_words` is true
|
|
|
|
***
|
|
|
|
### lowercase?
|
|
|
|
```ts
|
|
optional lowercase: boolean;
|
|
```
|
|
|
|
whether to lowercase tokens
|
|
|
|
***
|
|
|
|
### maxTokenLength?
|
|
|
|
```ts
|
|
optional maxTokenLength: number;
|
|
```
|
|
|
|
maximum token length
|
|
tokens longer than this length will be ignored
|
|
|
|
***
|
|
|
|
### ngramMaxLength?
|
|
|
|
```ts
|
|
optional ngramMaxLength: number;
|
|
```
|
|
|
|
ngram max length
|
|
|
|
***
|
|
|
|
### ngramMinLength?
|
|
|
|
```ts
|
|
optional ngramMinLength: number;
|
|
```
|
|
|
|
ngram min length
|
|
|
|
***
|
|
|
|
### prefixOnly?
|
|
|
|
```ts
|
|
optional prefixOnly: boolean;
|
|
```
|
|
|
|
whether to only index the prefix of the token for ngram tokenizer
|
|
|
|
***
|
|
|
|
### removeStopWords?
|
|
|
|
```ts
|
|
optional removeStopWords: boolean;
|
|
```
|
|
|
|
whether to remove stop words
|
|
|
|
***
|
|
|
|
### stem?
|
|
|
|
```ts
|
|
optional stem: boolean;
|
|
```
|
|
|
|
whether to stem tokens
|
|
|
|
***
|
|
|
|
### withPosition?
|
|
|
|
```ts
|
|
optional withPosition: boolean;
|
|
```
|
|
|
|
Whether to build the index with positions.
|
|
True by default.
|
|
If set to false, the index will not store the positions of the tokens in the text,
|
|
which will make the index smaller and faster to build, but will not support phrase queries.
|