[**@lancedb/lancedb**](../README.md) • **Docs**

***

[@lancedb/lancedb](../globals.md) / BaseTokenizer

# Type Alias: BaseTokenizer

```ts
type BaseTokenizer:
  | "simple"
  | "whitespace"
  | "raw"
  | "ngram"
  | "icu"
  | "icu/split"
  | `jieba/${string}`
  | `lindera/${string}`;
```
