tokenizer option on text fastfield (#1945)

* tokenizer option on text fastfield

allow to set tokenizer option on text fastfield (fixes #1901)
handle PreTokenized strings in fast field

* change visibility

* remove custom de/serialization
This commit is contained in:
PSeitz
2023-03-31 16:03:38 +08:00
committed by GitHub
parent 4cf93dab7d
commit 5c4ea6a708
6 changed files with 231 additions and 41 deletions

View File

@@ -42,7 +42,7 @@ fn main() -> tantivy::Result<()> {
.set_index_option(IndexRecordOption::WithFreqs)
.set_tokenizer("raw"),
)
.set_fast()
.set_fast(None)
.set_stored();
schema_builder.add_text_field("category", text_fieldtype);
schema_builder.add_f64_field("stock", FAST);