PSeitz fdecb79273 tokenizer-api: reduce Tokenizer overhead (#2062) ...

* tokenizer-api: reduce Tokenizer overhead

Previously a new `Token` for each text encountered was created, which
contains `String::with_capacity(200)`
In the new API the token_stream gets mutable access to the tokenizer,
this allows state to be shared (in this PR Token is shared).
Ideally the allocation for the BoxTokenStream would also be removed, but
this may require some lifetime tricks.

* simplify api

* move lowercase and ascii folding buffer to global

* empty Token text as default

2023-06-08 18:37:58 +08:00

..

aggregation.rs

switch to Aggregation without serde_untagged (#2003)

2023-04-25 08:54:51 +02:00

basic_search.rs

re-enable examples (#1860)

2023-02-10 14:51:37 +01:00

custom_collector.rs

Renaming .values(DocId) to .values_for_doc(DocId) (#1906)

2023-02-27 12:15:13 +09:00

custom_tokenizer.rs

update examples for literate docs (#1880)

2023-02-17 11:48:22 +01:00

date_time_field.rs

Readding s in datetime precision variant names (#2065)

2023-06-01 06:39:46 +02:00

deleting_updating_documents.rs

re-enable examples (#1860)

2023-02-10 14:51:37 +01:00

faceted_search_with_tweaked_score.rs

update examples for literate docs (#1880)

2023-02-17 11:48:22 +01:00

faceted_search.rs

re-enable examples (#1860)

2023-02-10 14:51:37 +01:00

fuzzy_search.rs

Clippy warnings fixes (#1885)

2023-02-20 19:04:13 +09:00

index_from_multiple_threads.rs

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

index_with_json.rs

update examples for literate docs (#1880)

2023-02-17 11:48:22 +01:00

integer_range_search.rs

re-enable examples (#1860)

2023-02-10 14:51:37 +01:00

ip_field.rs

update examples for literate docs (#1880)

2023-02-17 11:48:22 +01:00

iterating_docs_and_positions.rs

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

json_field.rs

re-enable examples (#1860)

2023-02-10 14:51:37 +01:00

phrase_prefix_search.rs

Expose phrase-prefix queries via the built-in query parser (#2044)

2023-06-01 13:03:16 +02:00

pre_tokenized_text.rs

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

snippet.rs

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

stop_words.rs

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

warmer.rs

update examples for literate docs (#1880)

2023-02-17 11:48:22 +01:00