PSeitz fdecb79273 tokenizer-api: reduce Tokenizer overhead (#2062) ...

* tokenizer-api: reduce Tokenizer overhead

Previously a new `Token` for each text encountered was created, which
contains `String::with_capacity(200)`
In the new API the token_stream gets mutable access to the tokenizer,
this allows state to be shared (in this PR Token is shared).
Ideally the allocation for the BoxTokenStream would also be removed, but
this may require some lifetime tricks.

* simplify api

* move lowercase and ascii folding buffer to global

* empty Token text as default

2023-06-08 18:37:58 +08:00

..

aggregation

allow histogram bounds to be passed as Rfc3339 (#2076)

2023-06-08 09:07:08 +02:00

collector

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

core

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

directory

Added support for madvise (#2036)

2023-05-11 05:39:17 +02:00

fastfield

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

fieldnorm

Make BM25 scoring more flexible (#1855)

2023-02-16 19:14:12 +09:00

indexer

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

positions

fix clippy (#1725)

2022-12-20 07:30:06 +01:00

postings

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

query

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

reader

fix doc store cache docs (#1821)

2023-01-23 07:06:49 +01:00

schema

Readding s in datetime precision variant names (#2065)

2023-06-01 06:39:46 +02:00

snippet

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

space_usage

add memory limit for aggregations (#1942)

2023-03-16 06:21:07 +01:00

store

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

termdict

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

tokenizer

tokenizer-api: reduce Tokenizer overhead (#2062)

2023-06-08 18:37:58 +08:00

docset.rs

work in batches of docs (#1937)

2023-03-21 06:57:44 +01:00

error.rs

Inline format arguments where makes sense (#2038)

2023-05-10 18:03:59 +09:00

functional_test.rs

set docstore cache size at construction

2022-07-04 14:27:55 +08:00

future_result.rs

Removes all usage of block_on, and use a oneshot channel instead. (#1315)

2022-03-18 16:54:58 +09:00

lib.rs

Rename DatePrecision to DateTimePrecision (#2051)

2023-05-23 17:09:11 +02:00

macros.rs

Adding an IndexingContext object (#1268)

2022-02-04 15:08:01 +09:00