5 Commits

Author	SHA1	Message	Date
François Massot	0a23201338	Fix stackoverflow and add docs.	2023-07-03 22:05:11 +09:00
François Massot	81330aaf89	WIP	2023-07-03 22:05:10 +09:00
PSeitz	fdecb79273	tokenizer-api: reduce Tokenizer overhead (#2062) ... * tokenizer-api: reduce Tokenizer overhead Previously a new `Token` for each text encountered was created, which contains `String::with_capacity(200)` In the new API the token_stream gets mutable access to the tokenizer, this allows state to be shared (in this PR Token is shared). Ideally the allocation for the BoxTokenStream would also be removed, but this may require some lifetime tricks. * simplify api * move lowercase and ascii folding buffer to global * empty Token text as default	2023-06-08 18:37:58 +08:00
Pascal Seitz	1e4df54ab3	fix clippy	2021-07-01 17:41:53 +02:00
Paul Masurel	2f14a892ca	added a simple bench for the default analyzer	2021-01-06 19:11:26 +09:00