Adam Reichold 5f7d027a52 Avoid unconditional allocation in StemmerTokenStream. ...

This fixes the TODO in two ways: If the stemmer already yields an owned string,
it is used directly as the new text of the token. Otherwise, a temporary buffer
is used to copy the stemmed text (just as before) and then swapping it into the
token to reuse its existing buffer.

2022-10-26 18:11:15 +02:00

..

alphanum_only.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

ascii_folding_filter.rs

Removes all usage of block_on, and use a oneshot channel instead. (#1315)

2022-03-18 16:54:58 +09:00

empty_tokenizer.rs

Added JSON Type (#1270)

2022-02-24 16:25:22 +09:00

facet_tokenizer.rs

Dynamic analyzer (#755)

2020-01-29 18:23:37 +09:00

lower_caser.rs

Clippy comments (#1316)

2022-03-17 18:57:55 +09:00

mod.rs

Improvements to doc linking, grammar, etc.

2022-09-19 18:10:22 +07:00

ngram_tokenizer.rs

Fix warnings when doc'ing private items. (#1579)

2022-10-03 14:24:00 +09:00

raw_tokenizer.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

remove_long.rs

Minor refactoring (#1266)

2022-01-28 15:55:55 +09:00

simple_tokenizer.rs

Minor refactoring (#1266)

2022-01-28 15:55:55 +09:00

stemmer.rs

Avoid unconditional allocation in StemmerTokenStream.

2022-10-26 18:11:15 +02:00

stop_word_filter.rs

Minor refactoring (#1266)

2022-01-28 15:55:55 +09:00

tokenized_string.rs

Improvements to doc linking, grammar, etc.

2022-09-19 18:10:22 +07:00

tokenizer_manager.rs

doc fix (#1391)

2022-06-21 15:53:33 +09:00

tokenizer.rs

Improvements to doc linking, grammar, etc.

2022-09-19 18:10:22 +07:00

whitespace_tokenizer.rs

Minor refactoring (#1266)

2022-01-28 15:55:55 +09:00