Tomoko Uchida 74e36c7e97 Add unit tests for tokenizers and filters (#1156) ...

* add unit test for SimpleTokenizer
* add unit tests for tokenizers and filters.

2021-09-27 10:22:01 +09:00

..

alphanum_only.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

ascii_folding_filter.rs

fix clippy

2021-07-01 17:41:53 +02:00

facet_tokenizer.rs

Dynamic analyzer (#755)

2020-01-29 18:23:37 +09:00

lower_caser.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

mod.rs

Add WhitespaceTokenizer (#1147)

2021-08-29 18:20:49 +09:00

ngram_tokenizer.rs

fix clippy

2021-07-01 18:06:09 +02:00

raw_tokenizer.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

remove_long.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

simple_tokenizer.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

stemmer.rs

Removing serde-derive dependency (#786)

2020-03-06 23:33:58 +09:00

stop_word_filter.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00

token_stream_chain.rs

Dynamic analyzer (#755)

2020-01-29 18:23:37 +09:00

tokenized_string.rs

Removing serde-derive dependency (#786)

2020-03-06 23:33:58 +09:00

tokenizer_manager.rs

Add WhitespaceTokenizer (#1147)

2021-08-29 18:20:49 +09:00

tokenizer.rs

Removing serde-derive dependency (#786)

2020-03-06 23:33:58 +09:00

whitespace_tokenizer.rs

Add unit tests for tokenizers and filters (#1156)

2021-09-27 10:22:01 +09:00