trinity-1686a 064518156f refactor tokenization pipeline to use GATs (#1924) ...

* refactor tokenization pipeline to use GATs

* fix doctests

* fix clippy lints

* remove commented code

2023-03-09 09:39:37 +01:00

..

stop_word_filter

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

alphanum_only.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

ascii_folding_filter.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

empty_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

facet_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

lower_caser.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

mod.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

ngram_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

raw_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

regex_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

remove_long.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

simple_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

split_compound_words.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

stemmer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

tokenized_string.rs

Improvements to doc linking, grammar, etc.

2022-09-19 18:10:22 +07:00

tokenizer_manager.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00

whitespace_tokenizer.rs

refactor tokenization pipeline to use GATs (#1924)

2023-03-09 09:39:37 +01:00