rust/tantivy
1
0
Fork 0
mirror of https://github.com/quickwit-oss/tantivy.git synced 2026-01-06 09:12:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1994 from quickwit-oss/fmassot/expose-simple-token-stream

Browse Source 
Expose `SimpleTokenStream` to use it in quickwit for the multilanguage tokenizer
This commit is contained in:
François Massot
2023-04-13 18:55:02 +02:00
committed by GitHub
parent 205e8a0a92 64bce340b2
commit 36138c493b
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
src/tokenizer/mod.rs
View File

@@ -149,7 +149,7 @@ pub use self::ngram_tokenizer::NgramTokenizer;
pub use self::raw_tokenizer::RawTokenizer;
pub use self::regex_tokenizer::RegexTokenizer;
pub use self::remove_long::RemoveLongFilter;
pub use self::simple_tokenizer::SimpleTokenizer;
pub use self::simple_tokenizer::{SimpleTokenStream, SimpleTokenizer};
pub use self::split_compound_words::SplitCompoundWords;
pub use self::stemmer::{Language, Stemmer};
pub use self::stop_word_filter::StopWordFilter;