Dru Sellers e75bb1d6a1 Fix NGram processing of non-ascii characters (#430) ...

* A working version

* optimize the ngram parsing

* Decoding codepoint only once.

* Closes #429

* using leading_zeros to make code less cryptic

* lookup in a table

2018-10-31 08:35:27 +09:00

..

alphanum_only.rs

Implement StopWords Filter (#292)

2018-05-09 18:40:41 -07:00

facet_tokenizer.rs

Remove some unsafe stuff, justified some of it.

2018-05-07 23:57:53 -07:00

lower_caser.rs

Cargo Format (#420)

2018-09-15 07:44:22 +09:00

mod.rs

Fix NGram processing of non-ascii characters (#430)

2018-10-31 08:35:27 +09:00

ngram_tokenizer.rs

Fix NGram processing of non-ascii characters (#430)

2018-10-31 08:35:27 +09:00

raw_tokenizer.rs

Cargo Format (#420)

2018-09-15 07:44:22 +09:00

remove_long.rs

Implement StopWords Filter (#292)

2018-05-09 18:40:41 -07:00

simple_tokenizer.rs

Cargo fmt

2018-04-21 20:05:36 +09:00

stemmer.rs

Clippy (#422)

2018-09-15 20:20:22 +09:00

stop_word_filter.rs

Add default EN stopwords to the default analyzer (#381)

2018-08-22 10:49:39 +09:00

token_stream_chain.rs

Cargo Format (#420)

2018-09-15 07:44:22 +09:00

tokenizer_manager.rs

Merge branch 'issue/368b'

2018-09-11 20:16:14 +09:00

tokenizer.rs

Cargo Format (#420)

2018-09-15 07:44:22 +09:00