Paul Masurel
811fd0cb9e
Dynamic analyzer ( #755 )
...
* Removed generics in tokenizers
* lowercaser
* Added TokenizerExt
* Introducing BoxedTokenizer
* Introducing BoxXXXXX helper struct
* Closes #762 .
* Introducing a TextAnalyzer
2020-01-29 18:23:37 +09:00
Paul Masurel
dac50c6aeb
Dds merged ( #539 )
...
* add ascii folding support
* Minor change and added Changelog.
* add additional tests
* Add tests for ascii folding (#533 )
* first tests for ascii folding
* use a `RawTokenizer` for tokens using punctuation
* add test for all (?) folding, inspired by Lucene
* Simplification of the unit test code
2019-04-26 10:25:08 +09:00
Paul Masurel
78673172d0
Cargo fmt
2018-04-21 20:05:36 +09:00
Paul Masurel
1e55189db1
NOBUG rustfmt
2017-12-14 19:30:31 +09:00
Paul Masurel
8b1b389a76
NOBUG Clippy
2017-12-14 19:25:12 +09:00
Paul Masurel
f24e5f405e
NOBUG intellij misc lint
2017-12-14 18:23:35 +09:00
Paul Masurel
f30ec9b36b
Merge branch 'master' of github.com:tantivy-search/tantivy
...
Conflicts:
src/analyzer/mod.rs
src/schema/index_record_option.rs
src/tokenizer/lower_caser.rs
src/tokenizer/tokenizer.rs
2017-11-26 10:54:05 +09:00
Paul Masurel
acd7c1ea2d
Added comments
2017-11-26 10:44:49 +09:00
Paul Masurel
ac4d433fad
Renamed analyzer to tokenizer
2017-11-24 16:50:32 +09:00
