rust/tantivy
1
0
Fork 0
mirror of https://github.com/quickwit-oss/tantivy.git synced 2026-05-18 01:00:40 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,991 Commits 241 Branches 81 Tags
ccb09aaa83fb6f285786c731b043f6728dfdab6a
Commit Graph

4 Commits

Author SHA1 Message Date
PSeitz
e56addc63e enable tokenizer on json fields (#2053) 
* enable tokenizer on json fields

enable tokenizer on json fields for type text

* Avoid making the tokenizer within the TextAnalyzer pub(crate)

* Moving BoxableTokenizer to tantivy.

---------

Co-authored-by: Paul Masurel <paul@quickwit.io>
 2023-05-24 10:47:39 +02:00
trinity-1686a
064518156f refactor tokenization pipeline to use GATs (#1924) 
* refactor tokenization pipeline to use GATs

* fix doctests

* fix clippy lints

* remove commented code
 2023-03-09 09:39:37 +01:00
Adrien Guillo
f3621c0487 Add license to tokenizer-api crate (#1778) 2023-01-11 05:26:41 +01:00
PSeitz
514d23a20c move tokenizer API to seperate crate (#1767) 
closes #1766

Finding tantivy tokenizers is a frustrating experience currently, since
they need be updated for each tantivy version. That's unnecessary since
the API is rather stable anyway.
 2023-01-09 06:37:38 +01:00