pub fn tokenize(text: &str) -> Vec<&str>Expand description
Returns the tokens the fulltext bloom index stores for text.
A token is a maximal run of ASCII alphanumerics or of non-Han alphanumerics;
_ and other symbols only separate tokens. A run of Han characters is
emitted as its bigrams, or as itself when it has a single character, since
matches_term matches Han terms as plain substrings.