Module term_token

Expand description

Word boundaries shared by matches_term and the fulltext bloom index.

Bloom pruning is only sound if the index splits text exactly where matches_term sees word boundaries, so both use classify_char, and the query side derives its probes with extract_term_probes and extract_like_probes.

Enums§

CharClass: Classes of characters that decide where matches_term sees a word boundary.

classify_char: Returns the CharClass of c. tokenize splits text wherever the class changes and at every CharClass::Other character.
extract_like_probes: Returns tokens that every text satisfying text LIKE pattern contains, as produced by tokenize. \ escapes the next character, following arrow’s like kernel; a trailing \ is a literal backslash.
extract_term_probes: Returns tokens that every text satisfying matches_term(text, term) contains, as produced by tokenize.
for_each_word_run 🔒: Calls f on every maximal run of word characters of the same class.
has_irregular_lowercase: Returns whether lowercasing text as a whole can produce tokens that lowercasing each token of text doesn’t, which matches_term(lower(col), ..) and lower(col) LIKE .. rely on. Every other character lowercases to one character of the same class, but U+0130 and U+212A change length or class, and Σ lowercases depending on the characters after it.
is_han 🔒
push_han_tokens 🔒
push_literal_probes 🔒: Pushes the tokens of literal that stay whole tokens wherever literal occurs. A side is bounded when the character beyond it can’t extend the adjacent run.
tokenize: Returns the tokens the fulltext bloom index stores for text.