greptimedb/common_base/term_token/index.html

<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="Word boundaries shared by `matches_term` and the fulltext bloom index."><title>common_base::term_token - Rust</title><script>if(window.location.protocol!=="file:")document.head.insertAdjacentHTML("beforeend","SourceSerif4-Regular-6b053e98.ttf.woff2,FiraSans-Italic-81dc35de.woff2,FiraSans-Regular-0fe48ade.woff2,FiraSans-MediumItalic-ccf7e434.woff2,FiraSans-Medium-e1aa3f0a.woff2,SourceCodePro-Regular-8badfe75.ttf.woff2,SourceCodePro-Semibold-aa29a496.ttf.woff2".split(",").map(f=>`<link rel="preload" as="font" type="font/woff2"href="../../static.files/${f}">`).join(""))</script><link rel="stylesheet" href="../../static.files/normalize-9960930a.css"><link rel="stylesheet" href="../../static.files/rustdoc-1ed95124.css"><meta name="rustdoc-vars" data-root-path="../../" data-static-root-path="../../static.files/" data-current-crate="common_base" data-themes="" data-resource-suffix="" data-rustdoc-version="1.98.1 (48a229cea 2026-09-01)" data-channel="1.98.1" data-search-js="search-9944d736.js" data-stringdex-js="stringdex-2da4960a.js" data-settings-js="settings-170eb4bf.js" ><script src="../../static.files/storage-41dd4d93.js"></script><script defer src="../sidebar-items.js"></script><script defer src="../../static.files/main-417ebf0a.js"></script><noscript><link rel="stylesheet" href="../../static.files/noscript-f7c3ffd8.css"></noscript><link rel="alternate icon" type="image/png" href="../../static.files/favicon-32x32-eab170b8.png"><link rel="icon" type="image/svg+xml" href="../../static.files/favicon-044be391.svg"></head><body class="rustdoc mod"><a class="skip-main-content" href="#main-content">Skip to main content</a><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><rustdoc-topbar><h2><a href="#">Module term_token</a></h2></rustdoc-topbar><nav class="sidebar"><div class="sidebar-crate"><h2><a href="../../common_base/index.html">common_<wbr>base</a><span class="version">1.4.0-alpha.0</span></h2></div><div class="sidebar-elems"><section id="rustdoc-toc"><h2 class="location"><a href="#">Module term_<wbr>token</a></h2><h3><a href="#enums">Module Items</a></h3><ul class="block"><li><a href="#enums" title="Enums">Enums</a></li><li><a href="#statics" title="Statics">Statics</a></li><li><a href="#functions" title="Functions">Functions</a></li></ul></section><div id="rustdoc-modnav"><h2 class="in-crate"><a href="../index.html">In crate common_<wbr>base</a></h2></div></div></nav><div class="sidebar-resizer" title="Drag to resize sidebar"></div><main><div class="width-limiter"><section id="main-content" class="content" tabindex="-1"><div class="main-heading"><div class="rustdoc-breadcrumbs"><a href="../index.html">common_base</a></div><h1>Module <span>term_<wbr>token</span>&nbsp;<button id="copy-path" title="Copy item path to clipboard">Copy item path</button></h1><rustdoc-toolbar></rustdoc-toolbar><span class="sub-heading"><a class="src" href="../../src/common_base/term_token.rs.html#15-280">Source</a> </span></div><details class="toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><p>Word boundaries shared by <code>matches_term</code> and the fulltext bloom index.</p>
<p>Bloom pruning is only sound if the index splits text exactly where
<code>matches_term</code> sees word boundaries, so both use <a href="fn.classify_char.html" title="fn common_base::term_token::classify_char"><code>classify_char</code></a>, and the
query side derives its probes with <a href="fn.extract_term_probes.html" title="fn common_base::term_token::extract_term_probes"><code>extract_term_probes</code></a> and <a href="fn.extract_like_probes.html" title="fn common_base::term_token::extract_like_probes"><code>extract_like_probes</code></a>.</p>
</div></details><h2 id="enums" class="section-header">Enums<a href="#enums" class="anchor">§</a></h2><dl class="item-table"><dt><a class="enum" href="enum.CharClass.html" title="enum common_base::term_token::CharClass">Char<wbr>Class</a></dt><dd>Classes of characters that decide where <code>matches_term</code> sees a word boundary.</dd></dl><h2 id="statics" class="section-header">Statics<a href="#statics" class="anchor">§</a></h2><dl class="item-table"><dt><a class="static" href="static.HAN_SCRIPT_DATA.html" title="static common_base::term_token::HAN_SCRIPT_DATA">HAN_<wbr>SCRIPT_<wbr>DATA</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt></dl><h2 id="functions" class="section-header">Functions<a href="#functions" class="anchor">§</a></h2><dl class="item-table"><dt><a class="fn" href="fn.classify_char.html" title="fn common_base::term_token::classify_char">classify_<wbr>char</a></dt><dd>Returns the <a href="enum.CharClass.html" title="enum common_base::term_token::CharClass"><code>CharClass</code></a> of <code>c</code>. <a href="fn.tokenize.html" title="fn common_base::term_token::tokenize"><code>tokenize</code></a> splits text wherever the class
changes and at every <a href="enum.CharClass.html#variant.Other" title="variant common_base::term_token::CharClass::Other"><code>CharClass::Other</code></a> character.</dd><dt><a class="fn" href="fn.extract_like_probes.html" title="fn common_base::term_token::extract_like_probes">extract_<wbr>like_<wbr>probes</a></dt><dd>Returns tokens that every text satisfying <code>text LIKE pattern</code> contains, as
produced by <a href="fn.tokenize.html" title="fn common_base::term_token::tokenize"><code>tokenize</code></a>. <code>\</code> escapes the next character, following
arrow’s <code>like</code> kernel; a trailing <code>\</code> is a literal backslash.</dd><dt><a class="fn" href="fn.extract_term_probes.html" title="fn common_base::term_token::extract_term_probes">extract_<wbr>term_<wbr>probes</a></dt><dd>Returns tokens that every text satisfying <code>matches_term(text, term)</code> contains,
as produced by <a href="fn.tokenize.html" title="fn common_base::term_token::tokenize"><code>tokenize</code></a>.</dd><dt><a class="fn" href="fn.for_each_word_run.html" title="fn common_base::term_token::for_each_word_run">for_<wbr>each_<wbr>word_<wbr>run</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Calls <code>f</code> on every maximal run of word characters of the same class.</dd><dt><a class="fn" href="fn.has_irregular_lowercase.html" title="fn common_base::term_token::has_irregular_lowercase">has_<wbr>irregular_<wbr>lowercase</a></dt><dd>Returns whether lowercasing <code>text</code> as a whole can produce tokens that lowercasing
each token of <code>text</code> doesn’t, which <code>matches_term(lower(col), ..)</code> and
<code>lower(col) LIKE ..</code> rely on. Every other character lowercases to one character
of the same class, but U+0130 and U+212A change length or class, and <code>Σ</code>
lowercases depending on the characters after it.</dd><dt><a class="fn" href="fn.is_han.html" title="fn common_base::term_token::is_han">is_han</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="fn" href="fn.push_han_tokens.html" title="fn common_base::term_token::push_han_tokens">push_<wbr>han_<wbr>tokens</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="fn" href="fn.push_literal_probes.html" title="fn common_base::term_token::push_literal_probes">push_<wbr>literal_<wbr>probes</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Pushes the tokens of <code>literal</code> that stay whole tokens wherever <code>literal</code>
occurs. A side is bounded when the character beyond it can’t extend the
adjacent run.</dd><dt><a class="fn" href="fn.tokenize.html" title="fn common_base::term_token::tokenize">tokenize</a></dt><dd>Returns the tokens the fulltext bloom index stores for <code>text</code>.</dd></dl></section></div></main></body></html>