mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-05-26 01:40:36 +00:00
26 lines
7.2 KiB
HTML
26 lines
7.2 KiB
HTML
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="SST Files with Inverted Index Format Specification"><title>index::inverted_index::format - Rust</title><script>if(window.location.protocol!=="file:")document.head.insertAdjacentHTML("beforeend","SourceSerif4-Regular-6b053e98.ttf.woff2,FiraSans-Italic-81dc35de.woff2,FiraSans-Regular-0fe48ade.woff2,FiraSans-MediumItalic-ccf7e434.woff2,FiraSans-Medium-e1aa3f0a.woff2,SourceCodePro-Regular-8badfe75.ttf.woff2,SourceCodePro-Semibold-aa29a496.ttf.woff2".split(",").map(f=>`<link rel="preload" as="font" type="font/woff2"href="../../../static.files/${f}">`).join(""))</script><link rel="stylesheet" href="../../../static.files/normalize-9960930a.css"><link rel="stylesheet" href="../../../static.files/rustdoc-17e0aaed.css"><meta name="rustdoc-vars" data-root-path="../../../" data-static-root-path="../../../static.files/" data-current-crate="index" data-themes="" data-resource-suffix="" data-rustdoc-version="1.96.0-nightly (ac7f9ec7d 2026-03-20)" data-channel="nightly" data-search-js="search-63369b7b.js" data-stringdex-js="stringdex-2da4960a.js" data-settings-js="settings-170eb4bf.js" ><script src="../../../static.files/storage-41dd4d93.js"></script><script defer src="../sidebar-items.js"></script><script defer src="../../../static.files/main-5013f961.js"></script><noscript><link rel="stylesheet" href="../../../static.files/noscript-f7c3ffd8.css"></noscript><link rel="alternate icon" type="image/png" href="../../../static.files/favicon-32x32-eab170b8.png"><link rel="icon" type="image/svg+xml" href="../../../static.files/favicon-044be391.svg"></head><body class="rustdoc mod"><a class="skip-main-content" href="#main-content">Skip to main content</a><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><rustdoc-topbar><h2><a href="#">Module format</a></h2></rustdoc-topbar><nav class="sidebar"><div class="sidebar-crate"><h2><a href="../../../index/index.html">index</a><span class="version">1.0.0</span></h2></div><div class="sidebar-elems"><section id="rustdoc-toc"><h2 class="location"><a href="#">Module format</a></h2><h3><a href="#">Sections</a></h3><ul class="block top-toc"><li><a href="#sst-files-with-inverted-index-format-specification" title="SST Files with Inverted Index Format Specification">SST Files with Inverted Index Format Specification</a><ul><li><a href="#file-structure" title="File Structure">File Structure</a></li><li><a href="#inverted-index-internals" title="Inverted Index Internals">Inverted Index Internals</a></li><li><a href="#footer-details" title="Footer Details">Footer Details</a></li><li><a href="#reference" title="Reference">Reference</a></li></ul></li></ul><h3><a href="#modules">Module Items</a></h3><ul class="block"><li><a href="#modules" title="Modules">Modules</a></li><li><a href="#constants" title="Constants">Constants</a></li></ul></section><div id="rustdoc-modnav"><h2><a href="../index.html">In index::<wbr>inverted_<wbr>index</a></h2></div></div></nav><div class="sidebar-resizer" title="Drag to resize sidebar"></div><main><div class="width-limiter"><section id="main-content" class="content" tabindex="-1"><div class="main-heading"><div class="rustdoc-breadcrumbs"><a href="../../index.html">index</a>::<wbr><a href="../index.html">inverted_index</a></div><h1>Module <span>format</span> <button id="copy-path" title="Copy item path to clipboard">Copy item path</button></h1><rustdoc-toolbar></rustdoc-toolbar><span class="sub-heading"><a class="src" href="../../../src/index/inverted_index/format.rs.html#15-57">Source</a> </span></div><details class="toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><h2 id="sst-files-with-inverted-index-format-specification"><a class="doc-anchor" href="#sst-files-with-inverted-index-format-specification">§</a>SST Files with Inverted Index Format Specification</h2><h3 id="file-structure"><a class="doc-anchor" href="#file-structure">§</a>File Structure</h3>
|
||
<p>Each SST file includes a series of inverted indices followed by a footer.</p>
|
||
<p><code>inverted_index₀ inverted_index₁ ... inverted_indexₙ footer</code></p>
|
||
<ul>
|
||
<li>Each <code>inverted_indexᵢ</code> represents an index entry corresponding to tag values and their locations within the file.</li>
|
||
<li><code>footer</code>: Contains metadata about the inverted indices, encoded as a protobuf message.</li>
|
||
</ul>
|
||
<h3 id="inverted-index-internals"><a class="doc-anchor" href="#inverted-index-internals">§</a>Inverted Index Internals</h3>
|
||
<p>An inverted index comprises a collection of bitmaps, a null bitmap, and a finite state transducer (FST) indicating tag values’ positions:</p>
|
||
<p><code>null_bitmap bitmap₀ bitmap₁ bitmap₂ ... bitmapₙ fst</code></p>
|
||
<ul>
|
||
<li><code>null_bitmap</code>: Bitset tracking the presence of null values within the tag column.</li>
|
||
<li><code>bitmapᵢ</code>: Bitset indicating the presence of tag values within a row group.</li>
|
||
<li><code>fst</code>: Finite State Transducer providing an ordered map of bytes, representing the tag values.</li>
|
||
</ul>
|
||
<h3 id="footer-details"><a class="doc-anchor" href="#footer-details">§</a>Footer Details</h3>
|
||
<p>The footer encapsulates the metadata for inversion mappings:</p>
|
||
<p><code>footer_payload footer_payload_size</code></p>
|
||
<ul>
|
||
<li><code>footer_payload</code>: Protobuf-encoded <a href="https://github.com/GreptimeTeam/greptime-proto/blob/2aaee38de81047537dfa42af9df63bcfb866e06c/proto/greptime/v1/index/inverted_index.proto#L32-L64"><code>InvertedIndexMetas</code></a> describing the metadata of each inverted index.</li>
|
||
<li><code>footer_payload_size</code>: Size in bytes of the <code>footer_payload</code>, displayed as a <code>u32</code> integer.</li>
|
||
<li>The footer aids in the interpretation of the inverted indices, providing necessary offset and count information.</li>
|
||
</ul>
|
||
<h3 id="reference"><a class="doc-anchor" href="#reference">§</a>Reference</h3>
|
||
<p>More detailed information regarding the encoding of the inverted indices can be found in the <a href="https://github.com/GreptimeTeam/greptimedb/blob/main/docs/rfcs/2023-11-03-inverted-index.md">RFC</a>.</p>
|
||
</div></details><h2 id="modules" class="section-header">Modules<a href="#modules" class="anchor">§</a></h2><dl class="item-table"><dt><a class="mod" href="reader/index.html" title="mod index::inverted_index::format::reader">reader</a></dt><dt><a class="mod" href="writer/index.html" title="mod index::inverted_index::format::writer">writer</a></dt></dl><h2 id="constants" class="section-header">Constants<a href="#constants" class="anchor">§</a></h2><dl class="item-table"><dt><a class="constant" href="constant.FOOTER_PAYLOAD_SIZE_SIZE.html" title="constant index::inverted_index::format::FOOTER_PAYLOAD_SIZE_SIZE">FOOTER_<wbr>PAYLOAD_<wbr>SIZE_<wbr>SIZE</a><span title="Restricted Visibility"> 🔒</span> </dt><dt><a class="constant" href="constant.MIN_BLOB_SIZE.html" title="constant index::inverted_index::format::MIN_BLOB_SIZE">MIN_<wbr>BLOB_<wbr>SIZE</a><span title="Restricted Visibility"> 🔒</span> </dt></dl></section></div></main></body></html> |