rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-07-05 05:20:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
cd9d372dfb8e893b6fd3689f1b3fc77ea5554edd
greptimedb/mito2/wal/encoder/index.html
github-merge-queue[bot] cd9d372dfb Deploying to gh-pages from @ GreptimeTeam/greptimedb@16b60fe847 🚀
2026-06-29 09:14:20 +00:00

37 lines
8.4 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="A cached-size, single-pass encoder for `WalEntry`."><title>mito2::wal::encoder - Rust</title><script>if(window.location.protocol!=="file:")document.head.insertAdjacentHTML("beforeend","SourceSerif4-Regular-6b053e98.ttf.woff2,FiraSans-Italic-81dc35de.woff2,FiraSans-Regular-0fe48ade.woff2,FiraSans-MediumItalic-ccf7e434.woff2,FiraSans-Medium-e1aa3f0a.woff2,SourceCodePro-Regular-8badfe75.ttf.woff2,SourceCodePro-Semibold-aa29a496.ttf.woff2".split(",").map(f=>`<link rel="preload" as="font" type="font/woff2"href="../../../static.files/${f}">`).join(""))</script><link rel="stylesheet" href="../../../static.files/normalize-9960930a.css"><link rel="stylesheet" href="../../../static.files/rustdoc-17e0aaed.css"><meta name="rustdoc-vars" data-root-path="../../../" data-static-root-path="../../../static.files/" data-current-crate="mito2" data-themes="" data-resource-suffix="" data-rustdoc-version="1.96.0-nightly (ac7f9ec7d 2026-03-20)" data-channel="nightly" data-search-js="search-63369b7b.js" data-stringdex-js="stringdex-2da4960a.js" data-settings-js="settings-170eb4bf.js" ><script src="../../../static.files/storage-41dd4d93.js"></script><script defer src="../sidebar-items.js"></script><script defer src="../../../static.files/main-5013f961.js"></script><noscript><link rel="stylesheet" href="../../../static.files/noscript-f7c3ffd8.css"></noscript><link rel="alternate icon" type="image/png" href="../../../static.files/favicon-32x32-eab170b8.png"><link rel="icon" type="image/svg+xml" href="../../../static.files/favicon-044be391.svg"></head><body class="rustdoc mod"><a class="skip-main-content" href="#main-content">Skip to main content</a><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><rustdoc-topbar><h2><a href="#">Module encoder</a></h2></rustdoc-topbar><nav class="sidebar"><div class="sidebar-crate"><h2><a href="../../../mito2/index.html">mito2</a><span class="version">1.2.0</span></h2></div><div class="sidebar-elems"><section id="rustdoc-toc"><h2 class="location"><a href="#">Module encoder</a></h2><h3><a href="#">Sections</a></h3><ul class="block top-toc"><li><a href="#why" title="Why">Why</a></li><li><a href="#how" title="How">How</a></li><li><a href="#maintenance" title="Maintenance">Maintenance</a></li></ul><h3><a href="#structs">Module Items</a></h3><ul class="block"><li><a href="#structs" title="Structs">Structs</a></li><li><a href="#constants" title="Constants">Constants</a></li><li><a href="#functions" title="Functions">Functions</a></li></ul></section><div id="rustdoc-modnav"><h2><a href="../index.html">In mito2::<wbr>wal</a></h2></div></div></nav><div class="sidebar-resizer" title="Drag to resize sidebar"></div><main><div class="width-limiter"><section id="main-content" class="content" tabindex="-1"><div class="main-heading"><div class="rustdoc-breadcrumbs"><a href="../../index.html">mito2</a>::<wbr><a href="../index.html">wal</a></div><h1>Module <span>encoder</span>&nbsp;<button id="copy-path" title="Copy item path to clipboard">Copy item path</button></h1><rustdoc-toolbar></rustdoc-toolbar><span class="sub-heading"><a class="src" href="../../../src/mito2/wal/encoder.rs.html#15-514">Source</a> </span></div><details class="toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><p>A cached-size, single-pass encoder for [<code>WalEntry</code>].</p>
<h2 id="why"><a class="doc-anchor" href="#why">§</a>Why</h2>
<p><code>prost</code> does not cache message sizes the way protobuf-C++ does. When
<code>WalEntry::encode_to_vec</code> runs, the encode pass recomputes the
<code>encoded_len</code> of every nested message each time it needs a length
delimiter. For the deep WAL tree
(<code>WalEntry → Mutation → Rows → Row → Value</code>) the length of a leaf <code>Value</code>
ends up being recomputed roughly once per ancestor level — i.e. ~5 times.
Microbenchmarks show ~87% of <code>encode_to_vec</code> is spent in these repeated
length walks, not in writing bytes.</p>
<h2 id="how"><a class="doc-anchor" href="#how">§</a>How</h2>
<p>This encoder walks the tree once to compute and cache the body length of
every length-delimited message node (in pre-order into a flat <code>sizes</code>
vector), then walks it a second time to write bytes, reading each cached
length back via a cursor. Every node’s length is computed exactly once.</p>
<p>Leaf messages whose sizing is not recursively redundant (<code>ColumnSchema</code>,
<code>Value</code>, <code>WriteHint</code>, <code>BulkWalEntry</code>) are delegated to prost’s own
<code>encoded_len</code>/<code>encode_raw</code>, but their (single) computed length is still
cached so the encode pass never recomputes it.</p>
<p>The output is byte-for-byte identical to <code>WalEntry::encode_to_vec</code>; this is
asserted in tests and must hold to preserve WAL replay compatibility.</p>
<h2 id="maintenance"><a class="doc-anchor" href="#maintenance">§</a>Maintenance</h2>
<p>This encoder hard-codes the wire layout (field tags and field order) of
<code>WalEntry</code>, <code>Mutation</code>, <code>Rows</code> and <code>Row</code>. If any of these messages change in
greptime-proto, this file MUST be updated to match:</p>
<ul>
<li><strong>Adding or removing a field</strong> is caught at compile time: every one of
these messages is destructured exhaustively (no <code>..</code>), so a changed field
set fails to compile here.</li>
<li><strong>Changing a field’s tag number or type</strong> is caught by the byte-for-byte
equality tests against prost (which populate all fields). Keep those tests
exhaustive when adding fields.</li>
</ul>
<p>Leaf messages are delegated to prost, so changes to them need no update here.</p>
</div></details><h2 id="structs" class="section-header">Structs<a href="#structs" class="anchor">§</a></h2><dl class="item-table"><dt><a class="struct" href="struct.WalEntryEncoder.html" title="struct mito2::wal::encoder::WalEntryEncoder">WalEntry<wbr>Encoder</a></dt><dd>A reusable encoder that caches message body sizes between its size pass and
its encode pass.</dd></dl><h2 id="constants" class="section-header">Constants<a href="#constants" class="anchor">§</a></h2><dl class="item-table"><dt><a class="constant" href="constant.BULK_ENTRY_TAG.html" title="constant mito2::wal::encoder::BULK_ENTRY_TAG">BULK_<wbr>ENTRY_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.MUTATION_TAG.html" title="constant mito2::wal::encoder::MUTATION_TAG">MUTATION_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.OP_TYPE_TAG.html" title="constant mito2::wal::encoder::OP_TYPE_TAG">OP_<wbr>TYPE_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.ROWS_TAG.html" title="constant mito2::wal::encoder::ROWS_TAG">ROWS_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.ROW_TAG.html" title="constant mito2::wal::encoder::ROW_TAG">ROW_TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.SCHEMA_TAG.html" title="constant mito2::wal::encoder::SCHEMA_TAG">SCHEMA_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.SEQUENCE_TAG.html" title="constant mito2::wal::encoder::SEQUENCE_TAG">SEQUENCE_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.VALUE_TAG.html" title="constant mito2::wal::encoder::VALUE_TAG">VALUE_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dt><a class="constant" href="constant.WRITE_HINT_TAG.html" title="constant mito2::wal::encoder::WRITE_HINT_TAG">WRITE_<wbr>HINT_<wbr>TAG</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt></dl><h2 id="functions" class="section-header">Functions<a href="#functions" class="anchor">§</a></h2><dl class="item-table"><dt><a class="fn" href="fn.msg_field_len.html" title="fn mito2::wal::encoder::msg_field_len">msg_<wbr>field_<wbr>len</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Length contribution of a length-delimited message field:
key + length varint + body.</dd></dl></section></div></main></body></html>
Reference in New Issue View Git Blame Copy Permalink