greptimedb/log_store/object_store_wal/index.html

<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="Durable primitives of a WAL that stores entries as immutable object store objects."><title>log_store::object_store_wal - Rust</title><script>if(window.location.protocol!=="file:")document.head.insertAdjacentHTML("beforeend","SourceSerif4-Regular-6b053e98.ttf.woff2,FiraSans-Italic-81dc35de.woff2,FiraSans-Regular-0fe48ade.woff2,FiraSans-MediumItalic-ccf7e434.woff2,FiraSans-Medium-e1aa3f0a.woff2,SourceCodePro-Regular-8badfe75.ttf.woff2,SourceCodePro-Semibold-aa29a496.ttf.woff2".split(",").map(f=>`<link rel="preload" as="font" type="font/woff2"href="../../static.files/${f}">`).join(""))</script><link rel="stylesheet" href="../../static.files/normalize-9960930a.css"><link rel="stylesheet" href="../../static.files/rustdoc-1ed95124.css"><meta name="rustdoc-vars" data-root-path="../../" data-static-root-path="../../static.files/" data-current-crate="log_store" data-themes="" data-resource-suffix="" data-rustdoc-version="1.98.1 (48a229cea 2026-09-01)" data-channel="1.98.1" data-search-js="search-9944d736.js" data-stringdex-js="stringdex-2da4960a.js" data-settings-js="settings-170eb4bf.js" ><script src="../../static.files/storage-41dd4d93.js"></script><script defer src="../sidebar-items.js"></script><script defer src="../../static.files/main-417ebf0a.js"></script><noscript><link rel="stylesheet" href="../../static.files/noscript-f7c3ffd8.css"></noscript><link rel="alternate icon" type="image/png" href="../../static.files/favicon-32x32-eab170b8.png"><link rel="icon" type="image/svg+xml" href="../../static.files/favicon-044be391.svg"></head><body class="rustdoc mod"><a class="skip-main-content" href="#main-content">Skip to main content</a><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><rustdoc-topbar><h2><a href="#">Module object_store_wal</a></h2></rustdoc-topbar><nav class="sidebar"><div class="sidebar-crate"><h2><a href="../../log_store/index.html">log_<wbr>store</a><span class="version">1.4.0-alpha.0</span></h2></div><div class="sidebar-elems"><section id="rustdoc-toc"><h2 class="location"><a href="#">Module object_<wbr>store_<wbr>wal</a></h2><h3><a href="#modules">Module Items</a></h3><ul class="block"><li><a href="#modules" title="Modules">Modules</a></li><li><a href="#structs" title="Structs">Structs</a></li><li><a href="#functions" title="Functions">Functions</a></li></ul></section><div id="rustdoc-modnav"><h2 class="in-crate"><a href="../index.html">In crate log_<wbr>store</a></h2></div></div></nav><div class="sidebar-resizer" title="Drag to resize sidebar"></div><main><div class="width-limiter"><section id="main-content" class="content" tabindex="-1"><div class="main-heading"><div class="rustdoc-breadcrumbs"><a href="../index.html">log_store</a></div><h1>Module <span>object_<wbr>store_<wbr>wal</span>&nbsp;<button id="copy-path" title="Copy item path to clipboard">Copy item path</button></h1><rustdoc-toolbar></rustdoc-toolbar><span class="sub-heading"><a class="src" href="../../src/log_store/object_store_wal.rs.html#15-82">Source</a> </span></div><details class="toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><p>Durable primitives of a WAL that stores entries as immutable object store
objects.</p>
<p>Entries of many regions are batched into a single object, written once and
never mutated afterwards. An object is laid out as</p>
<div class="example-wrap"><pre class="language-text"><code>header | segment (region 1) | ... | segment (region N) | footer | trailer</code></pre></div>
<p>The header carries the magic <code>GTWALOBJ</code>, the format version, the object
sequence, the epoch of the instance that wrote the object, the link to the
object it extends, the reclaim boundary, and a CRC32 of the header. Each
segment holds the entries of exactly one region, ordered by entry id, and
segments are ordered by region id. The footer indexes every segment with its region id, entry id
range, byte range and CRC32. The fixed-size trailer points at the footer and
carries the CRC32 of the footer and of the whole object, so a reader locates
the footer by reading the fixed-length trailer at the end of the object.</p>
<p>Object sequences increase monotonically within one prefix and may leave
gaps, so recovery continues after the largest sequence present. An object is
created conditionally: rewriting a sequence with the content it already
holds is a no-op at the object store, while different content under a taken
sequence is a conflict. A create can fail with an unknown outcome and its
object can still appear, so objects form a chain: every object links to the
object it extends by sequence and epoch, and recovery replays only the chain
that ends at the complete object with the largest epoch and sequence. An
object is complete when every link on its chain names a present object of
the recorded epoch, back to an object that starts the chain or to a link
below the reclaim boundary of the object. Objects off the chain are
orphans, which are never replayed but keep their sequences. Each open
writes an object without segments above every present object before it
accepts writes, and its epoch is one above the sequence of that object, so
no two instances share an epoch and a late object of an earlier instance
never ends the chain.</p>
<p>The reclaim boundary of an object states that every object with a lower
sequence was fully flushed when the object was sealed, so the chain need
not reach below it. Every present object below the boundary of the last
indexed object is deleted, on the chain or not. A region that is dropped,
or whose entries are all made obsolete, is released: a release record of
the region goes into an object sealed at once, as the single entry of a
segment of a reserved region id, so the object keeps format version 1.
Recovery reads those segments of the chain and seeds the obsolete
watermarks from them, so a store opened later collects the objects of a
region that is never opened again.</p>
<p>Recovery lists the objects, reads and verifies only the header, trailer and
footer of each, and indexes the footers of the chain in sequence order to
rebuild the object catalog, which rejects a sequence it already holds. Of
the segments it reads only those of release records, and one that does not
decode fails recovery.
Segments are read and checksummed only when a read decodes them; a segment
that does not decode is fetched once more and then skipped as a hole of its
region or fails the read, as <code>on_corrupted_segment</code> says.</p>
<p>Entry ids are object-sequence-major, see <a href="fn.entry_id.html" title="fn log_store::object_store_wal::entry_id"><code>entry_id</code></a>: the high bits of an
id name the object that holds the entry, the low bits its position among
the entries of its region in that object.</p>
</div></details><h2 id="modules" class="section-header">Modules<a href="#modules" class="anchor">§</a></h2><dl class="item-table"><dt><a class="mod" href="batch/index.html" title="mod log_store::object_store_wal::batch">batch</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Accumulation of admitted entries into the batch that becomes the next
object, and the entry id scheme.</dd><dt><a class="mod" href="catalog/index.html" title="mod log_store::object_store_wal::catalog">catalog</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>In-memory index over the footers of the objects of one WAL prefix.</dd><dt><a class="mod" href="format/index.html" title="mod log_store::object_store_wal::format">format</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Encoding and decoding of a single WAL object. The byte layout is described
in the <a href="index.html" title="mod log_store::object_store_wal">module documentation</a>.</dd><dt><a class="mod" href="io/index.html" title="mod log_store::object_store_wal::io">io</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Object store access for WAL objects: a deterministic key layout and
conditional creates that make a retry of the same object a no-op.</dd><dt><a class="mod" href="store/index.html" title="mod log_store::object_store_wal::store">store</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>Object store WAL construction, recovery, writes and region reads.</dd></dl><h2 id="structs" class="section-header">Structs<a href="#structs" class="anchor">§</a></h2><dl class="item-table"><dt><a class="struct" href="struct.ObjectStoreLogStore.html" title="struct log_store::object_store_wal::ObjectStoreLogStore">Object<wbr>Store<wbr>LogStore</a></dt><dd>A log store over the immutable WAL objects under one prefix.</dd><dt><a class="struct" href="struct.WalHole.html" title="struct log_store::object_store_wal::WalHole">WalHole</a></dt><dd>A segment a read skipped because it did not decode: the entries of one
region in one object are missing from every read of that region.</dd></dl><h2 id="functions" class="section-header">Functions<a href="#functions" class="anchor">§</a></h2><dl class="item-table"><dt><a class="fn" href="fn.entry_id.html" title="fn log_store::object_store_wal::entry_id">entry_<wbr>id</a></dt><dd>Returns the id of the entry at <code>position</code> among the entries of its region
in the object <code>object_seq</code>.</dd></dl></section></div></main></body></html>