greptimedb/mito2/engine/region_hook/index.html

<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="Region hook extension point for observing SST writes and manifest mutations."><title>mito2::engine::region_hook - Rust</title><script>if(window.location.protocol!=="file:")document.head.insertAdjacentHTML("beforeend","SourceSerif4-Regular-6b053e98.ttf.woff2,FiraSans-Italic-81dc35de.woff2,FiraSans-Regular-0fe48ade.woff2,FiraSans-MediumItalic-ccf7e434.woff2,FiraSans-Medium-e1aa3f0a.woff2,SourceCodePro-Regular-8badfe75.ttf.woff2,SourceCodePro-Semibold-aa29a496.ttf.woff2".split(",").map(f=>`<link rel="preload" as="font" type="font/woff2"href="../../../static.files/${f}">`).join(""))</script><link rel="stylesheet" href="../../../static.files/normalize-9960930a.css"><link rel="stylesheet" href="../../../static.files/rustdoc-1ed95124.css"><meta name="rustdoc-vars" data-root-path="../../../" data-static-root-path="../../../static.files/" data-current-crate="mito2" data-themes="" data-resource-suffix="" data-rustdoc-version="1.98.1 (48a229cea 2026-09-01)" data-channel="1.98.1" data-search-js="search-9944d736.js" data-stringdex-js="stringdex-2da4960a.js" data-settings-js="settings-170eb4bf.js" ><script src="../../../static.files/storage-41dd4d93.js"></script><script defer src="../sidebar-items.js"></script><script defer src="../../../static.files/main-417ebf0a.js"></script><noscript><link rel="stylesheet" href="../../../static.files/noscript-f7c3ffd8.css"></noscript><link rel="alternate icon" type="image/png" href="../../../static.files/favicon-32x32-eab170b8.png"><link rel="icon" type="image/svg+xml" href="../../../static.files/favicon-044be391.svg"></head><body class="rustdoc mod"><a class="skip-main-content" href="#main-content">Skip to main content</a><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><rustdoc-topbar><h2><a href="#">Module region_hook</a></h2></rustdoc-topbar><nav class="sidebar"><div class="sidebar-crate"><h2><a href="../../../mito2/index.html">mito2</a><span class="version">1.4.0-alpha.0</span></h2></div><div class="sidebar-elems"><section id="rustdoc-toc"><h2 class="location"><a href="#">Module region_<wbr>hook</a></h2><h3><a href="#">Sections</a></h3><ul class="block top-toc"><li><a href="#design" title="Design">Design</a></li><li><a href="#coverage" title="Coverage">Coverage</a></li><li><a href="#region-lifecycle" title="Region lifecycle">Region lifecycle</a></li><li><a href="#invocation-points" title="Invocation points">Invocation points</a></li><li><a href="#future-work" title="Future work">Future work</a></li></ul><h3><a href="#structs">Module Items</a></h3><ul class="block"><li><a href="#structs" title="Structs">Structs</a></li><li><a href="#traits" title="Traits">Traits</a></li><li><a href="#types" title="Type Aliases">Type Aliases</a></li></ul></section><div id="rustdoc-modnav"><h2><a href="../index.html">In mito2::<wbr>engine</a></h2></div></div></nav><div class="sidebar-resizer" title="Drag to resize sidebar"></div><main><div class="width-limiter"><section id="main-content" class="content" tabindex="-1"><div class="main-heading"><div class="rustdoc-breadcrumbs"><a href="../../index.html">mito2</a>::<wbr><a href="../index.html">engine</a></div><h1>Module <span>region_<wbr>hook</span>&nbsp;<button id="copy-path" title="Copy item path to clipboard">Copy item path</button></h1><rustdoc-toolbar></rustdoc-toolbar><span class="sub-heading"><a class="src" href="../../../src/mito2/engine/region_hook.rs.html#15-474">Source</a> </span></div><details class="toggle top-doc" open><summary class="hideme"><span>Expand description</span></summary><div class="docblock"><p>Region hook extension point for observing SST writes and manifest mutations.</p>
<h3 id="design"><a class="doc-anchor" href="#design">§</a>Design</h3>
<p>The <a href="trait.RegionHook.html" title="trait mito2::engine::region_hook::RegionHook"><code>RegionHook</code></a> trait observes region activity through two categories of
callbacks — manifest/file observation and region lifecycle:</p>
<ul>
<li>
<p><a href="trait.RegionHook.html#method.on_sst_files_written" title="method mito2::engine::region_hook::RegionHook::on_sst_files_written"><code>on_sst_files_written</code></a>: Fires when mito2 physically writes SST <strong>data files</strong>.
Provides per-file <a href="../../sst/parquet/struct.SstInfo.html" title="struct mito2::sst::parquet::SstInfo"><code>SstInfo</code></a> + <a href="../../sst/file/struct.FileMeta.html" title="struct mito2::sst::file::FileMeta"><code>FileMeta</code></a>; metadata richness varies by path
(see <a href="struct.SstFileInfo.html" title="struct mito2::engine::region_hook::SstFileInfo"><code>SstFileInfo</code></a> and the coverage footnote).</p>
</li>
<li>
<p><a href="trait.RegionHook.html#method.on_manifest_updated" title="method mito2::engine::region_hook::RegionHook::on_manifest_updated"><code>on_manifest_updated</code></a>: Fires after a manifest write is committed to the <strong>live</strong>
(normal) manifest directory. Writes to the staging directory (enter staging,
operations during staging, the intermediate apply-staging edit) are suppressed —
their effects are accumulated and delivered in a single notification when the
staged actions are promoted to the live manifest. Receives the full
<a href="../../manifest/action/struct.RegionMetaActionList.html" title="struct mito2::manifest::action::RegionMetaActionList"><code>RegionMetaActionList</code></a> so consumers can inspect what changed (file additions/
removals, schema changes, truncation, partition expression changes, etc.).</p>
</li>
<li>
<p><a href="trait.RegionHook.html#method.on_region_opened" title="method mito2::engine::region_hook::RegionHook::on_region_opened"><code>on_region_opened</code></a> / <a href="trait.RegionHook.html#method.on_region_closed" title="method mito2::engine::region_hook::RegionHook::on_region_closed"><code>on_region_closed</code></a> / <a href="trait.RegionHook.html#method.on_region_dropped" title="method mito2::engine::region_hook::RegionHook::on_region_dropped"><code>on_region_dropped</code></a> / <a href="trait.RegionHook.html#method.on_region_files_removed" title="method mito2::engine::region_hook::RegionHook::on_region_files_removed"><code>on_region_files_removed</code></a>:
Region <strong>lifecycle</strong> callbacks for open, close, logical drop, and physical file removal.
See <a href="#region-lifecycle">Region lifecycle</a> below.</p>
</li>
</ul>
<p>Hook implementations are registered via the <a href="common_base::Plugins"><code>Plugins</code></a> system:</p>

<div class="example-wrap ignore"><a href="#" class="tooltip" title="This example is not tested">ⓘ</a><pre class="rust rust-example-rendered"><code>plugins.insert(Arc::new(MyHook) <span class="kw">as </span>RegionHookRef);</code></pre></div><h3 id="coverage"><a class="doc-anchor" href="#coverage">§</a>Coverage</h3>
<p>Only manifest writes to the <strong>normal</strong> (live) manifest directory trigger
<code>on_manifest_updated</code>. Writes to the staging manifest directory (operations
that happen while the region is in staging mode) are intentionally suppressed:
their effects are accumulated and delivered in a single notification when
the staged actions are promoted to the live manifest via <code>exit_staging_on_success</code>.</p>
<div><table><thead><tr><th>Scenario</th><th style="text-align: center"><code>on_sst_files_written</code></th><th style="text-align: center"><code>on_manifest_updated</code></th></tr></thead><tbody>
<tr><td>Flush (memtable → SST)</td><td style="text-align: center">✅ Yes</td><td style="text-align: center">✅ Yes</td></tr>
<tr><td>Local compaction</td><td style="text-align: center">✅ Yes</td><td style="text-align: center">✅ Yes</td></tr>
<tr><td>Remote compaction</td><td style="text-align: center">✅ (compactor node) ¹</td><td style="text-align: center">✅ (compactor node) ¹</td></tr>
<tr><td>RegionEdit / bulk ingestion</td><td style="text-align: center">❌ (files pre-written)</td><td style="text-align: center">✅ Yes</td></tr>
<tr><td>Copy region</td><td style="text-align: center">❌ (object-store copy)</td><td style="text-align: center">✅ Yes</td></tr>
<tr><td>Apply staging (promote)</td><td style="text-align: center">❌ (delegates to edit)</td><td style="text-align: center">✅ Yes ²</td></tr>
<tr><td>Alter (schema change)</td><td style="text-align: center">❌ (no SST files)</td><td style="text-align: center">✅ Yes</td></tr>
<tr><td>Truncate</td><td style="text-align: center">❌ (removes files)</td><td style="text-align: center">✅ Yes</td></tr>
<tr><td>Enter staging</td><td style="text-align: center">❌ (no SST files)</td><td style="text-align: center">❌ (staging dir)</td></tr>
<tr><td>Operations during staging</td><td style="text-align: center">N/A</td><td style="text-align: center">❌ (staging dir)</td></tr>
<tr><td>Async index build</td><td style="text-align: center">❌ (index files only)</td><td style="text-align: center">✅ Yes</td></tr>
</tbody></table>
</div>
<p>¹ Remote compaction runs on a dedicated compactor node via <code>open_compaction_region()</code>;
pass plugins via <code>OpenCompactionRegionRequest</code> to enable hooks there. <code>sst_infos</code> is
<code>#[serde(skip)]</code> over the wire, so each <a href="../../sst/parquet/struct.SstInfo.html" title="struct mito2::sst::parquet::SstInfo"><code>SstInfo</code></a> is rebuilt from <a href="../../sst/file/struct.FileMeta.html" title="struct mito2::sst::file::FileMeta"><code>FileMeta</code></a> with
empty footer/index — see <a href="struct.SstFileInfo.html" title="struct mito2::engine::region_hook::SstFileInfo"><code>SstFileInfo</code></a> for field-level detail.
² Apply staging fires <code>on_manifest_updated</code> once when <code>exit_staging_on_success</code> promotes
all staged manifest actions (including the SST file additions) into the live manifest.
The intermediate staging <code>RegionEdit</code> is written to the staging directory and does not
fire the hook — its file list is included in the promote notification.</p>
<p>The following paths do <strong>not</strong> trigger any hook:</p>
<ul>
<li>Follower region sync / catchup (manifest read-only; followers don’t author changes)</li>
<li>GC / checkpoint / remap (internal bookkeeping, not logical state changes)</li>
</ul>
<p>An explicit region <strong>drop</strong> does fire lifecycle hooks — see
<a href="#region-lifecycle">Region lifecycle</a>.</p>
<h3 id="region-lifecycle"><a class="doc-anchor" href="#region-lifecycle">§</a>Region lifecycle</h3>
<p>Beyond manifest/SST observation, the hook observes the high-level lifecycle of an
active region:</p>
<div><table><thead><tr><th>Event</th><th>Method</th><th>When</th></tr></thead><tbody>
<tr><td>Open</td><td><a href="trait.RegionHook.html#method.on_region_opened" title="method mito2::engine::region_hook::RegionHook::on_region_opened"><code>on_region_opened</code></a></td><td>A create or open request registers the region as active (the counterpart to close/drop). Does not fire for the compactor’s transient regions or the catch-up reopen.</td></tr>
<tr><td>Close</td><td><a href="trait.RegionHook.html#method.on_region_closed" title="method mito2::engine::region_hook::RegionHook::on_region_closed"><code>on_region_closed</code></a></td><td>A close request (or a close-after-flush) removes the region from the active set. Data files, manifest and WAL state are <strong>preserved</strong>; the region may be reopened.</td></tr>
<tr><td>Logical drop</td><td><a href="trait.RegionHook.html#method.on_region_dropped" title="method mito2::engine::region_hook::RegionHook::on_region_dropped"><code>on_region_dropped</code></a></td><td>A drop request has been handled: the region leaves the active set and its WAL entries are marked obsolete. Data files are <strong>not yet deleted</strong>.</td></tr>
<tr><td>Physical file removal</td><td><a href="trait.RegionHook.html#method.on_region_files_removed" title="method mito2::engine::region_hook::RegionHook::on_region_files_removed"><code>on_region_files_removed</code></a></td><td>The drop GC worker has deleted the region directory. Terminal file-lifecycle event.</td></tr>
<tr><td>Global GC pass</td><td><a href="trait.RegionHook.html#method.on_region_gc" title="method mito2::engine::region_hook::RegionHook::on_region_gc"><code>on_region_gc</code></a></td><td>The datanode’s global GC worker finished a GC pass for a region — both periodic GC for live regions and the global reclamation of dropped/repartitioned regions (<code>is_region_dropped</code>). Also fired by the offline cleanup path (<code>handle_offline_cleanup_request</code>, i.e. soft-drop PURGE) once the region directory is removed, with <code>is_region_dropped = true</code> and <code>full_file_listing = true</code>.</td></tr>
</tbody></table>
</div>
<p>Notes:</p>
<ul>
<li><code>on_region_closed</code> / <code>on_region_dropped</code> run <strong>inline in the region worker loop</strong>,
so implementations must be fast (same contract as <code>on_manifest_updated</code>).</li>
<li><code>on_region_opened</code> runs inline in the worker loop on the <strong>create</strong> path, but on the
<strong>open</strong> path it fires inside the spawned open task (<code>common_runtime::spawn_global</code>),
i.e. concurrently with the worker loop — after WAL replay, before the region is
registered and its open request is acknowledged. Implementations must still be fast
and must not assume worker-loop-thread affinity or strict ordering against concurrent
requests to other regions.</li>
<li><code>on_region_files_removed</code> runs on the background drop GC task, outside the worker loop.</li>
<li>When global GC is enabled and a normal table region is dropped with <code>partial_drop</code>, its
directory is left for global reclamation and <code>on_region_files_removed</code> is <strong>not</strong> fired
by the drop worker (observe it via the global GC path instead).</li>
<li>The offline cleanup path (<code>handle_offline_cleanup_request</code>, reached by a soft-drop
<code>PURGE</code>) force-removes the region directory but has no <code>RegionMetadataRef</code> for the
offline region, so it cannot fire <code>on_region_files_removed</code>. It fires <code>on_region_gc</code>
instead — with <code>is_region_dropped = true</code> and <code>full_file_listing = true</code> — so
extensions can reclaim sidecar state for the now-removed region. A hook <code>Err</code> is
propagated so the caller retries the (idempotent) cleanup.</li>
<li>Logical file removal (compaction, region edit, truncate) is already observable via
<a href="trait.RegionHook.html#method.on_manifest_updated" title="method mito2::engine::region_hook::RegionHook::on_manifest_updated"><code>on_manifest_updated</code></a> (<code>Edit.files_to_remove</code> / <code>Truncate</code> action); only the drop
worker’s physical directory deletion needs a dedicated file hook.</li>
</ul>
<h3 id="invocation-points"><a class="doc-anchor" href="#invocation-points">§</a>Invocation points</h3>
<p><code>on_sst_files_written</code> is invoked at the SST write site (flush task or compaction task),
immediately after SST files are written but <strong>before</strong> the manifest is committed.</p>
<p><code>on_manifest_updated</code> is funneled through <a href="../../region/struct.ManifestContext.html#method.update_locked" title="method mito2::region::ManifestContext::update_locked"><code>ManifestContext::update_locked</code></a>,
the sole caller of the low-level <a href="../../manifest/manager/struct.RegionManifestManager.html#method.update" title="method mito2::manifest::manager::RegionManifestManager::update"><code>RegionManifestManager::update</code></a>, which
packages each successful write into a <a href="struct.PendingManifestHook.html" title="struct mito2::engine::region_hook::PendingManifestHook"><code>PendingManifestHook</code></a>. The caller
owns the write lock, drops it, and <em>then</em> fires the receipt — the hook must
never run under the lock. <a href="../../region/struct.ManifestContext.html#method.update_manifest" title="method mito2::region::ManifestContext::update_manifest"><code>ManifestContext::update_manifest</code></a> is the common
case: it acquires the lock, delegates to <code>update_locked</code>, and fires the
receipt in one go. Multi-step sequences (staging-exit, role-state backfill)
call <code>update_locked</code> directly under their own held guard.</p>
<p>Non-logical writes (GC, staging bookkeeping) call the manager’s own methods
directly and intentionally do not fire the hook.</p>
<h3 id="future-work"><a class="doc-anchor" href="#future-work">§</a>Future work</h3>
<p><code>on_region_files_removed</code> currently covers only the <strong>drop</strong> GC worker’s physical
directory removal. The global-GC reclamation path and the offline-cleanup path
(soft-drop PURGE) are covered by <code>on_region_gc</code> instead. A broader per-file
<code>on_files_removed</code> hook covering compaction removal and truncate is not yet implemented
(though logical file removal is already observable via <code>on_manifest_updated</code>,
and the global GC reclamation path is covered by <code>on_region_gc</code>).
Role/leadership transitions (<code>on_region_role_changed</code>) are also not hooked.</p>
</div></details><h2 id="structs" class="section-header">Structs<a href="#structs" class="anchor">§</a></h2><dl class="item-table"><dt><a class="struct" href="struct.PendingManifestHook.html" title="struct mito2::engine::region_hook::PendingManifestHook">Pending<wbr>Manifest<wbr>Hook</a><span title="Restricted Visibility">&nbsp;🔒</span> </dt><dd>A deferred <a href="trait.RegionHook.html#method.on_manifest_updated" title="method mito2::engine::region_hook::RegionHook::on_manifest_updated"><code>RegionHook::on_manifest_updated</code></a> notification produced by a
logical manifest write via <a href="../../region/struct.ManifestContext.html#method.update_locked" title="method mito2::region::ManifestContext::update_locked"><code>ManifestContext::update_locked</code></a>.</dd><dt><a class="struct" href="struct.RegionGcInfo.html" title="struct mito2::engine::region_hook::RegionGcInfo">Region<wbr>GcInfo</a></dt><dd>What mito2’s GC pass deleted for a region, handed to
<a href="trait.RegionHook.html#method.on_region_gc" title="method mito2::engine::region_hook::RegionHook::on_region_gc"><code>RegionHook::on_region_gc</code></a>.</dd><dt><a class="struct" href="struct.SstFileInfo.html" title="struct mito2::engine::region_hook::SstFileInfo">SstFile<wbr>Info</a></dt><dd>Information about a single SST data file written during flush or compaction.</dd></dl><h2 id="traits" class="section-header">Traits<a href="#traits" class="anchor">§</a></h2><dl class="item-table"><dt><a class="trait" href="trait.RegionHook.html" title="trait mito2::engine::region_hook::RegionHook">Region<wbr>Hook</a></dt><dd>Hook for observing region mutations in mito2.</dd></dl><h2 id="types" class="section-header">Type Aliases<a href="#types" class="anchor">§</a></h2><dl class="item-table"><dt><a class="type" href="type.RegionHookRef.html" title="type mito2::engine::region_hook::RegionHookRef">Region<wbr>Hook<wbr>Ref</a></dt></dl></section></div></main></body></html>