* feat(log-store): persist bounded region releases in shared WAL batches Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): release safe WAL history when closing regions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): cover bounded WAL release across close and restart Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): clamp bounded releases to durable entries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): observe close release failures and cover recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: describe bounded WAL release lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): make bounded release fence sealing deterministic Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): bound queued releases and ignore shutdown hints Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover close release persistence and shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): wait for failed close flush cleanup Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: clarify bounded WAL release backpressure and shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(mito2): clarify staging test retry invariant Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): back off durable release retries Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): drive release retry after backoff Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): share and verify WAL retry backoff Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: clarify retained WAL release retry semantics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com>
14 KiB
mito2 — Agent & Contributor Guide
Navigation aid for src/mito2. Keep it short and point to code; do not duplicate
the code here. Paths are relative to the repo root.
Repo-wide rules that apply here: .agents/architecture-invariants.md.
What this crate does
Mito2 is GreptimeDB's primary time-series region storage engine. It owns the
write path (memtable + WAL), flushing memtables to Parquet SST files,
TWCS/windowed compaction, and the read path (multi-level merge + dedup with
snapshot isolation). It implements the RegionEngine trait from store-api.
Module map
| Module | Path | Purpose |
|---|---|---|
engine |
src/mito2/src/engine.rs |
MitoEngine (the RegionEngine impl) and request dispatch |
worker |
src/mito2/src/worker.rs, src/mito2/src/worker/ |
Per-region worker loop; write/alter/flush handlers |
region |
src/mito2/src/region.rs, src/mito2/src/region/version.rs |
MitoRegion state and copy-on-write VersionControl snapshots |
request |
src/mito2/src/request.rs |
WriteRequest/RegionRequest types and result channels |
wal |
src/mito2/src/wal.rs |
Write-ahead log wrapper over log-store |
memtable |
src/mito2/src/memtable/ |
In-memory write buffers (time-series / bulk / partition) |
flush |
src/mito2/src/flush.rs |
FlushScheduler, WriteBufferManager, memtable → SST |
compaction |
src/mito2/src/compaction/ |
Compaction scheduler (scheduler.rs + scheduler/), TWCS picker, strict-window manual picker, compactor, memory control |
compaction/multi_region |
src/mito2/src/compaction/multi_region.rs |
Enterprise lossless routing by a captured table partition rule; selected SSTs to per-region outputs, without manifest publication |
engine/table_compaction |
src/mito2/src/engine/table_compaction.rs |
Enterprise conditional publication of table-compaction results: shared-input snapshot, verified-output checks, and the manifest-locked swap of shared inputs for L0 outputs |
access_layer |
src/mito2/src/access_layer.rs |
SST read/write over the object store |
sst |
src/mito2/src/sst/ |
Parquet format, file metadata, index layout |
series_index |
src/mito2/src/series_index/ |
Series index writer/searcher, bucket planning, catalogs, worker-owned reconciliation (maintenance.rs), shared approximate usage (task.rs), catalogs, and index lifecycle |
read |
src/mito2/src/read/ |
ScanRegion, merge, dedup, projection, streaming |
manifest |
src/mito2/src/manifest/ |
RegionManifestManager, manifest actions/edits |
cache |
src/mito2/src/cache.rs |
Write/file/page caches |
gc |
src/mito2/src/gc.rs, src/mito2/src/gc/ |
Dropped-file cleanup worker |
schedule |
src/mito2/src/schedule/ |
Local/remote background job scheduling |
remap_manifest |
src/mito2/src/remap_manifest.rs |
Manifest path remapping for region copy/migration |
config |
src/mito2/src/config.rs |
MitoConfig tuning knobs |
tail_replay |
src/mito2/src/tail_replay.rs, src/mito2/src/worker/handle_tail_replay.rs |
Background replay of pending WAL tails into SSTs, published with one manifest edit |
temp_file_sweeper |
src/mito2/src/temp_file_sweeper.rs |
Reclaims temporary files a crashed writer left in a region, triggered when the region becomes a writable leader or a create opens an existing region (under the hdfs-object-store feature) |
test_util |
src/mito2/src/test_util.rs |
TestEnv and builders (under the test feature) |
Write path
MitoEngine::handle_request (engine.rs), or handle_batch_put_requests for the puts
of one request to several regions (one message per worker) → worker loop
(worker/handle_write.rs) → sequence + WAL assembly (region_write_ctx.rs) →
wal.rs → memtable (memtable/) → when buffer pressure trips, flush.rs
writes SSTs via access_layer.rs and appends a RegionEdit to the manifest
(manifest/manager.rs).
Read path
MitoEngine::handle_query (engine.rs) → read/scan_region.rs takes an
immutable Version (region/version.rs) → scans memtables and Parquet SSTs
(sst/parquet.rs) → merges (read/) and dedups by sequence → projected,
filtered RecordBatch stream.
Internal series merges use the query runtime pool forwarded through
QueryScanContext; reader and output-batch accounting uses the separate scan
memory tracker. Direct scanner callers default to an unbounded operator pool.
Live compaction queries read shared per-region state directly from MitoRegion,
without sending a worker request. The scheduler publishes active and pending task
state together; readers clone task Arcs under a short read lock, then filter and
build rows outside the lock. Execution handles provide live background phases.
There is no second task registry or completed-task history. See
compaction/scheduler/state.rs. The datanode provider reuses the lazy inspection scan
path and retains residual SQL filters.
Task cancellation uses RegionRequest::CancelCompaction through Region.Handle;
the owning worker matches the live task ID without entering the DDL queue.
Task-specific cancellation preserves other pending manual work when no DDL is
pending. DDL cancellation retains its existing scheduling fence and priority.
Public surface
- Entry:
MitoEngineinsrc/mito2/src/engine.rs, built viaMitoEngineBuilder. - Trait:
impl RegionEngine for MitoEngine(store-api's region engine contract). - Consumed by
datanode(sendsRegionRequests) and the query layer (scans).
When you change X, also touch Y
- Manifest format (
manifest/action.rs): affects crash recovery and follower replay. Keep it backward compatible. Compaction task state does not belong in the manifest; publication uses ordinary file edits after locked input checks. Shared SSTs require Enterprise capability. Compatibility fixtures:tests/compatibility/manifest/table_compaction/andtests/compatibility/manifest/pending_tails/. - SST/Parquet layout (
sst/): readers must stay compatible with existing files. - Series-index disk limit: tasks estimate usage from entry sizes in open-region snapshots and defer new builds when the shared estimate is full. Cleanup still prunes obsolete range entries and expires series indexes to reclaim published usage. Concurrent builds may overshoot; closed-region files are not counted. Series handles and the SST purger own file deletion.
- Series-index coverage (
series_index/catalog.rs):SeriesIndexEntrystores compaction-window width and SST summaries keyed by aligned start in both catalogs and Parquet footers. - Request types (
request.rs): usually tied to proto definitions consumed bydatanode. - WAL/memtable encoding (
wal/,memtable/): breaks replay if changed incompatibly.
Testing
cargo nextest run -p mito2
Tests live next to the code as *_test.rs (e.g. src/mito2/src/engine/flush_test.rs).
TestEnv in test_util.rs spins up an engine over an in-process object store.
Gotchas
- Parquet row groups have uniform row counts except for the final group
(
sst/parquet/row_selection.rs). Memory-pressure flushing must not create arbitrary partial groups: readers and index row IDs rely on this layout. - Sequence numbers are strictly increasing per region; dedup and snapshot reads depend on this. Do not change assignment lightly.
- Manifest version is monotonic — never reset or skip it. Staging takes no version:
its actions stay in memory (
manifest/manager.rs) and are committed once, as the next version, when staging ends (MitoRegion::end_staging_locked). - In regions whose writes require a lease and whose WAL is the object store WAL,
each manifest version has one fixed object name, created conditionally. A late
create can succeed again once checkpoint cleanup frees the name, so a commit
checks its result (
ensure_version_not_checkpointedinmanifest/storage.rs). - In those regions a commit after the region's Leader deadline sends nothing, and a version
conflict fences the region (
ManifestContext::fence_for_reload): no lease or catchup promotes it until the worker swaps in an instance reloaded from the new manifest (worker/handle_reload.rs). Never await a worker sender while holding a manifest guard. - Pending tails (
RegionManifest::pending_tails) are WAL ranges a region still has to recover; the flushed entry id never prunes them. Opening a region replays its WAL afterRegionManifest::wal_replay_start. The WAL releases at open, after a flush, and on close stay below this node's tails (cap_wal_release); truncate, drop, offline cleanup and catchup release without that cap. Handoff floors are dropped once the flushed entry id reaches them, inRegionManifestBuilder::try_build. While a region has pending tails, scans and memtable/SST stats fail withRegionPendingTails, and so do truncate, entering staging, a repartition flush, a close withflush_on_close, and every ALTER other than adding nullable columns without a default or changing options; compaction pauses. These checks run before any region state changes. Full-listing GC skips a region that has pending tails or whose manifesthas_update()reports newer, or fails to check. - A leader that is writable, altering or editing replays its pending tails in one
background task (
tail_replay.rs), started by the worker when a lease, catchup or role change makes the region writable, or when it registers a region created or opened as a writable leader. Each tail goes into its own SSTs, since an SST can give all its rows its largest sequence. The task commits outside the edit queue (ManifestContext::update_manifest_for_tails) and always reports its end, which lets the worker start the next replay or reload a fenced region. The edit that records tails must raise the committed sequence to theirs, or writes accepted during the replay lose to replayed rows. This node's later entry ids must be above every tail'supto, and its unflushed entries at or below one must be in its own tails, or the tail commit's flushed entry id skips them. Replays have no admission or concurrency limit. Their SSTs build indexes as flushed ones (IndexBuildType::Flush), undercreate_on_flush. - Lock ordering: take the manifest lock before updating
version_control; the reverse deadlocks against concurrent flush/compaction. - All region I/O runs on tokio workers; never
block_oninside a worker. - Loaded regions have resident compaction state, including
Idleand handed-off DDLs. Publish create/open/catchup results through the worker'sregister_region; only close/drop unregisters them. Seecompaction/scheduler/state.rs. - Explicit truncate retires shared references too (
worker/handle_truncate.rs):Allremoves all data;ByTimeRangesremoves fully contained files. Other regions/readers still protect the shared objects. Stale compaction results must not reinstate truncated inputs. - Manifest and runtime file views share immutable
FileMetaRefvalues. The engine-local metadata interner compares complete metadata before sharing; file identity alone is insufficient. Partition expressions are shared by table and complete expression content, so historical SST partition rules remain distinct after repartition. Interned references reclaim entries on final release, including delayed readers, without a maintenance task or new writes. Hash table capacity remains allocated until the interner is dropped. Manifest consumers receive the interner directly. Scrape-time entry counts use weak references and do not add maintenance tasks or retain engine storage. Index replacement preserves old readers and updates the lifetime guard to the newest metadata. - Local compaction units use the snapshot-free
CompactionContext. Retaining a fullCompactionRegionin a unit pins other units' inputs and delays SST purge. - Candidate waiting history is owned by the region scheduler (
scheduler/history.rs). Background planning works on a private snapshot; only a fenced result may replace resident history, and an accepted plan consumes the fairness turn for each selected key. Execution success or failure must not rewrite waiting ages. TWCS discovery must not mutate scheduler state. - Compaction admission is engine-wide (
compaction/scheduler/coordinator.rs), but pick/DDL state remains worker-owned. Repick only after every batch unit'sReleased; FIFO submission is not admission and pipeline children share the parent unit's slot. Keep service accounting across batches until region unload. - In flat batches, equal
__primary_keydictionary keys imply the same series, but not the reverse:wrap_pk_binary_to_dictgives every row its own key. Compare key bytes when grouping rows by series. - Staging regions, ending staging included, never flush: a flush would stage its SST and
release the rows' WAL, losing them if staging does not end successfully. Every flush
trigger in
worker/handle_flush.rsmust skip them: background triggers testis_writable()andis_staging(), a flush requestis_flushable()andis_staging()(is_writable()andis_flushable()are false while ending staging).handle_closeleaves staging first. Tail replays never run for them: entering staging fails while tails are pending. - Direct SST imports without a staging binding cannot publish into staging.
worker/handle_manifest.rsseals edit admission while EnterStaging drains accepted edits. Scheduler-owned DDLs resume through execution, bypassing initial admission.
Maintenance contract
Update this file when you add/rename a top-level module, change the write/read path entry points, or alter a persisted format (manifest, SST, WAL).