Files
greptimedb/src/mito2/AGENTS.md
T
Weny Xu beb1f88696 feat: persist bounded WAL releases on region close (#10329)
* feat(log-store): persist bounded region releases in shared WAL batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): release safe WAL history when closing regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover bounded WAL release across close and restart

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): clamp bounded releases to durable entries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): observe close release failures and cover recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: describe bounded WAL release lifecycle

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): make bounded release fence sealing deterministic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): bound queued releases and ignore shutdown hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): cover close release persistence and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): wait for failed close flush cleanup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify bounded WAL release backpressure and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(mito2): clarify staging test retry invariant

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): back off durable release retries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): drive release retry after backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): share and verify WAL retry backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify retained WAL release retry semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 12:44:46 +00:00

14 KiB

mito2 — Agent & Contributor Guide

Navigation aid for src/mito2. Keep it short and point to code; do not duplicate the code here. Paths are relative to the repo root.

Repo-wide rules that apply here: .agents/architecture-invariants.md.

What this crate does

Mito2 is GreptimeDB's primary time-series region storage engine. It owns the write path (memtable + WAL), flushing memtables to Parquet SST files, TWCS/windowed compaction, and the read path (multi-level merge + dedup with snapshot isolation). It implements the RegionEngine trait from store-api.

Module map

Module Path Purpose
engine src/mito2/src/engine.rs MitoEngine (the RegionEngine impl) and request dispatch
worker src/mito2/src/worker.rs, src/mito2/src/worker/ Per-region worker loop; write/alter/flush handlers
region src/mito2/src/region.rs, src/mito2/src/region/version.rs MitoRegion state and copy-on-write VersionControl snapshots
request src/mito2/src/request.rs WriteRequest/RegionRequest types and result channels
wal src/mito2/src/wal.rs Write-ahead log wrapper over log-store
memtable src/mito2/src/memtable/ In-memory write buffers (time-series / bulk / partition)
flush src/mito2/src/flush.rs FlushScheduler, WriteBufferManager, memtable → SST
compaction src/mito2/src/compaction/ Compaction scheduler (scheduler.rs + scheduler/), TWCS picker, strict-window manual picker, compactor, memory control
compaction/multi_region src/mito2/src/compaction/multi_region.rs Enterprise lossless routing by a captured table partition rule; selected SSTs to per-region outputs, without manifest publication
engine/table_compaction src/mito2/src/engine/table_compaction.rs Enterprise conditional publication of table-compaction results: shared-input snapshot, verified-output checks, and the manifest-locked swap of shared inputs for L0 outputs
access_layer src/mito2/src/access_layer.rs SST read/write over the object store
sst src/mito2/src/sst/ Parquet format, file metadata, index layout
series_index src/mito2/src/series_index/ Series index writer/searcher, bucket planning, catalogs, worker-owned reconciliation (maintenance.rs), shared approximate usage (task.rs), catalogs, and index lifecycle
read src/mito2/src/read/ ScanRegion, merge, dedup, projection, streaming
manifest src/mito2/src/manifest/ RegionManifestManager, manifest actions/edits
cache src/mito2/src/cache.rs Write/file/page caches
gc src/mito2/src/gc.rs, src/mito2/src/gc/ Dropped-file cleanup worker
schedule src/mito2/src/schedule/ Local/remote background job scheduling
remap_manifest src/mito2/src/remap_manifest.rs Manifest path remapping for region copy/migration
config src/mito2/src/config.rs MitoConfig tuning knobs
tail_replay src/mito2/src/tail_replay.rs, src/mito2/src/worker/handle_tail_replay.rs Background replay of pending WAL tails into SSTs, published with one manifest edit
temp_file_sweeper src/mito2/src/temp_file_sweeper.rs Reclaims temporary files a crashed writer left in a region, triggered when the region becomes a writable leader or a create opens an existing region (under the hdfs-object-store feature)
test_util src/mito2/src/test_util.rs TestEnv and builders (under the test feature)

Write path

MitoEngine::handle_request (engine.rs), or handle_batch_put_requests for the puts of one request to several regions (one message per worker) → worker loop (worker/handle_write.rs) → sequence + WAL assembly (region_write_ctx.rs) → wal.rs → memtable (memtable/) → when buffer pressure trips, flush.rs writes SSTs via access_layer.rs and appends a RegionEdit to the manifest (manifest/manager.rs).

Read path

MitoEngine::handle_query (engine.rs) → read/scan_region.rs takes an immutable Version (region/version.rs) → scans memtables and Parquet SSTs (sst/parquet.rs) → merges (read/) and dedups by sequence → projected, filtered RecordBatch stream.

Internal series merges use the query runtime pool forwarded through QueryScanContext; reader and output-batch accounting uses the separate scan memory tracker. Direct scanner callers default to an unbounded operator pool.

Live compaction queries read shared per-region state directly from MitoRegion, without sending a worker request. The scheduler publishes active and pending task state together; readers clone task Arcs under a short read lock, then filter and build rows outside the lock. Execution handles provide live background phases. There is no second task registry or completed-task history. See compaction/scheduler/state.rs. The datanode provider reuses the lazy inspection scan path and retains residual SQL filters. Task cancellation uses RegionRequest::CancelCompaction through Region.Handle; the owning worker matches the live task ID without entering the DDL queue. Task-specific cancellation preserves other pending manual work when no DDL is pending. DDL cancellation retains its existing scheduling fence and priority.

Public surface

  • Entry: MitoEngine in src/mito2/src/engine.rs, built via MitoEngineBuilder.
  • Trait: impl RegionEngine for MitoEngine (store-api's region engine contract).
  • Consumed by datanode (sends RegionRequests) and the query layer (scans).

When you change X, also touch Y

  • Manifest format (manifest/action.rs): affects crash recovery and follower replay. Keep it backward compatible. Compaction task state does not belong in the manifest; publication uses ordinary file edits after locked input checks. Shared SSTs require Enterprise capability. Compatibility fixtures: tests/compatibility/manifest/table_compaction/ and tests/compatibility/manifest/pending_tails/.
  • SST/Parquet layout (sst/): readers must stay compatible with existing files.
  • Series-index disk limit: tasks estimate usage from entry sizes in open-region snapshots and defer new builds when the shared estimate is full. Cleanup still prunes obsolete range entries and expires series indexes to reclaim published usage. Concurrent builds may overshoot; closed-region files are not counted. Series handles and the SST purger own file deletion.
  • Series-index coverage (series_index/catalog.rs): SeriesIndexEntry stores compaction-window width and SST summaries keyed by aligned start in both catalogs and Parquet footers.
  • Request types (request.rs): usually tied to proto definitions consumed by datanode.
  • WAL/memtable encoding (wal/, memtable/): breaks replay if changed incompatibly.

Testing

cargo nextest run -p mito2

Tests live next to the code as *_test.rs (e.g. src/mito2/src/engine/flush_test.rs). TestEnv in test_util.rs spins up an engine over an in-process object store.

Gotchas

  • Parquet row groups have uniform row counts except for the final group (sst/parquet/row_selection.rs). Memory-pressure flushing must not create arbitrary partial groups: readers and index row IDs rely on this layout.
  • Sequence numbers are strictly increasing per region; dedup and snapshot reads depend on this. Do not change assignment lightly.
  • Manifest version is monotonic — never reset or skip it. Staging takes no version: its actions stay in memory (manifest/manager.rs) and are committed once, as the next version, when staging ends (MitoRegion::end_staging_locked).
  • In regions whose writes require a lease and whose WAL is the object store WAL, each manifest version has one fixed object name, created conditionally. A late create can succeed again once checkpoint cleanup frees the name, so a commit checks its result (ensure_version_not_checkpointed in manifest/storage.rs).
  • In those regions a commit after the region's Leader deadline sends nothing, and a version conflict fences the region (ManifestContext::fence_for_reload): no lease or catchup promotes it until the worker swaps in an instance reloaded from the new manifest (worker/handle_reload.rs). Never await a worker sender while holding a manifest guard.
  • Pending tails (RegionManifest::pending_tails) are WAL ranges a region still has to recover; the flushed entry id never prunes them. Opening a region replays its WAL after RegionManifest::wal_replay_start. The WAL releases at open, after a flush, and on close stay below this node's tails (cap_wal_release); truncate, drop, offline cleanup and catchup release without that cap. Handoff floors are dropped once the flushed entry id reaches them, in RegionManifestBuilder::try_build. While a region has pending tails, scans and memtable/SST stats fail with RegionPendingTails, and so do truncate, entering staging, a repartition flush, a close with flush_on_close, and every ALTER other than adding nullable columns without a default or changing options; compaction pauses. These checks run before any region state changes. Full-listing GC skips a region that has pending tails or whose manifest has_update() reports newer, or fails to check.
  • A leader that is writable, altering or editing replays its pending tails in one background task (tail_replay.rs), started by the worker when a lease, catchup or role change makes the region writable, or when it registers a region created or opened as a writable leader. Each tail goes into its own SSTs, since an SST can give all its rows its largest sequence. The task commits outside the edit queue (ManifestContext::update_manifest_for_tails) and always reports its end, which lets the worker start the next replay or reload a fenced region. The edit that records tails must raise the committed sequence to theirs, or writes accepted during the replay lose to replayed rows. This node's later entry ids must be above every tail's upto, and its unflushed entries at or below one must be in its own tails, or the tail commit's flushed entry id skips them. Replays have no admission or concurrency limit. Their SSTs build indexes as flushed ones (IndexBuildType::Flush), under create_on_flush.
  • Lock ordering: take the manifest lock before updating version_control; the reverse deadlocks against concurrent flush/compaction.
  • All region I/O runs on tokio workers; never block_on inside a worker.
  • Loaded regions have resident compaction state, including Idle and handed-off DDLs. Publish create/open/catchup results through the worker's register_region; only close/drop unregisters them. See compaction/scheduler/state.rs.
  • Explicit truncate retires shared references too (worker/handle_truncate.rs): All removes all data; ByTimeRanges removes fully contained files. Other regions/readers still protect the shared objects. Stale compaction results must not reinstate truncated inputs.
  • Manifest and runtime file views share immutable FileMetaRef values. The engine-local metadata interner compares complete metadata before sharing; file identity alone is insufficient. Partition expressions are shared by table and complete expression content, so historical SST partition rules remain distinct after repartition. Interned references reclaim entries on final release, including delayed readers, without a maintenance task or new writes. Hash table capacity remains allocated until the interner is dropped. Manifest consumers receive the interner directly. Scrape-time entry counts use weak references and do not add maintenance tasks or retain engine storage. Index replacement preserves old readers and updates the lifetime guard to the newest metadata.
  • Local compaction units use the snapshot-free CompactionContext. Retaining a full CompactionRegion in a unit pins other units' inputs and delays SST purge.
  • Candidate waiting history is owned by the region scheduler (scheduler/history.rs). Background planning works on a private snapshot; only a fenced result may replace resident history, and an accepted plan consumes the fairness turn for each selected key. Execution success or failure must not rewrite waiting ages. TWCS discovery must not mutate scheduler state.
  • Compaction admission is engine-wide (compaction/scheduler/coordinator.rs), but pick/DDL state remains worker-owned. Repick only after every batch unit's Released; FIFO submission is not admission and pipeline children share the parent unit's slot. Keep service accounting across batches until region unload.
  • In flat batches, equal __primary_key dictionary keys imply the same series, but not the reverse: wrap_pk_binary_to_dict gives every row its own key. Compare key bytes when grouping rows by series.
  • Staging regions, ending staging included, never flush: a flush would stage its SST and release the rows' WAL, losing them if staging does not end successfully. Every flush trigger in worker/handle_flush.rs must skip them: background triggers test is_writable() and is_staging(), a flush request is_flushable() and is_staging() (is_writable() and is_flushable() are false while ending staging). handle_close leaves staging first. Tail replays never run for them: entering staging fails while tails are pending.
  • Direct SST imports without a staging binding cannot publish into staging. worker/handle_manifest.rs seals edit admission while EnterStaging drains accepted edits. Scheduler-owned DDLs resume through execution, bypassing initial admission.

Maintenance contract

Update this file when you add/rename a top-level module, change the write/read path entry points, or alter a persisted format (manifest, SST, WAL).