Files
greptimedb/src/mito2/AGENTS.md
T
jeremyhi 50a2c4d378 fix(mito2): publish manifest deltas by conditional create under a lease (#10234)
* fix(mito2): publish manifest deltas by conditional create under a lease

A late old leader and the region's new holder can commit the same manifest
version. For a region whose writes require a lease and whose WAL is the object
store WAL, the manifest now behaves as follows. Every other region keeps its
behaviour and issues no extra request.

- Each delta has one fixed name, `{version:020}.json.gz`, written by a
  conditional create. Without `compress_manifest` the delta is written at gzip
  level 0, so that all writers share the name.
- On an existing object, the writer reads it from storage: identical content is
  a retry of a write that landed, anything else is a retryable
  `ManifestVersionConflict` that keeps the output files.
- The version advances only after a successful save.
- Recovery loads the highest checkpoint found by LIST and uses the checkpoint
  pointer only for its checksum, because a stale writer's checkpoint task can
  leave the pointer at an older version.
- Checkpoint cleanup frees names, so a late create can still succeed on a
  deleted name. A commit whose save returns after the region's Leader deadline
  lists the checkpoints once: a checkpoint at or above its version, or a LIST
  error, is a conflict; otherwise it applies but starts no checkpoint. A commit
  within the deadline does one conditional PUT and no LIST.

`RegionManifestManager::update` becomes the one commit entry and takes
`allow_checkpoint` and `must_verify` as required parameters. A backend that does
not declare conditional create is refused at open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): treat every LIST error as a conflict when verifying a commit

The listing helper turned a NotFound from lister creation into an empty result,
so the check after a late commit could read a failed LIST as proof that no
checkpoint covers the version. List the checkpoints directly, so that any LIST
error reaches the caller. Recovery still treats a missing directory as having no
checkpoint.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the LIST failure in manifest conflicts and follow the style guide

A failed LIST during commit verification was reported as "OpenDAL operator
failed", because the conflict reason used the display of the wrapper error.
Report the root cause through `output_msg` and assert it in the test.

Align the change with docs/style-guide.md: name boolean methods `is_*`, name
the listing helper `list_highest_checkpoint` and move it after the public
methods, qualify `encode_gzip_stored` through its module, and name the mode
setter after the field it sets, which is now `pub(crate)`. Move the shared test
helpers to `crate::test_util` and `manifest/tests/utils.rs` so that the manifest
test modules stay private.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): order the manifest test helpers public before private

Move the private `NotFoundReader` and `CheckpointTaskBlocker::block_write`
below the public items next to them, and document
`fail_manifest_delta_reads_layer`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): fail recovery when a listed checkpoint disappears

Recovery treated a checkpoint that LIST found but that cleanup removed before
the read as no checkpoint at all. It then replayed from the first version over
deltas that were already deleted, and could report an empty region or build a
manifest from the surviving tail. Return a retryable version conflict instead.

Record the size of the stored object when a retry finds an identical delta, as
the other writer may have encoded it differently. Reopen the region in the test
of a commit that crosses its deadline, and reduce the AGENTS.md entry to the
invariant.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): verify conditional commits from callers without a deadline

Restore `RegionManifestManager::update(action_list, is_staging)`. Without a
Leader deadline a caller cannot show that its commit returned in time, so a
normal commit under conditional publication is now always verified on this
path and starts no checkpoint.

Only `ManifestContext::update_locked` has a deadline, so the variant that takes
`allow_checkpoint` and `must_verify` becomes `update_with_verification` and is
crate private. `sst-replace` and the existing tests return to the two-argument
call.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): name the verifying manifest update in the hook docs

`ManifestContext::update_locked` now calls `update_with_verification`, and the
public `update` also calls it, so neither doc can call `update_locked` the sole
caller of `RegionManifestManager::update`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep the conditional publication rule in region options

Move the rule that decides which regions publish manifests conditionally
to RegionOptions, so the manifest options only hold the result. Report a
checkpoint that vanished after LIST with its own error instead of a
version conflict, and keep the stored gzip encoder with the other
CompressionType helpers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-09 11:59:33 +00:00

8.2 KiB

mito2 — Agent & Contributor Guide

Navigation aid for src/mito2. Keep it short and point to code; do not duplicate the code here. Paths are relative to the repo root.

Repo-wide rules that apply here: .agents/architecture-invariants.md.

What this crate does

Mito2 is GreptimeDB's primary time-series region storage engine. It owns the write path (memtable + WAL), flushing memtables to Parquet SST files, TWCS/windowed compaction, and the read path (multi-level merge + dedup with snapshot isolation). It implements the RegionEngine trait from store-api.

Module map

Module Path Purpose
engine src/mito2/src/engine.rs MitoEngine (the RegionEngine impl) and request dispatch
worker src/mito2/src/worker.rs, src/mito2/src/worker/ Per-region worker loop; write/alter/flush handlers
region src/mito2/src/region.rs, src/mito2/src/region/version.rs MitoRegion state and copy-on-write VersionControl snapshots
request src/mito2/src/request.rs WriteRequest/RegionRequest types and result channels
wal src/mito2/src/wal.rs Write-ahead log wrapper over log-store
memtable src/mito2/src/memtable/ In-memory write buffers (time-series / bulk / partition)
flush src/mito2/src/flush.rs FlushScheduler, WriteBufferManager, memtable → SST
compaction src/mito2/src/compaction/ Compaction scheduler (scheduler.rs + scheduler/), TWCS picker, strict-window manual picker, compactor, memory control
compaction/multi_region src/mito2/src/compaction/multi_region.rs Enterprise lossless routing by a captured table partition rule; selected SSTs to per-region outputs, without manifest publication
access_layer src/mito2/src/access_layer.rs SST read/write over the object store
sst src/mito2/src/sst/ Parquet format, file metadata, index layout
series_index src/mito2/src/series_index/ Series index writer/searcher, bucket planning, catalogs, worker-owned reconciliation (maintenance.rs), shared approximate usage (task.rs), catalogs, and index lifecycle
read src/mito2/src/read/ ScanRegion, merge, dedup, projection, streaming
manifest src/mito2/src/manifest/ RegionManifestManager, manifest actions/edits
cache src/mito2/src/cache.rs Write/file/page caches
gc src/mito2/src/gc.rs, src/mito2/src/gc/ Dropped-file cleanup worker
schedule src/mito2/src/schedule/ Local/remote background job scheduling
remap_manifest src/mito2/src/remap_manifest.rs Manifest path remapping for region copy/migration
config src/mito2/src/config.rs MitoConfig tuning knobs
temp_file_sweeper src/mito2/src/temp_file_sweeper.rs Reclaims temporary files a crashed writer left in a region, triggered when the region becomes a writable leader or a create opens an existing region (under the hdfs-object-store feature)
test_util src/mito2/src/test_util.rs TestEnv and builders (under the test feature)

Write path

MitoEngine::handle_request (engine.rs), or handle_batch_put_requests for the puts of one request to several regions (one message per worker) → worker loop (worker/handle_write.rs) → sequence + WAL assembly (region_write_ctx.rs) → wal.rs → memtable (memtable/) → when buffer pressure trips, flush.rs writes SSTs via access_layer.rs and appends a RegionEdit to the manifest (manifest/manager.rs).

Read path

MitoEngine::handle_query (engine.rs) → read/scan_region.rs takes an immutable Version (region/version.rs) → scans memtables and Parquet SSTs (sst/parquet.rs) → merges (read/) and dedups by sequence → projected, filtered RecordBatch stream.

Internal series merges use the query runtime pool forwarded through QueryScanContext; reader and output-batch accounting uses the separate scan memory tracker. Direct scanner callers default to an unbounded operator pool.

Live compaction queries read shared per-region state directly from MitoRegion, without sending a worker request. The scheduler publishes active and pending task state together; readers clone task Arcs under a short read lock, then filter and build rows outside the lock. Execution handles provide live background phases. There is no second task registry or completed-task history. See compaction/scheduler/state.rs. The datanode provider reuses the lazy inspection scan path and retains residual SQL filters.

Public surface

  • Entry: MitoEngine in src/mito2/src/engine.rs, built via MitoEngineBuilder.
  • Trait: impl RegionEngine for MitoEngine (store-api's region engine contract).
  • Consumed by datanode (sends RegionRequests) and the query layer (scans).

When you change X, also touch Y

  • Manifest format (manifest/action.rs): affects crash recovery and follower replay. Keep it backward compatible.
  • SST/Parquet layout (sst/): readers must stay compatible with existing files.
  • Series-index disk limit: tasks estimate usage from entry sizes in open-region snapshots and defer new builds when the shared estimate is full. Cleanup still prunes obsolete range entries and expires series indexes to reclaim published usage. Concurrent builds may overshoot; closed-region files are not counted. Series handles and the SST purger own file deletion.
  • Series-index coverage (series_index/catalog.rs): SeriesIndexEntry stores compaction-window width and SST summaries keyed by aligned start in both catalogs and Parquet footers.
  • Request types (request.rs): usually tied to proto definitions consumed by datanode.
  • WAL/memtable encoding (wal/, memtable/): breaks replay if changed incompatibly.

Testing

cargo nextest run -p mito2

Tests live next to the code as *_test.rs (e.g. src/mito2/src/engine/flush_test.rs). TestEnv in test_util.rs spins up an engine over an in-process object store.

Gotchas

  • Parquet row groups have uniform row counts except for the final group (sst/parquet/row_selection.rs). Memory-pressure flushing must not create arbitrary partial groups: readers and index row IDs rely on this layout.
  • Sequence numbers are strictly increasing per region; dedup and snapshot reads depend on this. Do not change assignment lightly.
  • Manifest version is monotonic — never reset or skip it.
  • In regions whose writes require a lease and whose WAL is the object store WAL, each manifest version has one fixed object name and only one writer can create it (manifest/storage.rs).
  • Lock ordering: take the manifest lock before updating version_control; the reverse deadlocks against concurrent flush/compaction.
  • All region I/O runs on tokio workers; never block_on inside a worker.
  • Loaded regions have resident compaction state, including Idle and handed-off DDLs. Publish create/open/catchup results through the worker's register_region; only close/drop unregisters them. See compaction/scheduler/state.rs.
  • Local compaction units use the snapshot-free CompactionContext. Retaining a full CompactionRegion in a unit pins other units' inputs and delays SST purge.
  • Candidate waiting history is owned by the region scheduler (scheduler/history.rs). Background planning works on a private snapshot; only a fenced result may replace resident history, and an accepted plan consumes the fairness turn for each selected key. Execution success or failure must not rewrite waiting ages. TWCS discovery must not mutate scheduler state.
  • Compaction admission is engine-wide (compaction/scheduler/coordinator.rs), but pick/DDL state remains worker-owned. Repick only after every batch unit's Released; FIFO submission is not admission and pipeline children share the parent unit's slot. Keep service accounting across batches until region unload.
  • In flat batches, equal __primary_key dictionary keys imply the same series, but not the reverse: wrap_pk_binary_to_dict gives every row its own key. Compare key bytes when grouping rows by series.

Maintenance contract

Update this file when you add/rename a top-level module, change the write/read path entry points, or alter a persisted format (manifest, SST, WAL).