* fix(mito2): publish manifest deltas by conditional create under a lease
A late old leader and the region's new holder can commit the same manifest
version. For a region whose writes require a lease and whose WAL is the object
store WAL, the manifest now behaves as follows. Every other region keeps its
behaviour and issues no extra request.
- Each delta has one fixed name, `{version:020}.json.gz`, written by a
conditional create. Without `compress_manifest` the delta is written at gzip
level 0, so that all writers share the name.
- On an existing object, the writer reads it from storage: identical content is
a retry of a write that landed, anything else is a retryable
`ManifestVersionConflict` that keeps the output files.
- The version advances only after a successful save.
- Recovery loads the highest checkpoint found by LIST and uses the checkpoint
pointer only for its checksum, because a stale writer's checkpoint task can
leave the pointer at an older version.
- Checkpoint cleanup frees names, so a late create can still succeed on a
deleted name. A commit whose save returns after the region's Leader deadline
lists the checkpoints once: a checkpoint at or above its version, or a LIST
error, is a conflict; otherwise it applies but starts no checkpoint. A commit
within the deadline does one conditional PUT and no LIST.
`RegionManifestManager::update` becomes the one commit entry and takes
`allow_checkpoint` and `must_verify` as required parameters. A backend that does
not declare conditional create is refused at open.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): treat every LIST error as a conflict when verifying a commit
The listing helper turned a NotFound from lister creation into an empty result,
so the check after a late commit could read a failed LIST as proof that no
checkpoint covers the version. List the checkpoints directly, so that any LIST
error reaches the caller. Recovery still treats a missing directory as having no
checkpoint.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep the LIST failure in manifest conflicts and follow the style guide
A failed LIST during commit verification was reported as "OpenDAL operator
failed", because the conflict reason used the display of the wrapper error.
Report the root cause through `output_msg` and assert it in the test.
Align the change with docs/style-guide.md: name boolean methods `is_*`, name
the listing helper `list_highest_checkpoint` and move it after the public
methods, qualify `encode_gzip_stored` through its module, and name the mode
setter after the field it sets, which is now `pub(crate)`. Move the shared test
helpers to `crate::test_util` and `manifest/tests/utils.rs` so that the manifest
test modules stay private.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): order the manifest test helpers public before private
Move the private `NotFoundReader` and `CheckpointTaskBlocker::block_write`
below the public items next to them, and document
`fail_manifest_delta_reads_layer`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): fail recovery when a listed checkpoint disappears
Recovery treated a checkpoint that LIST found but that cleanup removed before
the read as no checkpoint at all. It then replayed from the first version over
deltas that were already deleted, and could report an empty region or build a
manifest from the surviving tail. Return a retryable version conflict instead.
Record the size of the stored object when a retry finds an identical delta, as
the other writer may have encoded it differently. Reopen the region in the test
of a commit that crosses its deadline, and reduce the AGENTS.md entry to the
invariant.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): verify conditional commits from callers without a deadline
Restore `RegionManifestManager::update(action_list, is_staging)`. Without a
Leader deadline a caller cannot show that its commit returned in time, so a
normal commit under conditional publication is now always verified on this
path and starts no checkpoint.
Only `ManifestContext::update_locked` has a deadline, so the variant that takes
`allow_checkpoint` and `must_verify` becomes `update_with_verification` and is
crate private. `sst-replace` and the existing tests return to the two-argument
call.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): name the verifying manifest update in the hook docs
`ManifestContext::update_locked` now calls `update_with_verification`, and the
public `update` also calls it, so neither doc can call `update_locked` the sole
caller of `RegionManifestManager::update`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep the conditional publication rule in region options
Move the rule that decides which regions publish manifests conditionally
to RegionOptions, so the manifest options only hold the result. Report a
checkpoint that vanished after LIST with its own error instead of a
version conflict, and keep the stored gzip encoder with the other
CompressionType helpers.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
8.2 KiB
mito2 — Agent & Contributor Guide
Navigation aid for src/mito2. Keep it short and point to code; do not duplicate
the code here. Paths are relative to the repo root.
Repo-wide rules that apply here: .agents/architecture-invariants.md.
What this crate does
Mito2 is GreptimeDB's primary time-series region storage engine. It owns the
write path (memtable + WAL), flushing memtables to Parquet SST files,
TWCS/windowed compaction, and the read path (multi-level merge + dedup with
snapshot isolation). It implements the RegionEngine trait from store-api.
Module map
| Module | Path | Purpose |
|---|---|---|
engine |
src/mito2/src/engine.rs |
MitoEngine (the RegionEngine impl) and request dispatch |
worker |
src/mito2/src/worker.rs, src/mito2/src/worker/ |
Per-region worker loop; write/alter/flush handlers |
region |
src/mito2/src/region.rs, src/mito2/src/region/version.rs |
MitoRegion state and copy-on-write VersionControl snapshots |
request |
src/mito2/src/request.rs |
WriteRequest/RegionRequest types and result channels |
wal |
src/mito2/src/wal.rs |
Write-ahead log wrapper over log-store |
memtable |
src/mito2/src/memtable/ |
In-memory write buffers (time-series / bulk / partition) |
flush |
src/mito2/src/flush.rs |
FlushScheduler, WriteBufferManager, memtable → SST |
compaction |
src/mito2/src/compaction/ |
Compaction scheduler (scheduler.rs + scheduler/), TWCS picker, strict-window manual picker, compactor, memory control |
compaction/multi_region |
src/mito2/src/compaction/multi_region.rs |
Enterprise lossless routing by a captured table partition rule; selected SSTs to per-region outputs, without manifest publication |
access_layer |
src/mito2/src/access_layer.rs |
SST read/write over the object store |
sst |
src/mito2/src/sst/ |
Parquet format, file metadata, index layout |
series_index |
src/mito2/src/series_index/ |
Series index writer/searcher, bucket planning, catalogs, worker-owned reconciliation (maintenance.rs), shared approximate usage (task.rs), catalogs, and index lifecycle |
read |
src/mito2/src/read/ |
ScanRegion, merge, dedup, projection, streaming |
manifest |
src/mito2/src/manifest/ |
RegionManifestManager, manifest actions/edits |
cache |
src/mito2/src/cache.rs |
Write/file/page caches |
gc |
src/mito2/src/gc.rs, src/mito2/src/gc/ |
Dropped-file cleanup worker |
schedule |
src/mito2/src/schedule/ |
Local/remote background job scheduling |
remap_manifest |
src/mito2/src/remap_manifest.rs |
Manifest path remapping for region copy/migration |
config |
src/mito2/src/config.rs |
MitoConfig tuning knobs |
temp_file_sweeper |
src/mito2/src/temp_file_sweeper.rs |
Reclaims temporary files a crashed writer left in a region, triggered when the region becomes a writable leader or a create opens an existing region (under the hdfs-object-store feature) |
test_util |
src/mito2/src/test_util.rs |
TestEnv and builders (under the test feature) |
Write path
MitoEngine::handle_request (engine.rs), or handle_batch_put_requests for the puts
of one request to several regions (one message per worker) → worker loop
(worker/handle_write.rs) → sequence + WAL assembly (region_write_ctx.rs) →
wal.rs → memtable (memtable/) → when buffer pressure trips, flush.rs
writes SSTs via access_layer.rs and appends a RegionEdit to the manifest
(manifest/manager.rs).
Read path
MitoEngine::handle_query (engine.rs) → read/scan_region.rs takes an
immutable Version (region/version.rs) → scans memtables and Parquet SSTs
(sst/parquet.rs) → merges (read/) and dedups by sequence → projected,
filtered RecordBatch stream.
Internal series merges use the query runtime pool forwarded through
QueryScanContext; reader and output-batch accounting uses the separate scan
memory tracker. Direct scanner callers default to an unbounded operator pool.
Live compaction queries read shared per-region state directly from MitoRegion,
without sending a worker request. The scheduler publishes active and pending task
state together; readers clone task Arcs under a short read lock, then filter and
build rows outside the lock. Execution handles provide live background phases.
There is no second task registry or completed-task history. See
compaction/scheduler/state.rs. The datanode provider reuses the lazy inspection scan
path and retains residual SQL filters.
Public surface
- Entry:
MitoEngineinsrc/mito2/src/engine.rs, built viaMitoEngineBuilder. - Trait:
impl RegionEngine for MitoEngine(store-api's region engine contract). - Consumed by
datanode(sendsRegionRequests) and the query layer (scans).
When you change X, also touch Y
- Manifest format (
manifest/action.rs): affects crash recovery and follower replay. Keep it backward compatible. - SST/Parquet layout (
sst/): readers must stay compatible with existing files. - Series-index disk limit: tasks estimate usage from entry sizes in open-region snapshots and defer new builds when the shared estimate is full. Cleanup still prunes obsolete range entries and expires series indexes to reclaim published usage. Concurrent builds may overshoot; closed-region files are not counted. Series handles and the SST purger own file deletion.
- Series-index coverage (
series_index/catalog.rs):SeriesIndexEntrystores compaction-window width and SST summaries keyed by aligned start in both catalogs and Parquet footers. - Request types (
request.rs): usually tied to proto definitions consumed bydatanode. - WAL/memtable encoding (
wal/,memtable/): breaks replay if changed incompatibly.
Testing
cargo nextest run -p mito2
Tests live next to the code as *_test.rs (e.g. src/mito2/src/engine/flush_test.rs).
TestEnv in test_util.rs spins up an engine over an in-process object store.
Gotchas
- Parquet row groups have uniform row counts except for the final group
(
sst/parquet/row_selection.rs). Memory-pressure flushing must not create arbitrary partial groups: readers and index row IDs rely on this layout. - Sequence numbers are strictly increasing per region; dedup and snapshot reads depend on this. Do not change assignment lightly.
- Manifest version is monotonic — never reset or skip it.
- In regions whose writes require a lease and whose WAL is the object store WAL,
each manifest version has one fixed object name and only one writer can create
it (
manifest/storage.rs). - Lock ordering: take the manifest lock before updating
version_control; the reverse deadlocks against concurrent flush/compaction. - All region I/O runs on tokio workers; never
block_oninside a worker. - Loaded regions have resident compaction state, including
Idleand handed-off DDLs. Publish create/open/catchup results through the worker'sregister_region; only close/drop unregisters them. Seecompaction/scheduler/state.rs. - Local compaction units use the snapshot-free
CompactionContext. Retaining a fullCompactionRegionin a unit pins other units' inputs and delays SST purge. - Candidate waiting history is owned by the region scheduler (
scheduler/history.rs). Background planning works on a private snapshot; only a fenced result may replace resident history, and an accepted plan consumes the fairness turn for each selected key. Execution success or failure must not rewrite waiting ages. TWCS discovery must not mutate scheduler state. - Compaction admission is engine-wide (
compaction/scheduler/coordinator.rs), but pick/DDL state remains worker-owned. Repick only after every batch unit'sReleased; FIFO submission is not admission and pipeline children share the parent unit's slot. Keep service accounting across batches until region unload. - In flat batches, equal
__primary_keydictionary keys imply the same series, but not the reverse:wrap_pk_binary_to_dictgives every row its own key. Compare key bytes when grouping rows by series.
Maintenance contract
Update this file when you add/rename a top-level module, change the write/read path entry points, or alter a persisted format (manifest, SST, WAL).