mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-21 10:14:06 +00:00
* fix(mito2): publish manifest deltas by conditional create under a lease
A late old leader and the region's new holder can commit the same manifest
version. For a region whose writes require a lease and whose WAL is the object
store WAL, the manifest now behaves as follows. Every other region keeps its
behaviour and issues no extra request.
- Each delta has one fixed name, `{version:020}.json.gz`, written by a
conditional create. Without `compress_manifest` the delta is written at gzip
level 0, so that all writers share the name.
- On an existing object, the writer reads it from storage: identical content is
a retry of a write that landed, anything else is a retryable
`ManifestVersionConflict` that keeps the output files.
- The version advances only after a successful save.
- Recovery loads the highest checkpoint found by LIST and uses the checkpoint
pointer only for its checksum, because a stale writer's checkpoint task can
leave the pointer at an older version.
- Checkpoint cleanup frees names, so a late create can still succeed on a
deleted name. A commit whose save returns after the region's Leader deadline
lists the checkpoints once: a checkpoint at or above its version, or a LIST
error, is a conflict; otherwise it applies but starts no checkpoint. A commit
within the deadline does one conditional PUT and no LIST.
`RegionManifestManager::update` becomes the one commit entry and takes
`allow_checkpoint` and `must_verify` as required parameters. A backend that does
not declare conditional create is refused at open.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): treat every LIST error as a conflict when verifying a commit
The listing helper turned a NotFound from lister creation into an empty result,
so the check after a late commit could read a failed LIST as proof that no
checkpoint covers the version. List the checkpoints directly, so that any LIST
error reaches the caller. Recovery still treats a missing directory as having no
checkpoint.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep the LIST failure in manifest conflicts and follow the style guide
A failed LIST during commit verification was reported as "OpenDAL operator
failed", because the conflict reason used the display of the wrapper error.
Report the root cause through `output_msg` and assert it in the test.
Align the change with docs/style-guide.md: name boolean methods `is_*`, name
the listing helper `list_highest_checkpoint` and move it after the public
methods, qualify `encode_gzip_stored` through its module, and name the mode
setter after the field it sets, which is now `pub(crate)`. Move the shared test
helpers to `crate::test_util` and `manifest/tests/utils.rs` so that the manifest
test modules stay private.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): order the manifest test helpers public before private
Move the private `NotFoundReader` and `CheckpointTaskBlocker::block_write`
below the public items next to them, and document
`fail_manifest_delta_reads_layer`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): fail recovery when a listed checkpoint disappears
Recovery treated a checkpoint that LIST found but that cleanup removed before
the read as no checkpoint at all. It then replayed from the first version over
deltas that were already deleted, and could report an empty region or build a
manifest from the surviving tail. Return a retryable version conflict instead.
Record the size of the stored object when a retry finds an identical delta, as
the other writer may have encoded it differently. Reopen the region in the test
of a commit that crosses its deadline, and reduce the AGENTS.md entry to the
invariant.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): verify conditional commits from callers without a deadline
Restore `RegionManifestManager::update(action_list, is_staging)`. Without a
Leader deadline a caller cannot show that its commit returned in time, so a
normal commit under conditional publication is now always verified on this
path and starts no checkpoint.
Only `ManifestContext::update_locked` has a deadline, so the variant that takes
`allow_checkpoint` and `must_verify` becomes `update_with_verification` and is
crate private. `sst-replace` and the existing tests return to the two-argument
call.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): name the verifying manifest update in the hook docs
`ManifestContext::update_locked` now calls `update_with_verification`, and the
public `update` also calls it, so neither doc can call `update_locked` the sole
caller of `RegionManifestManager::update`.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(mito2): keep the conditional publication rule in region options
Move the rule that decides which regions publish manifests conditionally
to RegionOptions, so the manifest options only hold the result. Report a
checkpoint that vanished after LIST with its own error instead of a
version conflict, and keep the stored gzip encoder with the other
CompressionType helpers.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>