mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-21 19:32:34 +00:00
jeremyhi
deae75ffee
* fix(log-store): refuse to open object store WAL regions whose history is missing A region persists only its WAL prefix, not the object store that holds it. After `[wal].storage_provider`, or the bucket or root of the provider it names, changes, or after the objects under the prefix are deleted, every region reopened against a prefix without its history, replayed nothing and silently lost its acknowledged writes that were not flushed. Add `LogStore::check_region_history` and `LogStore::start_region_history`, whose defaults accept and do nothing, so Raft Engine and Kafka are unchanged. The object store WAL reports the history of a region as missing when recovery found no object under the prefix, or when the region's watermark names an object above the newest object holding entries that recovery indexed, unless the catalog holds the region's entries through the watermark or the region's history starts in the store. Both rest on invariants of the prefix: an opened prefix is never empty, and a persisted watermark names an object at or below the newest object holding entries, which collection never deletes. The segment of release records counts as entries, since an object holding only release records can be the one collection keeps; start objects hold no segment and do not move the bound. When a store that found its prefix empty refuses a region, the actor gives up its start object, so the next start refuses the region again. It agrees only while nothing was handed an id, no batch, append, release or create waits, no failed release is kept, no other object was listed, the catalog holds only the start object and no region's history started in the store, deciding under the lock region registration takes. It then records a terminal error, deletes the object as one of its tracked deletes, which `stop` waits for, and answers every refusal once that delete settled. Mito runs the check before it replays an existing region on open, on catch-up and when a create finds the region present, with the larger of the flushed and truncated entry ids, and records the history of a region it creates; regions that skip the WAL and noop regions are not checked. A missing history fails the open with an error naming the region, the prefix, the configured storage and how to move the WAL by copying its objects. The new `[wal].accept_missing_history` option, off by default, opens such a region without its unflushed writes instead, with a warning and the counter `greptime_logstore_object_store_wal_missing_history_accepted_total`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the object store WAL history check Describe the open-time history check, the prefix invariants it relies on, how release records bound the history, the start object withdrawal of a refused open and the window that remains, the accept_missing_history option and how to move a WAL by copying its objects, and update the metrics, Mito integration, evidence, compatibility and cluster alignment sections. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): unindex the start object a refused open withdrew A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): separate the withdrawal windows of start objects and entries An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep a successor's chain complete across a late withdrawal A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): record a created region's WAL history before its manifest A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the start boundary of a chain without segments and the create order Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com>