fix(log-store): refuse to open object store WAL regions whose history is missing (#9561)

* fix(log-store): refuse to open object store WAL regions whose history is missing

A region persists only its WAL prefix, not the object store that holds it.
After `[wal].storage_provider`, or the bucket or root of the provider it
names, changes, or after the objects under the prefix are deleted, every
region reopened against a prefix without its history, replayed nothing and
silently lost its acknowledged writes that were not flushed.

Add `LogStore::check_region_history` and `LogStore::start_region_history`,
whose defaults accept and do nothing, so Raft Engine and Kafka are
unchanged. The object store WAL reports the history of a region as missing
when recovery found no object under the prefix, or when the region's
watermark names an object above the newest object holding entries that
recovery indexed, unless the catalog holds the region's entries through the
watermark or the region's history starts in the store. Both rest on
invariants of the prefix: an opened prefix is never empty, and a persisted
watermark names an object at or below the newest object holding entries,
which collection never deletes. The segment of release records counts as
entries, since an object holding only release records can be the one
collection keeps; start objects hold no segment and do not move the bound.

When a store that found its prefix empty refuses a region, the actor gives
up its start object, so the next start refuses the region again. It agrees
only while nothing was handed an id, no batch, append, release or create
waits, no failed release is kept, no other object was listed, the catalog
holds only the start object and no region's history started in the store,
deciding under the lock region registration takes. It then records a
terminal error, deletes the object as one of its tracked deletes, which
`stop` waits for, and answers every refusal once that delete settled.

Mito runs the check before it replays an existing region on open, on
catch-up and when a create finds the region present, with the larger of
the flushed and truncated entry ids, and records the history of a region
it creates; regions that skip the WAL and noop regions are not checked. A
missing history fails the open with an error naming the region, the
prefix, the configured storage and how to move the WAL by copying its
objects. The new `[wal].accept_missing_history` option, off by default,
opens such a region without its unflushed writes instead, with a warning
and the counter
`greptime_logstore_object_store_wal_missing_history_accepted_total`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the object store WAL history check

Describe the open-time history check, the prefix invariants it relies on,
how release records bound the history, the start object withdrawal of a
refused open and the window that remains, the accept_missing_history
option and how to move a WAL by copying its objects, and update the
metrics, Mito integration, evidence, compatibility and cluster alignment
sections.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): unindex the start object a refused open withdrew

A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): separate the withdrawal windows of start objects and entries

An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep a successor's chain complete across a late withdrawal

A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a created region's WAL history before its manifest

A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the start boundary of a chain without segments and the create order

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

config/config.md

@@ -131,6 +131,7 @@
 | `wal.max_unpersisted_bytes` | String | `64MB` | The size of the unpersisted backlog at which new appends stall until an upload completes, in `enqueued` mode.<br/>Nothing is dropped and nothing is rejected; the threshold does not bound what a crash during an outage can lose.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.max_unpersisted_age` | String | `8s` | The age of the oldest unpersisted entry at which new appends stall until an upload completes, in `enqueued` mode.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.on_corrupted_segment` | String | `skip` | What a read does with a segment that still does not decode after a second fetch, because its checksum does not match or its content disagrees with its footer entry.<br/>- `skip`: the segment is skipped and recorded as a WAL hole of its region, a metric is incremented and a warning is logged; the other regions of the object are unaffected (the default).<br/>- `fail`: the read fails, so the region does not open.<br/>**It's only used when the provider is `experimental_object_store`**. |
+| `wal.accept_missing_history` | Bool | `false` | Whether a region opens although the WAL objects under the prefix do not hold its history.<br/>An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.<br/>To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.<br/>When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.<br/>**It's only used when the provider is `experimental_object_store`**. |
 | `metadata_store` | -- | -- | Metadata storage options. |
 | `metadata_store.file_size` | String | `64MB` | The size of the metadata store log file. |
 | `metadata_store.purge_threshold` | String | `256MB` | The threshold of the metadata store size to trigger a purge. |
@@ -614,6 +615,7 @@
 | `wal.max_unpersisted_bytes` | String | `64MB` | The size of the unpersisted backlog at which new appends stall until an upload completes, in `enqueued` mode.<br/>Nothing is dropped and nothing is rejected; the threshold does not bound what a crash during an outage can lose.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.max_unpersisted_age` | String | `8s` | The age of the oldest unpersisted entry at which new appends stall until an upload completes, in `enqueued` mode.<br/>**It's only used when the provider is `experimental_object_store` and `ack_mode` is `enqueued`**. |
 | `wal.on_corrupted_segment` | String | `skip` | What a read does with a segment that still does not decode after a second fetch, because its checksum does not match or its content disagrees with its footer entry.<br/>- `skip`: the segment is skipped and recorded as a WAL hole of its region, a metric is incremented and a warning is logged; the other regions of the object are unaffected (the default).<br/>- `fail`: the read fails, so the region does not open.<br/>**It's only used when the provider is `experimental_object_store`**. |
+| `wal.accept_missing_history` | Bool | `false` | Whether a region opens although the WAL objects under the prefix do not hold its history.<br/>An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.<br/>To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.<br/>When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.<br/>**It's only used when the provider is `experimental_object_store`**. |
 | `query` | -- | -- | The query engine options. |
 | `query.parallelism` | Integer | `0` | Parallelism of the query engine.<br/>Default to 0, which means the number of CPU cores. |
 | `query.memory_pool_size` | String | 0 | Memory pool size for query execution operators (aggregation, sorting, join).<br/>Supports absolute size (e.g., "2GB", "4GB") or percentage of system memory (e.g., "20%").<br/>Setting it to 0 disables the limit. Defaults to 0 (unlimited); this example sets it to `50%`.<br/>When this limit is reached, queries will fail with ResourceExhausted error.<br/>NOTE: This does NOT limit memory used by table scans. |

config/datanode.example.toml

@@ -258,6 +258,13 @@ overwrite_entry_start_id = false
 ## **It's only used when the provider is `experimental_object_store`**.
 #+ on_corrupted_segment = "skip"
 
+## Whether a region opens although the WAL objects under the prefix do not hold its history.
+## An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.
+## To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.
+## When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.
+## **It's only used when the provider is `experimental_object_store`**.
+#+ accept_missing_history = false
+
 # The Kafka SASL configuration.
 # **It's only used when the provider is `kafka`**.
 # Available SASL mechanisms:

config/standalone.example.toml

@@ -449,6 +449,13 @@ overwrite_entry_start_id = false
 ## **It's only used when the provider is `experimental_object_store`**.
 #+ on_corrupted_segment = "skip"
 
+## Whether a region opens although the WAL objects under the prefix do not hold its history.
+## An existing region fails to open when the prefix held no object when the WAL opened, or when the flushed entry id of the region names an object above every object of the prefix that holds entries. Both happen after `storage_provider`, or the bucket or root of the provider it names, changed without the WAL objects being copied, or after the objects were deleted.
+## To move the WAL, stop the instance, copy every object under the prefix to the new location and then change the configuration; the regions open with all their writes.
+## When the old WAL is lost, set this to `true` for one start: the regions open without their writes that were not flushed, which are lost, a warning names each region, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts them. Turn it off again once the instance has written to the WAL after that start; until then a later start refuses a region whose flushed entry id names an object above every object of the prefix that holds entries.
+## **It's only used when the provider is `experimental_object_store`**.
+#+ accept_missing_history = false
+
 # The Kafka SASL configuration.
 # **It's only used when the provider is `kafka`**.
 # Available SASL mechanisms:

docs/rfcs/2026-09-06-object-store-wal.md

@@ -50,7 +50,7 @@ The provider is added to the four parallel WAL enums so that every resolution pa
 
 Two choices are deliberate:
 
-- The persisted `WalOptions` carry the derived node prefix `<prefix>/datanodes/<node_id>/epochs/<generation>` described under *Object format*, not the configured root. On reopen the log store compares the persisted prefix with the one it runs under and fails fast on a mismatch, so an operator who changes the prefix or the node id cannot silently point a region at an empty WAL. Changing the prefix of an existing region requires an explicit migration, which is out of scope here.
+- The persisted `WalOptions` carry the derived node prefix `<prefix>/datanodes/<node_id>/epochs/<generation>` described under *Object format*, not the configured root. On reopen the log store compares the persisted prefix with the one it runs under and fails fast on a mismatch, so an operator who changes the prefix or the node id cannot silently point a region at an empty WAL. Changing the prefix of an existing region requires an explicit migration, which is out of scope here. The prefix does not name the object store that holds it, so changing `storage_provider`, or the bucket or root of the provider it names, keeps the prefix; the history check under *Log store* catches such a change when a region opens.
 - `Provider::ObjectStore` is classified as a remote WAL. Mito's remote-WAL branches (initial flushed entry id from `latest_entry_id`, `topic_latest_entry_id` after replay, region-scoped reads, no topic grouping on batch open) apply unchanged. This is a classification of replay semantics, not of physical location.
 
 A datanode that is configured with a metasrv client rejects the provider before any store is built: `DatanodeBuilder` returns an error stating that the object store WAL is standalone-only. Separately, `DatanodeWalConfig::ObjectStore` cannot be converted into `MetasrvWalConfig`, so a metasrv cannot be configured with it either. The standalone bootstrap matches the datanode WAL configuration before that conversion and constructs `WalProvider::ObjectStore` directly.
@@ -102,11 +102,11 @@ Positions start at one, so entry id zero is never assigned: Mito treats zero as
 
 `latest_entry_id(provider)` is region-scoped: the highest entry id durable under the prefix for the provider's region, or zero when the region has none. A freshly created region therefore starts at zero even if other regions under the prefix have entries. This differs from Kafka, whose high watermark is per topic; it is sufficient here because ids are per region, and Mito consumes the value only for the region that asked.
 
-Two rules keep every new id of a region above every id it already has, on a prefix that already holds objects and across the watermark Mito hands the store. On open, the next object sequence is the larger of the largest listed sequence plus one, orphans included, and, over every region, the object named by the region's largest existing entry id plus one, that is `(max_entry_id >> 20) + 1`, so that the catalog's strictly increasing ranges hold; sequence gaps are tolerated, and a sequence at or above `2^44` is rejected as exhausted. Objects written under an earlier contiguous scheme are not rewritten: they remain readable, their ids carry no object information, and the rule places them by their high bits just the same.
+Two rules keep every new id of a region above every id it already has, on a prefix that already holds objects and across the watermark Mito hands the store. On open, the next object sequence is the larger of the largest listed sequence plus one, orphans included, and, over every region, the object named by the region's largest existing entry id plus one, that is `(max_entry_id >> 20) + 1`, so that the catalog's strictly increasing ranges hold; sequence gaps are tolerated, and a sequence at or above `2^44` is rejected as exhausted. Ids have been object-sequence-major since the first version of the provider; a contiguous scheme existed only in the reference implementation before release, so no deployed prefix holds ids without object information, and the history check under *Log store* relies on that.
 
-The second rule is the sequence floor of `obsolete`. Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, and the actor applies the watermark and a floor derived from that entry id together: the next object sequence is raised to `(entry_id >> 20) + 1` unless it is there already, while zero, the watermark of a region without entries, names no object and needs no floor, so that a watermark assigned under another prefix, or one whose object this prefix no longer holds, is never passed by a new id. The floor moves only the next sequence, never the chain: the next batch still links to its predecessor, so a skipped sequence leaves no gap in the history recovery replays. While the open batch holds ids under the next sequence, the call is refused with a retryable error and records neither the watermark nor the floor, because those ids would fall below the floor. In the `enqueued` mode an id the store itself handed out needs no floor, even before it is durable: it is never handed out again while the store runs, so every later id of the region is greater. In the current single-prefix layout the floor never needs to move, since every watermark Mito hands the store names an object below the next sequence; the rule exists for the cross-prefix cases under *Alignment with cluster mode*. No additional persisted state is introduced.
+The second rule is the sequence floor of `obsolete`. Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, and the actor applies the watermark and a floor derived from that entry id together: the next object sequence is raised to `(entry_id >> 20) + 1` unless it is there already, while zero, the watermark of a region without entries, names no object and needs no floor, so that a watermark assigned under another prefix, or one whose object this prefix no longer holds, is never passed by a new id. The floor moves only the next sequence, never the chain: the next batch still links to its predecessor, so a skipped sequence leaves no gap in the history recovery replays. While the open batch holds ids under the next sequence, the call is refused with a retryable error and records neither the watermark nor the floor, because those ids would fall below the floor. In the `enqueued` mode an id the store itself handed out needs no floor, even before it is durable: it is never handed out again while the store runs, so every later id of the region is greater. In the current single-prefix layout the floor never needs to move for a prefix that holds the region's history, since every watermark Mito hands the store then names an object below the next sequence. A watermark above the newest object holding entries that the store found means the prefix does not hold that history: the history check under *Log store* refuses such an open before Mito calls `obsolete`, and the floor moves only when `accept_missing_history` accepted the loss. Otherwise the rule exists for the cross-prefix cases under *Alignment with cluster mode*. No additional persisted state is introduced.
 
-Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after a release in full, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after every release and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes the watermark of every region Mito opens again, and the boundary advances on the same rule. A region that is never opened again needs the release described below. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing.
+Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after a release in full, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after every release and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes the watermark of every region Mito opens again, and the boundary advances on the same rule. A region that is never opened again needs the release described below. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, or its own sequence when the recovered chain holds no segment, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing.
 
 Every present object with a sequence below the durable boundary is deletable, on the chain or not: the objects the catalog indexes, the objects recovery listed but did not index, the objects of an earlier epoch that the start object moved past, and the sequences of batches that failed in this run, whose objects may or may not exist. A collection starts after a successful `obsolete`, after a release and after an indexing that raised the durable boundary. Deletes run in the background, at most four at a time and in no particular order, and never hold up admission, sealing, uploads or acknowledgements; an indexed object is unindexed only after its delete succeeded, and the WAL holes recorded for it are dropped with it. Every deleted object is counted in `greptime_logstore_object_store_wal_deleted_objects_total`; a failed delete is logged, counted in `greptime_logstore_object_store_wal_failed_deletes_total` and retried at the next collection, and the object stays indexed meanwhile. An object deleted but not yet unindexed when the process crashes is simply absent at the next recovery, which lists the prefix. Nothing is collected once `stop` has begun or while the store is poisoned; a delete in flight when `stop` begins runs to completion before `stop` returns, and the actor settles every attempt it abandons when it exits.
 
@@ -160,13 +160,21 @@ A release is a waiter of the batch that carries its record in both modes: it ret
 
 **Reading.** `read(provider, entry_id)` returns the region's entries with id at least `entry_id`, in order, fetching only that region's segments with range reads and decoding nothing else, and hides entries at or below the region's obsolete watermark. `obsolete` records the watermark in memory, clamped to the durable entry id in the `enqueued` mode; the watermark is re-established after a restart because Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, so the store persists no watermark of a region Mito opens again. The release of a region, which `obsolete_all` and `release_region` make, is persisted as described under *Entry ids*. `obsolete_all` hides the whole region. `latest_entry_id` is not affected by the watermark. Every method taking a provider requires `Provider::ObjectStore` with the store's prefix and the caller's region.
 
-**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, and `stalled_appends_total` with `stalled_append_seconds` for an append held back because the sealed batches were full, in either mode, or by the `enqueued` backlog thresholds, timed until it is admitted or refused, a stop, a terminal error, a failing store or the teardown of the actor releasing it included. The gauge `failing` counts the stores whose backlog the object store refuses with an error that requires intervention, and `rejected_appends_total` the appends and releases such a store rejected; the attempts of a failing store are counted in `create_failures_total`. The delete counters of a collection and the release record counter are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB.
+**History check.** A region persists its prefix, not the object store that holds the prefix, so pointing `storage_provider`, or the bucket or root of the provider it names, at another location, or deleting the objects under the prefix, would let every region open against a prefix without its history, replay nothing and lose its acknowledged writes that were not flushed. Two invariants of a prefix expose such a location without any persisted state of their own. A prefix that a store opened is never empty afterwards, unless the store withdrew its start object as described below: the store writes its start object before it admits an append, and collection deletes only objects below the durable boundary, the boundary of the last indexed object, which is at or below that object's own sequence, since a larger one fails the header check. And a watermark a region persisted names an object at or below the newest object holding entries that any later recovery of the prefix indexes: the entry it names was durable and indexed when the watermark was persisted, as described under *Acknowledgement*; no durable boundary passes the newest object holding entries, since a batch carries at most the sequence after the last indexed object and a start object carries the boundary recovery found, so collection keeps that object and recovery indexes it on the chain of the tip. The segment of release records counts as entries: it never holds the reclaim boundary, so an object holding only release records can be the one that collection keeps while it deletes every older object. Objects without segments, start objects and the like, do not move it. `LogStore::check_region_history(provider, entry_id)` therefore reports the history of the provider's region as missing when recovery found no object under the prefix, or when `entry_id` names an object above the newest object holding entries that recovery indexed, unless the catalog holds the region's entries through `entry_id`, as it does for a region written since the store opened, or the region's history starts in the store: Mito calls `LogStore::start_region_history` when it creates a region, before it persists the region's manifest, and an open that accepted a missing history records the region the same way, in memory, so the region reopens on the same store without any entry and is counted once. Both methods default to accepting and to doing nothing, so Raft Engine and Kafka are unchanged. Mito calls it before it replays an existing region from its persisted state, on open, on catch-up and when a create finds the region already present, with the larger of the manifest's `flushed_entry_id` and `truncated_entry_id`; a region whose options skip the WAL keeps no history there and is not checked, nor is a region with the noop provider. A missing history fails the open, and such a create, which does not fall back to creating the region over its manifest, with an `InvalidArguments` error that names the region, the prefix, the configured storage provider with its scheme, bucket and root, and the two ways forward below; in standalone a region that fails to open stops startup. The check runs before Mito calls `obsolete` with the watermark, so the sequence floor never hides a missing history.
+
+A refused open must not make the next start pass. When the store found its prefix empty and refuses a region, it gives up the start object it wrote: the actor agrees only while nothing was ever handed an id, no batch, append, release or create waits, no failed release is kept for a later object, recovery listed no other object, the catalog holds only the start object and no region's history started in the store, and from then on the store records a terminal error and accepts nothing, since a later object would link to the given-up one. A release seals its own batch without handing an id, so once its object is indexed the catalog shows it, alongside the start object or alone after collection deleted the start object; the actor deletes the object like a collected one, so `stop` waits for that delete and a cancelled caller does not abandon it, and every refusal returns its error only once the delete settled. A region registers its history and the actor decides the withdrawal under one lock, so a region created at that moment either prevents the withdrawal or fails to register, and then fails its create before it leaves a manifest that a later start would refuse. Another instance may open the prefix while the delete is in flight, as when two processes of the same node overlap: its recovery finds a chain without any segment, so its start object reclaims every object below it, and the delete, which may land after that start object, leaves its chain complete. The terminal error is not counted in `poisoned_total`. In standalone, regions open before the instance serves, so at that point nothing else can have written to the prefix, and the next start, an operator retry or an orchestrator restart, finds the prefix empty again and refuses the same regions. A refused start on a prefix that held objects writes a start object, which holds no entry and so does not move the newest object holding entries: later starts refuse the same watermarks.
+
+To move the WAL to another location, stop the instance, copy every object under the prefix to the same keys in the new location, and then change the configuration: recovery finds the same objects and every region opens with all of its acknowledged writes. When the old objects are lost, `accept_missing_history = true` turns the failure into an accepted loss for that start: the region opens without its writes that were not flushed, a warning names the region, the prefix, the location and the watermark above which nothing was replayed, and `greptime_logstore_object_store_wal_missing_history_accepted_total` counts the region. The option is meant to be turned off again once the instance has written to the WAL after that start: until a write creates an object holding entries at or above the sequence the accepted watermark names, which the floor raised by the accepted open makes the next write do, a later start reports such a region again.
+
+The check does not catch every change of location. A location whose prefix already holds objects passes a region that never flushed, and a region whose watermark names an object at or below the newest object holding entries there, whatever history those objects hold. A refused start on an empty prefix leaves its start object behind when the process ends before the delete completes or when the delete fails, which is logged; the next start then finds a prefix with only start objects and passes a region that never flushed, while it still refuses every region whose watermark names an object. The same holds when the store created a region before the refusal. When it admitted an append or a release before the refusal, as it can when regions open while the instance already serves, the store withdraws nothing and the prefix keeps an object holding entries, which may already have collected the start object; the next start then also passes every region whose watermark names an object at or below that one.
+
+**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, `missing_history_accepted_total` for a region that `accept_missing_history` let open without its history, and `stalled_appends_total` with `stalled_append_seconds` for an append held back because the sealed batches were full, in either mode, or by the `enqueued` backlog thresholds, timed until it is admitted or refused, a stop, a terminal error, a failing store or the teardown of the actor releasing it included. The gauge `failing` counts the stores whose backlog the object store refuses with an error that requires intervention, and `rejected_appends_total` the appends and releases such a store rejected; the attempts of a failing store are counted in `create_failures_total`. The delete counters of a collection and the release record counter are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB.
 
 A `testing` cargo feature exposes hooks to wait for admitted appends, to seal the open batch, to hold and release creates and to make creates fail, so engine tests can flush deterministically and observe entries that are admitted but not durable.
 
 ## Mito integration
 
-Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provider(region_id, prefix)`; resolving object store options against a Raft Engine or Kafka log store reports the existing incompatible-provider error. Reads go through the region-scoped reader used for Kafka, so a namespace shared by many regions never leaks entries across regions even if a log store returned them. Object store regions are opened individually rather than grouped by topic. After replay, `topic_latest_entry_id` follows the existing remote-WAL rule: it is taken from the store's latest entry id only when the region's memtables are empty, otherwise it stays at the replay start, because that value feeds pruning and must never advance past unflushed data. Before a flush, a full truncate or a discard of unflushed data publishes a manifest watermark, Mito waits on the durability barrier described under *Acknowledgement*; with the `durable` mode and with every other provider the wait returns at once. When Mito drops a region it calls `release_region` after `obsolete`, as described under *Entry ids*; the default of that method does nothing, so the other providers handle a dropped region as before. Neither a release nor an `obsolete` of an object store region that fails, the latter for instance on a store that the release of another region poisoned, fails the drop, since nothing sends the drop of a removed region again and a caller that drops several regions, such as the metric engine with its data and metadata regions, would leave the others undone: the region's runtime state is cleaned up and its files are scheduled for deletion as for any drop, and the failure is logged; an `obsolete` that fails for another provider fails the drop as before. A store that is neither stopped nor poisoned keeps the record and writes it with its next object; a stopped or poisoned store keeps none, and the region stays unreleased after a restart. The store carries the record in a later object, as described under *Entry ids*; a crash before any later create succeeds loses it, and the dropped region's segments then hold the reclaim boundary after the restart.
+Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provider(region_id, prefix)`; resolving object store options against a Raft Engine or Kafka log store reports the existing incompatible-provider error. Reads go through the region-scoped reader used for Kafka, so a namespace shared by many regions never leaks entries across regions even if a log store returned them. Object store regions are opened individually rather than grouped by topic. After replay, `topic_latest_entry_id` follows the existing remote-WAL rule: it is taken from the store's latest entry id only when the region's memtables are empty, otherwise it stays at the replay start, because that value feeds pruning and must never advance past unflushed data. Before a flush, a full truncate or a discard of unflushed data publishes a manifest watermark, Mito waits on the durability barrier described under *Acknowledgement*; with the `durable` mode and with every other provider the wait returns at once. When Mito drops a region it calls `release_region` after `obsolete`, as described under *Entry ids*; the default of that method does nothing, so the other providers handle a dropped region as before. Neither a release nor an `obsolete` of an object store region that fails, the latter for instance on a store that the release of another region poisoned, fails the drop, since nothing sends the drop of a removed region again and a caller that drops several regions, such as the metric engine with its data and metadata regions, would leave the others undone: the region's runtime state is cleaned up and its files are scheduled for deletion as for any drop, and the failure is logged; an `obsolete` that fails for another provider fails the drop as before. A store that is neither stopped nor poisoned keeps the record and writes it with its next object; a stopped or poisoned store keeps none, and the region stays unreleased after a restart. The store carries the record in a later object, as described under *Entry ids*; a crash before any later create succeeds loses it, and the dropped region's segments then hold the reclaim boundary after the restart. Before Mito replays an existing region on open, on catch-up or for a create that finds the region present, it runs the history check described under *Log store*.
 
 ## Datanode wiring and lifecycle
 
@@ -211,6 +219,10 @@ max_unpersisted_age = "8s"
 # regions of the object are unaffected (the default). "fail": the read
 # fails, so the region does not open.
 on_corrupted_segment = "skip"
+# Whether a region opens although the prefix does not hold its WAL
+# history, losing its writes that were not flushed; see the history
+# check under *Log store*. Meant for one start after the old WAL is lost.
+accept_missing_history = false
 ```
 
 `config/config.md` is generated from the example files.
@@ -222,8 +234,9 @@ on_corrupted_segment = "skip"
 - The configuration tag carries `experimental_`; the persisted tag does not, so promoting the provider to stable is a configuration rename with an alias, not a metadata migration.
 - `LogStore::wait_durable` has a default implementation that returns at once, so Raft Engine, Kafka and the noop provider keep their own acknowledgement and persistence semantics; the durability barrier binds only the object store provider.
 - `LogStore::release_region` has a default implementation that does nothing, so Raft Engine, Kafka and the noop provider keep their handling of a dropped region.
-- The object-sequence-major id scheme needs no migration of existing objects. A prefix written under the contiguous scheme keeps its objects, and the sequence allocation rule under *Entry ids* keeps new ids above the old ones. Garbage collection reasons from footers and from the reclaim boundary, not from the id encoding, so it applies to objects of either scheme.
+- The object-sequence-major id scheme is the only one the provider has written, so no prefix needs a migration; the contiguous scheme of the pre-release reference implementation has no deployed prefixes, and the history check relies on every id naming its object.
 - `on_corrupted_segment` is a new configuration option with a default, so existing configurations decode unchanged; it changes what a read does with a damaged segment, not what is written.
+- `accept_missing_history` is a new configuration option with a default of `false`, so existing configurations decode unchanged. `LogStore::check_region_history` has a default implementation that accepts and `LogStore::start_region_history` one that does nothing, so only the object store provider checks a region's history, and the check reads only the recovered catalog: no object format or region metadata changes. A region that previously opened against a prefix without its history, replaying nothing, now fails to open until its objects are restored or the loss is accepted.
 
 ## Evidence
 
@@ -240,7 +253,7 @@ The reference implementation was validated on the following evidence; the tests
 | A manifest watermark never names an entry that is not durable | Durability barrier tests: a flush held until its entry is durable; a crash in `enqueued` mode after an acknowledgement and a flush attempt but before the object exists, followed by a restart, new writes and a second crash; a drop cancelling a flush that waits; a truncate that waits; a flush after a backlog was lost. | Every durable entry replays; the flush after the lost backlog publishes no frontier; `obsolete` never passes the durable id. |
 | A region replays only its own entries | The two-region recovery test above. | Per-region row-level scan equality with the other region unflushed. |
 | Recovery never silently loses data | Unit coverage of every corruption class at recovery and on read; footer-only recovery compared with a full decode; a footer longer than the tail window. | Structural damage fails construction of the store; a corrupted segment is skipped, recorded as a hole of its region and counted, or fails the read under `fail`; the footer-only catalog equals the full decode. |
-| A region cannot be silently pointed at another WAL | Reopen on a mismatched prefix; reopen on an empty prefix. | The mismatch is rejected with both prefixes named and the region is not opened; the empty prefix opens cleanly with the expected row counts. |
+| A region cannot be silently pointed at another WAL | Reopen on a mismatched prefix; open and catch-up against a location whose prefix is empty, and open against one whose prefix holds objects only below a flushed or a truncated watermark; a copy of every object on a new location; a restart after collection left only the newest object; the option accepting the loss; store tests of the same cases. | The mismatch is rejected with both prefixes named; a missing history fails the open and the catch-up with the region, the prefix and the location named, while a region that skips the WAL, a region whose history the location holds and a new region still open; the copy and the restart open with every row; the option opens the region without its unflushed rows and counts it once. |
 | Garbage collection deletes everything below the persisted reclaim boundary, and recovery stays correct while objects disappear | Unit tests of the boundary computation (a region without a watermark holds it at its first object, a watermark inside an object keeps the object, `obsolete_all` releases it, last indexed plus one when everything is flushed), of stamping and durability (headers carry the boundary current at sealing, a failed create's boundary permits nothing, the start object carries the recovered boundary), of deletion (chain objects, listed orphans and failed batches below the durable boundary, nothing at or above it, a failed delete retried, `stop` waiting for deletes in flight, holes dropped), of recovery (two restarts with a collection between them, a late object of an earlier epoch below the boundary collected, a missing predecessor above the boundary still failing) and of a read that listed an object before it was collected; the Mito flush-then-collect test in both acknowledgement modes; the standalone integration test's assertion after a flush. | Objects below the durable boundary are gone after a flush, the start object included; a restart replays everything above the boundary; exactly one WAL object remains after a flush and one more insert in the integration test. |
 | A dropped region holds no object after a restart that never opens it | Unit tests of the release record an object carries and the boundary it allows; of recovery seeding the watermark from the release record before any boundary is computed, with collection proceeding once the remaining region is flushed; of a failed release carried by a timer tick on an idle prefix and by a `durable` stop, which reports the record lost when its create fails too; of the release of a region without segments, and of a region id used again after its release; of a release record that outlives a restart together with the segment it covers and goes once both are below the boundary, deleted out of order; the fault-matrix cases of a release, which return only once its object is durable and indexed, fail with its create, and fail at once while the store is failing; the Mito test that drops one of two regions, restarts gracefully, abruptly, or after a release that failed, with nothing written in between, then flushes the other region and writes once more; the metric engine test of a physical region whose data and metadata regions wrote to the WAL, dropped while the release of the data region fails, while both releases fail, and while the release of the data region meets a conflicting object that poisons the store, which still drops both regions and forgets the physical region, and, in the first two cases, after which a store opened later keeps no segment of either; the frozen object with release records. | Recovery seeds the release, and after the flush and the next write every object written before the restart is deleted. |
 | Every row of the *Failure matrix* holds, and every metric moves as documented | A table-driven fault matrix, `test_store_fault_matrix`, that drives every failure the object store access can produce through the store and asserts, per case, which waiters fail and with what error, whether the store is poisoned, whether the sequence is consumed, the indexed gauges at the failure itself, the exact movement of every metric, and what a read of the store sees afterwards; every timer is observed twice around two forced waits whose intervals cannot overlap. | 42 cases: 24 for the eleven rows of the matrix, five of them for the create of a release record, and 18 for the failures listed under *Failures driven through the store*. The test fails if a row of the matrix is left uncovered. |
@@ -345,7 +358,7 @@ Distributed mode is out of scope, but several decisions were taken so that the c
 
 Sealing may only exclude objects whose writes were never acknowledged, which takes two fences. Admission is fenced by the lease mechanism Mito already has: when a region's lease expires the datanode demotes it to a follower, which refuses writes, and the metasrv waits for that deadline before it upgrades the candidate. Completion is fenced by the store: every conditional create runs under an upload deadline, a create that has not completed by its deadline is reported failed to its waiters and is never acknowledged afterwards even if the object turns out to exist, and when a region's lease expires the store fails every waiter of that region that is not yet acknowledged, so that a create still in flight at that moment cannot be acknowledged later. The metasrv seals the segment only after the lease deadline plus the upload deadline have both passed, so an object that appears after sealing belongs to a create the old writer had already reported as failed. Under the durable acknowledgement mode this means a sealed set never excludes an acknowledged write; under the `enqueued` mode the entries in the old writer's unpersisted backlog at that moment are lost, which is that mode's stated recovery point objective. The late-completing create is part of the acceptance criteria for takeover, stated in terms of the sealing instant: an object that was absent at sealing, whether it fills a hole later or lies above the cutover, is excluded on every open of the region, and so is a visible object outside the selected chain; an object on the sealed chain replays on every open even if its write was never acknowledged, which is the same outcome as a crash between an object's creation and its acknowledgement; and in the durable mode every caller whose create had not completed when the lease expired must have been failed, so no acknowledged write is among the excluded objects. The generation in the key is an additional guarantee that such a late object lands in a directory the new writer never reads beyond the sealed set. Object-sequence-major entry ids are what make this cheap: `entry_id >> 20` locates the object under the right prefix without the source node's in-memory catalog.
 
-**Entry id floor across prefixes.** A region that arrives under a new prefix, by migration or by failover, brings a watermark and replayed ids that were assigned under another prefix's sequence. Before the target assigns the region its first new id, the target store raises its next object sequence above `max_id >> 20`, where `max_id` is the largest of the region's inherited watermark and every id it replayed, so that every new id of that region is greater than every id it already has; sequence gaps are tolerated and the sequence space is wide enough for this. The rule is applied on every open from the region's persisted state, in the same place Mito already hands the store the manifest watermark, so it needs no persisted state of its own and survives a crash before the target's first flush: the restart re-derives the floor and the region's acknowledged-but-unflushed entries under the target prefix are above the inherited watermark and replay.
+**Entry id floor across prefixes.** A region that arrives under a new prefix, by migration or by failover, brings a watermark and replayed ids that were assigned under another prefix's sequence. Before the target assigns the region its first new id, the target store raises its next object sequence above `max_id >> 20`, where `max_id` is the largest of the region's inherited watermark and every id it replayed, so that every new id of that region is greater than every id it already has; sequence gaps are tolerated and the sequence space is wide enough for this. The rule is applied on every open from the region's persisted state, in the same place Mito already hands the store the manifest watermark, so it needs no persisted state of its own and survives a crash before the target's first flush: the restart re-derives the floor and the region's acknowledged-but-unflushed entries under the target prefix are above the inherited watermark and replay. The history check under *Log store* stays compatible with this: it judges a watermark against the prefix a region persists, and a region taken over from another prefix carries that prefix as an earlier segment of its `WalOptions`, so its inherited watermark is judged against the segment that assigned it rather than against the target prefix, and the floor, not the check, applies to it on the target.
 
 **Acknowledgement modes.** Both are implemented. Cluster mode is expected to run `enqueued` for latency-sensitive workloads; the object layout, the flush durability barrier and the recovery path are identical in both modes.
 

src/common/wal/src/config.rs

@@ -298,6 +298,7 @@ mod tests {
         assert_eq!(config.max_unpersisted_bytes, ReadableSize::mb(64));
         assert_eq!(config.max_unpersisted_age, Duration::from_secs(8));
         assert_eq!(config.on_corrupted_segment, CorruptedSegmentAction::Skip);
+        assert!(!config.accept_missing_history);
         assert!(matches!(
             MetasrvWalConfig::try_from(datanode_wal_config).unwrap_err(),
             Error::UnsupportedWalProvider { .. }
@@ -313,6 +314,7 @@ mod tests {
             max_unpersisted_bytes = "32MB"
             max_unpersisted_age = "4s"
             on_corrupted_segment = "fail"
+            accept_missing_history = true
         "#;
         let datanode_wal_config: DatanodeWalConfig = toml::from_str(toml_str).unwrap();
         let expected = ObjectStoreWalConfig {
@@ -324,6 +326,7 @@ mod tests {
             max_unpersisted_bytes: ReadableSize::mb(32),
             max_unpersisted_age: Duration::from_secs(4),
             on_corrupted_segment: CorruptedSegmentAction::Fail,
+            accept_missing_history: true,
         };
         assert_eq!(
             datanode_wal_config,
@@ -334,6 +337,7 @@ mod tests {
         assert_eq!(table["ack_mode"].as_str(), Some("enqueued"));
         assert_eq!(table["max_unpersisted_bytes"].as_str(), Some("32MiB"));
         assert_eq!(table["max_unpersisted_age"].as_str(), Some("4s"));
+        assert_eq!(table["accept_missing_history"].as_bool(), Some(true));
         assert_eq!(
             datanode_wal_config,
             toml::from_str::<DatanodeWalConfig>(&serialized).unwrap()

src/common/wal/src/config/object_store.rs

@@ -81,6 +81,11 @@ pub struct ObjectStoreWalConfig {
     /// second fetch, because its checksum does not match or its content
     /// disagrees with its footer entry, defaults to `skip`.
     pub on_corrupted_segment: CorruptedSegmentAction,
+    /// Whether a region whose WAL history the configured location does not
+    /// hold opens anyway, with its writes that were not flushed lost, instead
+    /// of failing to open, defaults to false. Meant for one start after the
+    /// old WAL location is lost.
+    pub accept_missing_history: bool,
 }
 
 impl Default for ObjectStoreWalConfig {
@@ -94,6 +99,7 @@ impl Default for ObjectStoreWalConfig {
             max_unpersisted_bytes: ReadableSize::mb(64),
             max_unpersisted_age: Duration::from_secs(8),
             on_corrupted_segment: CorruptedSegmentAction::Skip,
+            accept_missing_history: false,
         }
     }
 }

src/log-store/src/error.rs

@@ -424,6 +424,37 @@ pub enum Error {
         location: Location,
     },
 
+    #[snafu(display(
+        "Object store WAL prefix {} on {} does not hold the history of region {}: {}. \
+         If the WAL was moved, stop the instance, copy every object under the prefix from the \
+         old location to this one and start again. If the old WAL is lost, start once with \
+         `[wal].accept_missing_history = true`, which opens the region without its writes that \
+         were not flushed, and turn the option off again once the instance has written to the WAL",
+        prefix,
+        storage,
+        region_id,
+        reason
+    ))]
+    MissingWalHistory {
+        region_id: RegionId,
+        prefix: String,
+        /// The storage the prefix is read from, as configured.
+        storage: String,
+        reason: String,
+        #[snafu(implicit)]
+        location: Location,
+    },
+
+    #[snafu(display(
+        "Object store WAL gave up its start object {} after refusing a region without its history, and accepts nothing more until it opens again",
+        path
+    ))]
+    WalStartWithdrawn {
+        path: String,
+        #[snafu(implicit)]
+        location: Location,
+    },
+
     #[snafu(display("Object store WAL log store is stopped"))]
     ObjectStoreWalStopped {
         #[snafu(implicit)]
@@ -478,6 +509,7 @@ impl ErrorExt for Error {
             | OverrideCompactedEntry { .. }
             | InvalidWalObjectStore { .. }
             | MismatchedWalPrefix { .. }
+            | MissingWalHistory { .. }
             | MismatchedWalRegion { .. }
             | IncompleteWalEntry { .. }
             | InvalidWalEntryRange { .. } => StatusCode::InvalidArguments,
@@ -494,6 +526,7 @@ impl ErrorExt for Error {
             | WaitProduceResultReceiver { .. }
             | MetaLengthExceededLimit { .. }
             | ObjectStoreWalStopped { .. }
+            | WalStartWithdrawn { .. }
             | WalSegmentFetch { .. } => StatusCode::Internal,
 
             CorruptedWalObject { .. }

src/log-store/src/metrics.rs

@@ -97,6 +97,17 @@ lazy_static! {
         "object store logstore skipped corrupted segments total",
     )
     .unwrap();
+
+    /// Counter of regions the object store logstore let open although its
+    /// prefix does not hold their WAL history, because
+    /// `accept_missing_history` is set; their writes that were not flushed
+    /// are lost.
+    pub static ref METRIC_OBJECT_STORE_WAL_MISSING_HISTORY_ACCEPTED_TOTAL: IntCounter = register_int_counter!(
+        "greptime_logstore_object_store_wal_missing_history_accepted_total",
+        "object store logstore regions opened without their WAL history total",
+    )
+    .unwrap();
+
     /// Counter of objects the object store logstore deleted because no
     /// region needs them any more.
     pub static ref METRIC_OBJECT_STORE_WAL_DELETED_OBJECTS_TOTAL: IntCounter = register_int_counter!(

src/log-store/src/object_store_wal/io.rs

@@ -181,6 +181,18 @@ impl ObjectStoreIo {
         )
     }
 
+    /// Describes the object store the objects live in by its scheme, its
+    /// bucket when it has one and its root, none of which is a credential.
+    pub(crate) fn storage(&self) -> String {
+        let info = self.store.info();
+        let name = info.name();
+        if name.is_empty() {
+            format!("{}, root {:?}", info.scheme(), info.root())
+        } else {
+            format!("{}, bucket {name:?}, root {:?}", info.scheme(), info.root())
+        }
+    }
+
     fn parse_object_seq(&self, path: &str) -> Option<u64> {
         let value = path
             .strip_prefix(&self.object_prefix)?

src/mito2/src/engine/object_store_wal_recovery_test.rs

@@ -35,7 +35,9 @@ use common_wal::config::object_store::{
 };
 use common_wal::options::{ObjectStoreWalOptions, WAL_OPTIONS_KEY, WalOptions};
 use log_store::ObjectStoreLogStore;
-use log_store::metrics::METRIC_OBJECT_STORE_WAL_FAILING;
+use log_store::metrics::{
+    METRIC_OBJECT_STORE_WAL_FAILING, METRIC_OBJECT_STORE_WAL_MISSING_HISTORY_ACCEPTED_TOTAL,
+};
 use log_store::object_store_wal::{WalHole, entry_id};
 use object_store::ObjectStore;
 use object_store::layers::mock::{self, MockLayerBuilder, oio};
@@ -47,8 +49,8 @@ use store_api::logstore::provider::Provider;
 use store_api::mito_engine_options::SKIP_WAL_KEY;
 use store_api::region_engine::{RegionEngine, RegionRole};
 use store_api::region_request::{
-    PathType, RegionDropRequest, RegionFlushRequest, RegionOpenRequest, RegionPutRequest,
-    RegionRequest, RegionTruncateRequest,
+    PathType, RegionCatchupRequest, RegionCloseRequest, RegionDropRequest, RegionFlushRequest,
+    RegionOpenRequest, RegionPutRequest, RegionRequest, RegionTruncateRequest,
 };
 use store_api::storage::{RegionId, ScanRequest};
 
@@ -652,8 +654,72 @@ async fn test_open_region_rejects_mismatched_wal_prefix() {
     assert!(!engine.is_region_exists(REGION_A));
 }
 
+/// Opens a store under [`PREFIX`] like [`open_store`] that lets a region
+/// open although the prefix does not hold its WAL history.
+async fn open_store_accepting_missing_history(
+    object_store: &ObjectStore,
+) -> Arc<ObjectStoreLogStore> {
+    let config = ObjectStoreWalConfig {
+        accept_missing_history: true,
+        ..store_config(AckMode::Durable)
+    };
+    ObjectStoreLogStore::try_new(object_store.clone(), &config, 0, STANDALONE_GENERATION)
+        .await
+        .unwrap()
+}
+
+/// Copies every WAL object under [`PREFIX`] from `from` to the same key in
+/// `to`, which is how the WAL moves to another location.
+async fn copy_wal_objects(from: &ObjectStore, to: &ObjectStore) {
+    for object_seq in wal_object_seqs(from).await {
+        let path = format!("{PREFIX}/objects/{object_seq:020}.wal");
+        let bytes = from.read(&path).await.unwrap().to_vec();
+        to.write(&path, bytes).await.unwrap();
+    }
+}
+
+/// The request that opens a region under [`PREFIX`] and replays its WAL.
+fn open_request(table_dir: &str) -> RegionOpenRequest {
+    RegionOpenRequest {
+        engine: String::new(),
+        table_dir: table_dir.to_string(),
+        options: wal_options(),
+        skip_wal_replay: false,
+        path_type: PathType::Bare,
+        checkpoint: None,
+        requirements: Default::default(),
+    }
+}
+
+async fn close_region(engine: &MitoEngine, region_id: RegionId) {
+    engine
+        .handle_request(
+            region_id,
+            RegionRequest::Close(RegionCloseRequest::default()),
+        )
+        .await
+        .unwrap();
+}
+
+/// Asserts that an open of `region_id` failed because the WAL does not hold
+/// its history for `reason`, with the ways forward named.
+fn assert_missing_history(err: &BoxedError, region_id: RegionId, reason: &str) {
+    assert_eq!(StatusCode::InvalidArguments, err.status_code());
+    let message = err.output_msg();
+    for expected in [
+        &region_id.to_string(),
+        PREFIX,
+        "the default storage (memory",
+        reason,
+        "copy every object under the prefix",
+        "`[wal].accept_missing_history = true`",
+    ] {
+        assert!(message.contains(expected), "unexpected error: {message}");
+    }
+}
+
 #[tokio::test]
-async fn test_reopen_on_empty_prefix_without_durable_entries() {
+async fn test_open_on_an_empty_prefix_requires_accepting_the_missing_history() {
     let mut env = TestEnv::with_prefix("object-store-wal-empty-prefix").await;
     let store = open_store(&memory_store(), AckMode::Durable).await;
     let engine = new_engine(&mut env, store.clone()).await;
@@ -668,16 +734,52 @@ async fn test_reopen_on_empty_prefix_without_durable_entries() {
     drop(engine);
     drop(store);
 
-    // A fresh object store holds no object under the prefix.
+    // A fresh object store holds no object under the prefix. Region B keeps
+    // no history in the WAL and opens from its SST alone. Nothing tells
+    // region A, which never wrote, from a region whose unflushed entries
+    // were left behind, so it does not open, and the refused open deletes
+    // the start object it wrote: a second start, as an orchestrator that
+    // restarts a failed process makes, finds the prefix empty again and
+    // refuses region A the same way.
+    let new_location = memory_store();
+    for _ in 0..2 {
+        let store = open_store(&new_location, AckMode::Durable).await;
+        let engine = new_engine(&mut env, store.clone()).await;
+        open_region(&engine, REGION_B, &table_dir_b, &[(SKIP_WAL_KEY, "true")])
+            .await
+            .unwrap();
+        assert_eq!(rows_b, scan_rows(&engine, REGION_B).await);
+        let err = open_region(&engine, REGION_A, &table_dir_a, &[])
+            .await
+            .unwrap_err();
+        assert_missing_history(
+            &err,
+            REGION_A,
+            "the prefix held no object when the WAL opened",
+        );
+        assert!(!engine.is_region_exists(REGION_A));
+        assert!(wal_object_seqs(&new_location).await.is_empty());
+        engine.stop().await.unwrap();
+        drop(engine);
+        drop(store);
+    }
+
+    // Accepting the missing history opens region A without it, counted
+    // once; region B is not checked.
     let object_store = memory_store();
-    let store = open_store(&object_store, AckMode::Durable).await;
+    let store = open_store_accepting_missing_history(&object_store).await;
     let engine = new_engine(&mut env, store.clone()).await;
+    let accepted = METRIC_OBJECT_STORE_WAL_MISSING_HISTORY_ACCEPTED_TOTAL.get();
     open_region(&engine, REGION_A, &table_dir_a, &[])
         .await
         .unwrap();
     open_region(&engine, REGION_B, &table_dir_b, &[(SKIP_WAL_KEY, "true")])
         .await
         .unwrap();
+    assert_eq!(
+        accepted + 1,
+        METRIC_OBJECT_STORE_WAL_MISSING_HISTORY_ACCEPTED_TOTAL.get()
+    );
 
     assert_eq!(0, latest(&store, REGION_A));
     assert_eq!(0, latest(&store, REGION_B));
@@ -711,6 +813,301 @@ async fn test_reopen_on_empty_prefix_without_durable_entries() {
     assert_eq!(2, engine.get_region_statistic(REGION_A).unwrap().num_rows);
 }
 
+#[tokio::test]
+async fn test_create_after_a_withdrawal_leaves_no_manifest() {
+    let mut env = TestEnv::with_prefix("object-store-wal-create-after-withdrawal").await;
+    let store = open_store(&memory_store(), AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let (table_dir_a, _) = create_region(&engine, REGION_A, &[]).await;
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(store);
+
+    // On a new location the refused open of region A gives up the start
+    // object, and a create of region B after it fails before it writes a
+    // manifest.
+    let new_location = memory_store();
+    let store = open_store(&new_location, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    open_region(&engine, REGION_A, &table_dir_a, &[])
+        .await
+        .unwrap_err();
+    let request = CreateRequestBuilder::new()
+        .insert_option(WAL_OPTIONS_KEY, &wal_options()[WAL_OPTIONS_KEY])
+        .build();
+    let err = engine
+        .handle_request(REGION_B, RegionRequest::Create(request.clone()))
+        .await
+        .unwrap_err();
+    assert!(
+        format!("{err:?}").contains("gave up its start object"),
+        "unexpected error: {err:?}"
+    );
+    assert!(!engine.is_region_exists(REGION_B));
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(store);
+
+    // The next start creates region B, which a manifest left behind would
+    // have turned into an existing region without its history.
+    let store = open_store(&new_location, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    engine
+        .handle_request(REGION_B, RegionRequest::Create(request))
+        .await
+        .unwrap();
+    assert!(engine.is_region_exists(REGION_B));
+    engine.stop().await.unwrap();
+}
+
+#[tokio::test]
+async fn test_reopen_a_region_created_on_an_empty_prefix_without_writes() {
+    let mut env = TestEnv::with_prefix("object-store-wal-created-region").await;
+    let store = open_store(&memory_store(), AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let (table_dir, _) = create_region(&engine, REGION_A, &[]).await;
+
+    // The region's history starts in this store, so it opens again on it,
+    // directly and through a catch-up, although it holds no entry.
+    close_region(&engine, REGION_A).await;
+    open_region(&engine, REGION_A, &table_dir, &[])
+        .await
+        .unwrap();
+    close_region(&engine, REGION_A).await;
+    engine
+        .handle_request(
+            REGION_A,
+            RegionRequest::Open(RegionOpenRequest {
+                skip_wal_replay: true,
+                ..open_request(&table_dir)
+            }),
+        )
+        .await
+        .unwrap();
+    engine
+        .handle_request(
+            REGION_A,
+            RegionRequest::Catchup(RegionCatchupRequest {
+                set_writable: true,
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap();
+    assert!(region(&engine, REGION_A).is_writable());
+    assert_eq!(
+        EMPTY_RECOVERY_STATE,
+        region_recovery_state(&engine, REGION_A).await
+    );
+}
+
+#[tokio::test]
+async fn test_moving_the_wal_keeps_the_history_only_with_its_objects() {
+    let mut env = TestEnv::with_prefix("object-store-wal-move").await;
+    let old_location = memory_store();
+    let store = open_store(&old_location, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let (table_dir, schema) = create_region(&engine, REGION_A, &[]).await;
+    // Object 1 holds the region's rows, which are never flushed.
+    let mut writer = SealedWriter::new(&store);
+    writer
+        .put_and_seal(&engine, vec![(REGION_A, rows(&schema, 0, 10))])
+        .await;
+    let rows_before = scan_rows(&engine, REGION_A).await;
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(writer);
+    drop(store);
+
+    // On a location without the objects the region opens neither directly
+    // nor through a catch-up after an open that skipped the replay.
+    let store = open_store(&memory_store(), AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let reason = "the prefix held no object when the WAL opened";
+    let err = open_region(&engine, REGION_A, &table_dir, &[])
+        .await
+        .unwrap_err();
+    assert_missing_history(&err, REGION_A, reason);
+    engine
+        .handle_request(
+            REGION_A,
+            RegionRequest::Open(RegionOpenRequest {
+                skip_wal_replay: true,
+                ..open_request(&table_dir)
+            }),
+        )
+        .await
+        .unwrap();
+    let err = engine
+        .handle_request(
+            REGION_A,
+            RegionRequest::Catchup(RegionCatchupRequest {
+                set_writable: true,
+                ..Default::default()
+            }),
+        )
+        .await
+        .unwrap_err();
+    assert_missing_history(&err, REGION_A, reason);
+    assert!(!region(&engine, REGION_A).is_writable());
+    // A repeated create of the existing region fails the same way rather
+    // than creating the region again over its manifest.
+    close_region(&engine, REGION_A).await;
+    let request = CreateRequestBuilder::new()
+        .insert_option(WAL_OPTIONS_KEY, &wal_options()[WAL_OPTIONS_KEY])
+        .build();
+    assert_eq!(table_dir, request.table_dir);
+    let err = engine
+        .handle_request(REGION_A, RegionRequest::Create(request))
+        .await
+        .unwrap_err();
+    assert_missing_history(&err, REGION_A, reason);
+    assert!(!engine.is_region_exists(REGION_A));
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(store);
+
+    // Moving the WAL means copying every object: the region replays all of
+    // its rows on the new location.
+    let new_location = memory_store();
+    copy_wal_objects(&old_location, &new_location).await;
+    let store = open_store(&new_location, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    open_region(&engine, REGION_A, &table_dir, &[])
+        .await
+        .unwrap();
+    assert_eq!(rows_before, scan_rows(&engine, REGION_A).await);
+    assert_eq!(10, engine.get_region_statistic(REGION_A).unwrap().num_rows);
+}
+
+#[tokio::test]
+async fn test_open_region_fails_when_its_watermark_is_above_the_wal_objects() {
+    let mut env = TestEnv::with_prefix("object-store-wal-watermark-above").await;
+    let store = open_store(&memory_store(), AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let (table_dir_a, schema_a) = create_region(&engine, REGION_A, &[]).await;
+    let (table_dir_b, schema_b) = create_region(&engine, REGION_B, &[]).await;
+    // Region A flushes its entry of object 2, region B is truncated after its
+    // entry of object 3.
+    let mut writer = SealedWriter::new(&store);
+    writer
+        .put_and_seal(&engine, vec![(REGION_A, rows(&schema_a, 0, 2))])
+        .await;
+    writer
+        .put_and_seal(&engine, vec![(REGION_A, rows(&schema_a, 2, 4))])
+        .await;
+    flush_region(&engine, REGION_A, None).await;
+    writer
+        .put_and_seal(&engine, vec![(REGION_B, rows(&schema_b, 0, 2))])
+        .await;
+    engine
+        .handle_request(
+            REGION_B,
+            RegionRequest::Truncate(RegionTruncateRequest::All),
+        )
+        .await
+        .unwrap();
+    let manifest_a = region(&engine, REGION_A).manifest_ctx.manifest().await;
+    assert_eq!(entry_id(2, 1), manifest_a.flushed_entry_id);
+    let manifest_b = region(&engine, REGION_B).manifest_ctx.manifest().await;
+    assert_eq!(Some(entry_id(3, 1)), manifest_b.truncated_entry_id);
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(writer);
+    drop(store);
+
+    // The other location holds the start object 0 and object 1 of region C.
+    let other_location = memory_store();
+    let store = open_store(&other_location, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let region_c = RegionId::new(1, 3);
+    let (table_dir_c, schema_c) = create_region(&engine, region_c, &[]).await;
+    let mut writer = SealedWriter::new(&store);
+    writer
+        .put_and_seal(&engine, vec![(region_c, rows(&schema_c, 0, 3))])
+        .await;
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(writer);
+    drop(store);
+
+    // Its prefix held entries only in objects below 2, so neither watermark
+    // can name one of its objects, while region C, whose history it holds,
+    // opens. Every start adds a start object, which holds no entry, so a
+    // second start after the refused one refuses the regions as well.
+    for _ in 0..2 {
+        let store = open_store(&other_location, AckMode::Durable).await;
+        let engine = new_engine(&mut env, store.clone()).await;
+        for (region_id, table_dir, watermark) in [
+            (REGION_A, &table_dir_a, entry_id(2, 1)),
+            (REGION_B, &table_dir_b, entry_id(3, 1)),
+        ] {
+            let err = open_region(&engine, region_id, table_dir, &[])
+                .await
+                .unwrap_err();
+            let reason = format!(
+                "its watermark, entry {watermark}, names object {}, but the prefix held entries only in objects below 2 when the WAL opened",
+                watermark >> 20
+            );
+            assert_missing_history(&err, region_id, &reason);
+            assert!(!engine.is_region_exists(region_id));
+        }
+        open_region(&engine, region_c, &table_dir_c, &[])
+            .await
+            .unwrap();
+        assert_eq!(3, engine.get_region_statistic(region_c).unwrap().num_rows);
+        engine.stop().await.unwrap();
+        drop(engine);
+        drop(store);
+    }
+}
+
+#[tokio::test]
+async fn test_reopen_after_collection_left_only_the_newest_object() {
+    let mut env = TestEnv::with_prefix("object-store-wal-newest-object").await;
+    let object_store = memory_store();
+    let store = open_store(&object_store, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    let (table_dir_a, schema_a) = create_region(&engine, REGION_A, &[]).await;
+    let (table_dir_b, schema_b) = create_region(&engine, REGION_B, &[]).await;
+    // Region A flushes object 1; object 2 of region B carries the boundary
+    // past it, so once region B flushes too only object 2 is left.
+    let mut writer = SealedWriter::new(&store);
+    writer
+        .put_and_seal(&engine, vec![(REGION_A, rows(&schema_a, 0, 2))])
+        .await;
+    flush_region(&engine, REGION_A, None).await;
+    writer
+        .put_and_seal(&engine, vec![(REGION_B, rows(&schema_b, 0, 3))])
+        .await;
+    flush_region(&engine, REGION_B, None).await;
+    wait_for_collection(&store).await;
+    assert_eq!(vec![2], wal_object_seqs(&object_store).await);
+    engine.stop().await.unwrap();
+    drop(engine);
+    drop(writer);
+    drop(store);
+
+    // The object region A flushed is gone, but the newest object stays and
+    // keeps the next sequence above it, so both regions open.
+    let store = open_store(&object_store, AckMode::Durable).await;
+    let engine = new_engine(&mut env, store.clone()).await;
+    open_region(&engine, REGION_A, &table_dir_a, &[])
+        .await
+        .unwrap();
+    open_region(&engine, REGION_B, &table_dir_b, &[])
+        .await
+        .unwrap();
+    assert_eq!(
+        entry_id(1, 1),
+        region_recovery_state(&engine, REGION_A)
+            .await
+            .flushed_entry_id
+    );
+    assert_eq!(2, engine.get_region_statistic(REGION_A).unwrap().num_rows);
+    assert_eq!(3, engine.get_region_statistic(REGION_B).unwrap().num_rows);
+}
+
 #[rstest]
 #[case(AckMode::Durable)]
 #[case(AckMode::Enqueued)]

src/mito2/src/error.rs

@@ -461,6 +461,14 @@ pub enum Error {
         source: BoxedError,
     },
 
+    #[snafu(display("Failed to check the WAL history, region_id: {}", region_id))]
+    CheckWalHistory {
+        region_id: RegionId,
+        #[snafu(implicit)]
+        location: Location,
+        source: BoxedError,
+    },
+
     // Shared error for each writer in the write group.
     #[snafu(display("Failed to write region"))]
     WriteGroup { source: Arc<Error> },
@@ -1465,7 +1473,8 @@ impl ErrorExt for Error {
             WriteWal { source, .. }
             | ReadWal { source, .. }
             | DeleteWal { source, .. }
-            | WaitWalDurable { source, .. } => source.status_code(),
+            | WaitWalDurable { source, .. }
+            | CheckWalHistory { source, .. } => source.status_code(),
             CompressObject { .. }
             | DecompressObject { .. }
             | SerdeJson { .. }
@@ -1684,6 +1693,7 @@ impl ErrorExt for Error {
             | ReadWal { source, .. }
             | DeleteWal { source, .. }
             | WaitWalDurable { source, .. }
+            | CheckWalHistory { source, .. }
             | FetchManifests { source, .. }
             | External { source, .. } => source.retry_hint(),
 

src/mito2/src/region/catchup.rs

@@ -21,7 +21,7 @@ use store_api::logstore::LogStore;
 
 use crate::error::{self, Result};
 use crate::region::MitoRegion;
-use crate::region::opener::replay_memtable;
+use crate::region::opener::{check_wal_history, replay_memtable};
 use crate::wal::Wal;
 use crate::wal::entry_distributor::WalEntryReceiver;
 
@@ -89,6 +89,12 @@ impl<S: LogStore> RegionCatchupTask<S> {
             "Trying to replay memtable for region: {region_id}, provider: {:?}, replay from entry id: {replay_from_entry_id}, flushed entry id: {flushed_entry_id}",
             self.region.provider
         );
+        check_wal_history(
+            &self.wal,
+            &self.region.provider,
+            &self.region.version_control.current().version,
+        )
+        .await?;
         let timer = Instant::now();
         let wal_entry_reader = self
             .entry_receiver

src/mito2/src/region/opener.rs

@@ -63,7 +63,7 @@ use crate::memtable::{MemtableBuilderProvider, ensure_json2_not_use_time_series_
 use crate::metrics::{CACHE_FILL_DOWNLOADED_FILES, CACHE_FILL_PENDING_FILES};
 use crate::read::series_candidate::is_sparse_metric_metadata;
 use crate::region::options::RegionOptions;
-use crate::region::version::{VersionBuilder, VersionControl, VersionControlRef};
+use crate::region::version::{Version, VersionBuilder, VersionControl, VersionControlRef};
 use crate::region::{
     ManifestContext, ManifestStats, MitoRegion, MitoRegionRef, RegionLeaderState, RegionRoleState,
     RegionStats,
@@ -360,6 +360,9 @@ impl RegionOpener {
                     region_dir, self.region_id
                 );
             }
+            // The region exists but the WAL does not hold its history:
+            // creating it again would replace its manifest.
+            Err(e @ error::Error::CheckWalHistory { .. }) => return Err(e),
             Err(e) => {
                 warn!(e;
                     "Failed to open region {} before creating it, region_dir: {}",
@@ -385,6 +388,11 @@ impl RegionOpener {
         };
         ensure_json2_not_use_time_series_memtable(&metadata, &options)?;
         options.validate_column_compression(&metadata)?;
+        // The region's WAL history starts in this log store, so reopening
+        // the region on it passes the history check without any entry. It is
+        // recorded before the manifest exists: a log store that gave up its
+        // history meanwhile fails the create before it leaves a manifest.
+        wal.start_region_history(region_id, &provider)?;
         // Create a manifest manager for this region and writes regions to the manifest file.
         let mut region_manifest_options =
             RegionManifestOptions::new(config, &region_dir, &object_store);
@@ -502,6 +510,8 @@ impl RegionOpener {
     }
 
     /// Tries to open the region and returns `None` if the region directory is empty.
+    /// A region it replays must find its WAL history in the log store, see
+    /// [`check_wal_history`].
     async fn maybe_open<S: LogStore>(
         &mut self,
         config: &MitoConfig,
@@ -605,6 +615,7 @@ impl RegionOpener {
             .unwrap_or_default()
             .max(flushed_entry_id);
         let topic_latest_entry_id = if !self.skip_wal_replay {
+            check_wal_history(wal, &provider, &version_control.current().version).await?;
             info!(
                 "Start replaying memtable at replay_from_entry_id: {} for region {}, manifest version: {}, flushed entry id: {}, elapsed: {:?}",
                 replay_from_entry_id,
@@ -909,6 +920,25 @@ pub(crate) fn check_recovered_region(
     Ok(())
 }
 
+/// Checks that the log store holds the WAL history of a region that writes
+/// to the WAL through the largest replay watermark `version` persisted,
+/// before the region is replayed. A region that skips the WAL keeps no
+/// history there.
+pub(crate) async fn check_wal_history<S: LogStore>(
+    wal: &Wal<S>,
+    provider: &Provider,
+    version: &Version,
+) -> Result<()> {
+    if version.options.skip_wal {
+        return Ok(());
+    }
+    let watermark = version
+        .flushed_entry_id
+        .max(version.truncated_entry_id.unwrap_or_default());
+    wal.check_region_history(version.metadata.region_id, provider, watermark)
+        .await
+}
+
 /// Replays the mutations from WAL and inserts mutations to memtable of given region.
 pub(crate) async fn replay_memtable<F>(
     provider: &Provider,

src/mito2/src/wal.rs

@@ -36,7 +36,10 @@ use store_api::logstore::provider::Provider;
 use store_api::logstore::{AppendBatchResponse, LogStore};
 use store_api::storage::RegionId;
 
-use crate::error::{BuildEntrySnafu, DeleteWalSnafu, Result, WaitWalDurableSnafu, WriteWalSnafu};
+use crate::error::{
+    BuildEntrySnafu, CheckWalHistorySnafu, DeleteWalSnafu, Result, WaitWalDurableSnafu,
+    WriteWalSnafu,
+};
 use crate::wal::entry_reader::{LogStoreEntryReader, WalEntryReader};
 use crate::wal::raw_entry_reader::{LogStoreRawEntryReader, RegionRawEntryReader};
 
@@ -158,6 +161,41 @@ impl<S: LogStore> Wal<S> {
         }))
     }
 
+    /// Checks that the log store holds the WAL history of the region through
+    /// `entry_id`, the largest replay watermark the region persisted, before
+    /// the region is replayed from its persisted state.
+    pub(crate) async fn check_region_history(
+        &self,
+        region_id: RegionId,
+        provider: &Provider,
+        entry_id: EntryId,
+    ) -> Result<()> {
+        if let Provider::Noop = provider {
+            return Ok(());
+        }
+        self.store
+            .check_region_history(provider, entry_id)
+            .await
+            .map_err(BoxedError::new)
+            .context(CheckWalHistorySnafu { region_id })
+    }
+
+    /// Records that the WAL history of the region starts in the log store,
+    /// as it does for a region created on it.
+    pub(crate) fn start_region_history(
+        &self,
+        region_id: RegionId,
+        provider: &Provider,
+    ) -> Result<()> {
+        if let Provider::Noop = provider {
+            return Ok(());
+        }
+        self.store
+            .start_region_history(provider)
+            .map_err(BoxedError::new)
+            .context(CheckWalHistorySnafu { region_id })
+    }
+
     /// Returns a [WalEntryReader]
     pub(crate) fn wal_entry_reader(
         &self,

src/store-api/src/logstore.rs

@@ -122,6 +122,32 @@ pub trait LogStore: Send + Sync + 'static + std::fmt::Debug {
     ) -> Result<(), Self::Error> {
         Ok(())
     }
+
+    /// Checks, before an existing region is replayed from its persisted
+    /// state, that the log store holds the region's WAL history through
+    /// `entry_id`, the largest replay watermark the region persisted, so that
+    /// an open against a WAL that lost that history fails instead of
+    /// silently replaying nothing.
+    ///
+    /// The default accepts, which keeps the behaviour of every log store that
+    /// does not override it.
+    async fn check_region_history(
+        &self,
+        _provider: &Provider,
+        _entry_id: EntryId,
+    ) -> Result<(), Self::Error> {
+        Ok(())
+    }
+
+    /// Records that the WAL history of the provider's region starts in this
+    /// log store, as it does for a region created on it, so that
+    /// [`check_region_history`](Self::check_region_history) accepts a later
+    /// open of the region by the same log store.
+    ///
+    /// The default does nothing.
+    fn start_region_history(&self, _provider: &Provider) -> Result<(), Self::Error> {
+        Ok(())
+    }
 }
 
 /// The response of an `append` operation.