rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-10-12 23:12:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: add the object store WAL RFC (#9195)

Browse Source 
* docs: add the object store WAL RFC

Propose an object store backed write-ahead log as a WAL provider next to
Raft Engine and Kafka: a datanode batches the entries of its regions into
immutable objects under one prefix, creates each object conditionally
under a monotonically increasing sequence, acknowledges a write once its
object is durable by default, and replays each region from its own
segments after listing the prefix. A reference implementation exists and
follows this RFC in a series of pull requests; the document describes it
and marks what is proposed on top of it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: link the object store WAL RFC to its tracking issue

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: describe the predecessor chain and instance epoch of the object store WAL

Recovery replays only the chain of objects that ends at the latest
complete object, and every open starts a new epoch with an empty start
object, so an object whose create was reported as failed is never
replayed once a later append has been acknowledged. Update the object
format, recovery, writing, the failure matrix, the garbage collection
constraints, compatibility, evidence, drawbacks and the cluster mode
alignment accordingly.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: identify object store WAL objects by sequence and epoch

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: derive the object store WAL epoch from the start object sequence

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: separate the retryable and the conflicting start object outcomes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: describe garbage collection of the object store WAL through the reclaim boundary

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: judge vanished objects after chain selection and fix the evidence references

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: record the stop-time conflict and the completed fault matrix in the object store WAL RFC

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: limit the stop-time conflict to objects of the same or a later epoch

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: keep the conflicting-content condition in the stop-time conflict sentence

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: report the S3 write path measurements of the object store WAL

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: qualify the seal timer findings and link the S3 runs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs: state the serial durable cycle behind the one-batch-per-interval observation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
This commit is contained in:
jeremyhi
2026-10-12 12:00:55 +00:00
committed by GitHub
parent 70a0c826ca
commit 4ef312ecf2
1 changed files with 352 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfcs/2026-09-06-object-store-wal.md
+352
View File
@@ -0,0 +1,352 @@
---
Feature Name: Object Store WAL
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/9197
Date: 2026-09-06
Author: jeremyhi
---
# Summary
Add an object store backed write-ahead log as a first-class WAL provider, alongside Raft Engine and Kafka. A datanode batches the entries of all its regions into immutable objects under one prefix, creates each object conditionally under a monotonically increasing sequence number, and by default acknowledges a write only after the object holding it is durable; an opt-in acknowledgement mode returns on admission with a stated loss bound. Recovery lists the objects, selects the chain of objects that ends at the latest complete one, rebuilds an in-memory catalog from their footers, and replays each region from its own segments. The provider is standalone-only and marked experimental in this RFC; the persisted metadata and object format are designed so that lifting either restriction later does not require a migration.
A reference implementation of everything this document describes as implemented exists and was validated as reported under *Evidence*; it is submitted in a series of pull requests that follow this RFC, each covering one part of the design in the order the *Details* sections give. The document keeps two things apart: what that implementation does, and what is proposed on top of it. Every passage that describes proposed behaviour is introduced with the word *Proposed*; everything else describes the reference implementation. The *Future work* and *Unresolved questions* sections are proposed by definition and carry no marker.
# Motivation
GreptimeDB already stores data files in object storage. The WAL is the last component that still needs either local disk with durability guarantees (Raft Engine) or an external cluster (Kafka). Both are awkward for a cloud-native standalone deployment:
- Raft Engine ties the node to a persistent volume. Losing the volume loses unflushed writes; moving the node means moving the volume.
- Kafka is an operational dependency of its own, with topics, retention and a broker fleet to size, and it is disproportionate for a single node.
Object storage is already provisioned, already durable across zones, and already the place the rest of the data lives. An object store WAL takes the WAL out of the local state that must survive a restart: unflushed writes are no longer tied to a volume, and a node restarted against the same bucket and prefix replays them. It does not yet make a standalone node independent of local disk altogether, because standalone keeps its metadata in a local key-value store under the data home regardless of the WAL provider; moving that metadata off local disk is a separate change. The cost is write latency under the default durable acknowledgement: a write waits for its batch to seal, for every earlier object to be durable, and for one PUT. That is acceptable for the ingestion patterns GreptimeDB targets when writes are batched; *Write path measurements* under *Evidence* shows where the cost lies.
# Goals
- A WAL provider that needs nothing but an object store and a prefix.
- Exactly-once replay of acknowledged writes across graceful and ungraceful restarts under the default acknowledgement mode. The `enqueued` mode is an explicit choice whose loss is bounded to the unpersisted backlog and reported.
- Region isolation: many regions share objects, but a region only ever replays its own entries.
- Recovery that never silently loses data. Structural corruption fails recovery. A corrupted segment is skipped at segment granularity by default, and the skip is recorded for its region, counted and logged rather than hidden; `on_corrupted_segment = "fail"` fails the read instead.
- Default behaviour of the existing providers unchanged; the new provider is opt-in through configuration.
# Non-goals
- Distributed mode. The provider is rejected when a datanode is controlled by a metasrv; the metasrv side has no allocation logic for it yet.
- Persistence of per-region obsolete watermarks. Mito re-establishes every watermark when it opens a region, which is what garbage collection reasons from; persisting them is a follow-up.
- Matching Raft Engine's write latency. The provider trades latency for the absence of local durable WAL state.
- Making standalone metadata independent of local disk. The metadata store is unchanged by this RFC.
# Details
## Identity
The provider is added to the four parallel WAL enums so that every resolution path handles it explicitly:
| Enum | Variant | Notes |
| --- | --- | --- |
| `DatanodeWalConfig` | `ObjectStore(ObjectStoreWalConfig)` | serde tag `experimental_object_store`; fields `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes`, `ack_mode`, `max_unpersisted_bytes`, `max_unpersisted_age`, `on_corrupted_segment` |
| `WalOptions` | `ObjectStore(ObjectStoreWalOptions { prefix })` | persisted per region with the stable tag `object_store`; flat key `wal.object_store.prefix`; the value is the derived node prefix |
| `WalProvider` | `ObjectStore { prefix }` | allocates the per-region options in standalone under the derived node prefix; `start` is a no-op |
| `Provider` | `ObjectStore(Arc<ObjectStoreProvider { region_id, prefix }>)` | `is_remote_wal()` is true |
Two choices are deliberate:
- The persisted `WalOptions` carry the derived node prefix `<prefix>/datanodes/<node_id>/epochs/<generation>` described under *Object format*, not the configured root. On reopen the log store compares the persisted prefix with the one it runs under and fails fast on a mismatch, so an operator who changes the prefix or the node id cannot silently point a region at an empty WAL. Changing the prefix of an existing region requires an explicit migration, which is out of scope here.
- `Provider::ObjectStore` is classified as a remote WAL. Mito's remote-WAL branches (initial flushed entry id from `latest_entry_id`, `topic_latest_entry_id` after replay, region-scoped reads, no topic grouping on batch open) apply unchanged. This is a classification of replay semantics, not of physical location.
A datanode that is configured with a metasrv client rejects the provider before any store is built: `DatanodeBuilder` returns an error stating that the object store WAL is standalone-only. Separately, `DatanodeWalConfig::ObjectStore` cannot be converted into `MetasrvWalConfig`, so a metasrv cannot be configured with it either. The standalone bootstrap matches the datanode WAL configuration before that conversion and constructs `WalProvider::ObjectStore` directly.
## Object format
Every object holds one sealed batch:
```text
header | segment(region 1) | ... | segment(region N) | footer | trailer
```
- The header carries the magic `GTWALOBJ`, format version 1, the object sequence number, the epoch of the store instance that wrote the object, a link to the object this one extends, as that object's sequence number and epoch or none for the first object of a chain, the reclaim boundary described under *Entry ids*, and a CRC32 of the preceding header bytes; the header is 54 bytes. A reclaim boundary above the object's own sequence fails the header check. A sequence number and an epoch together identify an object: an instance never writes two objects under one sequence, and instances have distinct epochs, as described under *Log store*. Recovery selects the history it replays by the links and the epochs, as described under *Log store*.
- An object may hold no segment. A store writes such an object when it opens, as described under *Log store*; it is a chain object like any other, and its segments are trivially all flushed.
- Each segment holds the entries of exactly one region, ordered by entry id; segments are ordered by region id. Ordering makes the encoding deterministic, which the identical-retry rule below depends on.
- The footer records, per segment, the region id, the entry id range, the entry count, the byte range and a CRC32 of the segment.
- The fixed-size trailer carries the magic `GTWALTRL`, the footer location, a CRC32 of the footer and a CRC32 of the whole object. A reader locates the footer by reading the trailer at the end of the object and can decode a single region's segment without touching the others.
Corruption is handled at two levels; nothing else is checked in production. Structural damage fails recovery: recovery reads only the header, the trailer and the footer of every object, as described under *Log store*, and verifies the header and trailer magic, the format version, the header CRC, the header sequence against the key, the footer CRC and that the segment byte ranges tile the object body exactly; the first object that fails any of these checks fails construction of the store, since nothing can be trusted about such an object. A segment is verified by the read that decodes it, and a segment that does not decode, because its CRC does not match or its content disagrees with its footer entry, is a data-level failure confined to one region. The read fetches the segment once more, since the download rather than the object may be damaged, and decodes it again; if it still does not decode, the behaviour follows `on_corrupted_segment`. `skip`, the default, skips the segment, records a WAL hole for the region with the object key, the object sequence and the entry id range the footer holds for the segment, increments the counter `greptime_logstore_object_store_wal_skipped_segments_total` and logs a warning with those fields, and the read continues with the next object; every other region in the object is unaffected. The holes are kept in memory per region and can be listed from the store; nothing is persisted, so a store opened later on the same prefix records a hole again when a read meets the segment. `fail` fails the read with the decode error, so a region whose replay meets the segment does not open. Skipping loses only the unflushed entries of that region in that object: nothing already flushed is affected, and the loss is reported rather than hidden. A failed fetch is not corruption and fails the read in both modes, unless garbage collection deleted the object, as described under *Entry ids*, in which case the object is skipped; an object the catalog still holds once a delete of it has settled fails the read whether it is missing or unreadable, and a segment of such an object that does not decode follows `on_corrupted_segment` as it would without any collection. The whole-object CRC is not a recovery requirement; it stays in the trailer as a fixture-level check. Because recovery never checks it, the header carries its own CRC, so a damaged link fails recovery instead of silently cutting acknowledged objects off the chain. The format is frozen at version 1.
The operator configures only the prefix root. The store runs under a node prefix derived from it, `<prefix>/datanodes/<node_id>/epochs/<generation>`, and object keys are `<node prefix>/objects/<sequence>.wal`, where the sequence is zero-padded to 20 decimal digits. Standalone uses its datanode id, zero unless configured, and generation zero, so with prefix `wal` the first object is `wal/datanodes/0/epochs/0/objects/00000000000000000000.wal`. Every place that names the prefix, whether to open the store or to allocate the WAL options of a region, derives it from the same function, so the prefix a region persists matches the one its store runs under. Listing accepts only well-formed keys under `<node prefix>/objects/` and returns them in sequence order. This is the standalone form of the key layout under *Alignment with cluster mode*; the cluster assignment of node id and generation is proposed there.
## Catalog
The catalog is an in-memory index rebuilt at recovery from the objects on the replayed chain: objects by sequence number, and per region the objects that hold its entries with their entry id ranges. Its invariants are enforced on insertion: a region's entry id ranges are strictly increasing along the object sequence, an object holds at most one segment per region, and an already indexed sequence number is rejected, whether or not the footer matches; an object without segments is indexed like any other and takes its sequence. The next sequence number is the larger of the largest indexed one plus one and the floor that any region's largest entry id sets, as described under *Entry ids*, checked against the sequence limit; construction raises it above every listed object, as described under *Log store*, and gaps in the sequence are tolerated.
## Conditional create
Objects are created with an if-not-exists put. Three outcomes matter:
| Outcome | Meaning |
| --- | --- |
| created | the batch is durable |
| exists with identical content | a retry of a batch that was durable before the client observed it; accepted |
| exists with different content | another object owns this sequence; its epoch decides, as described under *Log store* |
Deterministic encoding is what makes the middle row decidable.
## Entry ids
Entry ids are object-sequence-major. The high bits are the sequence number of the object that holds the entry, the low bits are the entry's position among that region's entries inside the object:
```text
entry_id = (object_seq << 20) | position (1 <= position < 2^20, object_seq < 2^44)
```
Positions start at one, so entry id zero is never assigned: Mito treats zero as the watermark of a region that has no durable entry and replays from the next id, and the first object has sequence zero. Ids are assigned at admission, from the sequence the open batch takes when it is sealed. A store never hands out a sequence twice, so a batch that fails is never written again under its sequence, and a retry of its entries is a new batch with new ids. A batch that would take a region past the position range is sealed first and the entries open the next batch; an append that alone exceeds the range is rejected. This keeps two properties at once. `entry_id >> 20` names the object that holds an entry without consulting any index, which is what a future cross-node takeover needs to describe a WAL position as a range of object sequences. And every entry has a unique, strictly increasing id within its region, so a flush that lands between two appends of the same region that ended up in the same object records exactly which entries it covered; replay from `flushed_entry_id` can never skip an unflushed entry or repeat a flushed one. A region's ids have gaps wherever other regions or other positions took the sequence, exactly as Kafka offsets do, and Mito already tolerates gaps; the recovered history of a region may also lack whole objects where a batch was never acknowledged, as described under *Writing*. The id values are encoded in two places that must agree: in each record inside its segment and in the footer's entry id range for that segment.
`latest_entry_id(provider)` is region-scoped: the highest entry id durable under the prefix for the provider's region, or zero when the region has none. A freshly created region therefore starts at zero even if other regions under the prefix have entries. This differs from Kafka, whose high watermark is per topic; it is sufficient here because ids are per region, and Mito consumes the value only for the region that asked.
Two rules keep every new id of a region above every id it already has, on a prefix that already holds objects and across the watermark Mito hands the store. On open, the next object sequence is the larger of the largest listed sequence plus one, orphans included, and, over every region, the object named by the region's largest existing entry id plus one, that is `(max_entry_id >> 20) + 1`, so that the catalog's strictly increasing ranges hold; sequence gaps are tolerated, and a sequence at or above `2^44` is rejected as exhausted. Objects written under an earlier contiguous scheme are not rewritten: they remain readable, their ids carry no object information, and the rule places them by their high bits just the same.
The second rule is the sequence floor of `obsolete`. Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, and the actor applies the watermark and a floor derived from that entry id together: the next object sequence is raised to `(entry_id >> 20) + 1` unless it is there already, while zero, the watermark of a region without entries, names no object and needs no floor, so that a watermark assigned under another prefix, or one whose object this prefix no longer holds, is never passed by a new id. The floor moves only the next sequence, never the chain: the next batch still links to its predecessor, so a skipped sequence leaves no gap in the history recovery replays. While the open batch holds ids under the next sequence, the call is refused with a retryable error and records neither the watermark nor the floor, because those ids would fall below the floor. In the `enqueued` mode an id the store itself handed out needs no floor, even before it is durable: it is never handed out again while the store runs, so every later id of the region is greater. In the current single-prefix layout the floor never needs to move, since every watermark Mito hands the store names an object below the next sequence; the rule exists for the cross-prefix cases under *Alignment with cluster mode*. No additional persisted state is introduced.
Garbage collection deletes the objects below a reclaim boundary that the store persists in every header, so that recovery can tell a collected predecessor from one that never landed. The boundary is computed from the catalog and the watermarks the store already keeps: each region with a recorded watermark contributes the sequence of its first indexed object holding a segment whose largest entry id is above the watermark; a region without a recorded watermark contributes its first indexed object; a region whose watermark is the maximum entry id, after `obsolete_all`, contributes nothing; the boundary is the smallest contribution, or the last indexed sequence plus one when nothing contributes, and it never decreases. Whether a segment is flushed is decided per segment from the footer, on the full entry id, so a watermark that stops in the middle of an object keeps the object. The boundary is recomputed after every successful `obsolete`, after `obsolete_all` and after every object is indexed; Mito calls `obsolete` after a flush, after a truncate, when it drops a region and when it opens one, so a restart re-establishes every watermark and the boundary advances on the same rule. Every sealed batch carries the boundary current at sealing and the start object carries the boundary recovery found, so a boundary is always at or below the sequence of the object carrying it. The boundary that permits deletion is the one carried by the last indexed object; the boundary of a batch whose create failed or is in flight permits nothing.
Every present object with a sequence below the durable boundary is deletable, on the chain or not: the objects the catalog indexes, the objects recovery listed but did not index, the objects of an earlier epoch that the start object moved past, and the sequences of batches that failed in this run, whose objects may or may not exist. A collection starts after a successful `obsolete`, after `obsolete_all` and after an indexing that raised the durable boundary. Deletes run in the background, at most four at a time and in no particular order, and never hold up admission, sealing, uploads or acknowledgements; an indexed object is unindexed only after its delete succeeded, and the WAL holes recorded for it are dropped with it. Every deleted object is counted in `greptime_logstore_object_store_wal_deleted_objects_total`; a failed delete is logged, counted in `greptime_logstore_object_store_wal_failed_deletes_total` and retried at the next collection, and the object stays indexed meanwhile. An object deleted but not yet unindexed when the process crashes is simply absent at the next recovery, which lists the prefix. Nothing is collected once `stop` has begun or while the store is poisoned; a delete in flight when `stop` begins runs to completion before `stop` returns, and the actor settles every attempt it abandons when it exits.
A read never needs a deleted object, since entries at or below a region's watermark are hidden anyway, but a read that listed an object before it was collected skips the object when it gets there. A read that meets an error on such an object makes two observations, in the reverse of the order the actor writes them: it looks at the set of objects whose delete is in flight, and then at the catalog. The actor registers an object in that set before its delete starts, unindexes it if the delete succeeded, and only then takes it out, so an object absent from the set was either never collected, in which case the catalog still holds it, or is already unindexed by the time the catalog is read. Observing the catalog first would let a collection complete between the two observations and leave the read with an object that still looked indexed and no longer looked deleting. An object that is still in the set has an undecided delete, so the read waits for that one delete, a single object store request, rather than assume it succeeds: a delete that fails leaves the object present and indexed, and the error the read met is then reported as any other, `on_corrupted_segment` included. What the read waits for is the delete attempt its observation found, taken together with the lookup, not the object: a retry of the same object is a separate attempt, so the attempt the read holds releases it even while that retry is in flight, and a read is never handed from one attempt to the next. A read stream outlives the store it came from, so the actor settles every attempt it abandons when it exits, whether the store was stopped or dropped; the objects of those attempts stay present and indexed, so the reads waiting for them report the error they met. A collected object is skipped silently: it is not recorded as a hole and does not count as a skipped segment. An object the catalog still holds after those observations fails the read like any other fetch failure, `on_corrupted_segment` included. Within one run the catalog keeps the largest entry id of a region after the objects holding it were collected, so `latest_entry_id`, the durability wait and the `enqueued` clamp keep their reference point; a store opened later learns only what is listed, so after a restart `latest_entry_id` of a region whose later objects were collected is the largest id among the objects left, which only the pruning hint of Mito consumes. No state is persisted outside the headers and no configuration option is added.
## Log store
`ObjectStoreLogStore` implements the `LogStore` trait for `Provider::ObjectStore`.
**Construction and recovery.** The prefix, flush interval, batch size and backlog thresholds are validated, then the store lists the prefix and fetches, for up to eight objects at a time, the header and a window at the end of each object with range reads; a short object is read whole, and the footer is read separately only when it starts before the window. Each header and footer is verified as described under *Object format*; the first corrupted object fails construction; a fetch that finds the object gone is judged once the history is selected, as described below, and any other failed fetch abandons the rest. Segments are not read, so recovery costs a few small reads per object however large the objects are.
The store then selects the history it replays from the headers. An object's link holds when the object has no predecessor, when its predecessor is present and carries the epoch the link records, or when the predecessor's sequence is below the reclaim boundary the object carries, in which case the chain ends there whether or not the predecessor is present; above that boundary a missing predecessor breaks the link. An object is complete when every link on its chain holds, each judged with the boundary of the object carrying the link. The tip is the complete object with the largest epoch and, among those, the largest sequence. Only the objects on the tip's chain are indexed and replayed, in sequence order whatever the order the fetches completed in, so the catalog checks the entry ranges of every object against its predecessors like a sequential replay, and a conflicting object fails construction. Every other present object is an orphan: it is neither indexed nor read, but its sequence counts, so the next sequence is above every listed object, and the objects below the tip's boundary are collected. A listed object whose header or footer is gone by the time it is fetched is ignored when it lies below the tip's boundary and fails construction when it lies at or above it, since collection never deletes one there. Construction also fails when a listed object carries an epoch above the next sequence, which no instance can have produced, since an epoch is a start sequence plus one. When objects are present but none is complete, construction fails rather than start an empty history. Construction then records the largest durable entry id per region.
Before it admits any append, the store writes a start object: an object without segments at the next sequence, which is above every listed object, orphans included, linked to the tip, or to nothing on an empty prefix. The instance's epoch is the sequence of its start object plus one, and every object the instance writes carries it. The conditional create assigns each sequence to exactly one object, so no two instances share an epoch; and the start sequence is above every object present at open, including the start objects of every earlier instance, so the epoch of every later instance is greater than that of any object an earlier instance ever writes, including one whose create lands after that instance stopped or crashed, and such an object never becomes the tip. Only once the start object is created does the store accept appends, and the first batch links to it. If the sequence turns out to be taken, the store reads the object there: an object of an earlier epoch is stale, and the start object moves to the next sequence, taking the epoch that sequence gives; an object of the same or a later epoch means another writer, and construction fails with a conflict error that is not retried; an object with identical content, which another open of the same prefix has just written, fails construction with a retryable error, so that the next attempt lists it and starts above it. A create whose outcome is unknown fails construction with a retryable error too: an object it may have left is listed by the next attempt, which therefore starts above it. Two assumptions remain. Listing the prefix is strongly consistent, so an open sees every start object that was durable before it listed; this holds for S3, GCS, Azure Blob Storage and MinIO, and an S3-compatible store without it is not supported. And one process at a time opens a prefix: two processes opening it concurrently either contend for the same start sequence, where the conditional create lets one through and fails the other's construction, or, if one of them listed a late object the other did not, start under different sequences and epochs and both proceed until their sequences meet, where the writer that meets the other's object follows the conflict rules under *Writing* and the *Failure matrix*: in the `durable` mode an object of an earlier epoch fails its batch and one of a later epoch poisons it, and in the `enqueued` mode any conflict poisons it, so which writer is poisoned depends on who observes the collision, not on the epochs alone; the store does not fence concurrent writers beyond that, and in cluster mode the generation in the key prefix does.
**Writing.** One background actor per store admits appended entries into a single open batch and assigns entry ids as described under *Entry ids* above. The batch is sealed when its estimated size reaches `max_batch_bytes` or when `flush_interval` elapses with entries in it; a timer tick on an empty batch creates nothing. Sealing encodes the batch under the next sequence number and hands it to a pipelined uploader: the actor keeps admitting entries into the next batch while uploads are in flight, and up to four conditional creates run at a time, started in sequence order. Every sealed batch is encoded with the store's epoch and a link to its pipeline predecessor: the last batch sealed before it that is not indexed yet, or, when there is none, the last indexed object, which after opening is the start object. A link to the last indexed object while an earlier batch is still uploading would let a later chain skip that batch once it is acknowledged. Batches are indexed in sequence order in both acknowledgement modes: a created object is not indexed until every earlier batch is. The rest of this paragraph is `durable` mode behaviour; the `enqueued` mode, whose appends have already returned, is described under *Acknowledgement*. In the `durable` mode a batch is acknowledged when it is indexed, so a batch whose object is durable is not acknowledged until every earlier batch is durable too, and the acknowledged prefix of a region's history never has a missing predecessor. When a create fails transiently, every batch that is not indexed fails at once, including batches whose objects were already created and batches whose creates are still in flight. Their sequence numbers are never handed out again, the next batch links to the last indexed object, and the store keeps serving. A create still in flight completes later and finds no batch to acknowledge, so whatever it stored is an orphan. The next object acknowledged links past every failed batch, and from then on recovery replays none of them, whether their objects exist, land later or never do. A create that meets a different object reads that object's header: an object of an earlier epoch is stale and can never be the tip, so the batch fails as for a transient error and the store keeps serving; an object of the same or a later epoch belongs to another writer on the prefix, and the store poisons itself.
The guarantee is therefore about acknowledged history. An acknowledged entry is always replayed; an entry whose append failed is replayed only when nothing was acknowledged after it, the same outcome as a crash between an object's creation and its acknowledgement, which the engine already tolerates. The distinction matters because Mito does not advance a region's committed sequence when an append fails and reuses those row sequences in its next write, so replaying a failed batch next to a later acknowledged one could let the unacknowledged rows win deduplication.
**Acknowledgement.** `ack_mode` selects when `append_batch` returns. `durable`, the default, returns only after the object holding the entries is durable and indexed, so callers never see an entry id for an entry that is not durable; a write-ahead log is expected to mean durability on return, which is why this stays the default. `enqueued` returns as soon as the entries are admitted, with their entry ids already assigned, and creates the object in the background. In that mode the recovery point objective is the unpersisted backlog. `max_unpersisted_bytes` and `max_unpersisted_age` are admission thresholds: when the open batch plus the sealed batches that are not durable yet reach either, new appends are held back in arrival order until an upload completes, and the store seals the open batch if no create is in flight so that there is an upload to wait for; nothing is dropped and the backlog cannot grow without limit. They are not a hard bound on what a crash can lose: during an outage the writes already acknowledged stay in memory for as long as the outage lasts, however old they get, and a crash in that window loses them.
The failure contract of `enqueued` differs from the durable one because no caller remains to retry. A transient create failure is repeated by the store after a short delay instead of failing the batch. A permanent failure, meaning a conflicting object or an encoding or catalog error, poisons the store. Once `stop` has begun a transient failure is no longer repeated: the backlog is dropped and `stop` reports the loss as its error, since those entries were acknowledged but never became durable. Otherwise `stop` seals and uploads the remaining backlog before it returns.
The mode also needs a barrier that the durable mode gets for free. An entry id that Mito has been handed may not yet be durable, and a flush that persisted such an id as the manifest's `flushed_entry_id` would, after a crash before the object was created, leave a watermark that refers to an object which never existed, so that the sequence is reused on restart and later entries land below the watermark and are skipped by replay. The `LogStore` trait therefore has `wait_durable(provider, entry_id)`, with a default that returns at once, so that every existing provider keeps its own acknowledgement and persistence semantics, whatever they are; the durability guarantee this section describes belongs to the object store provider alone. The object store WAL answers it once the region is durable and indexed through `entry_id`; an id the store never handed out has nothing to wait for. Mito waits before it publishes a manifest watermark: a flush waits after its SSTs are written and before the manifest edit, a full truncate waits before it records the truncated entry id as the replay frontier, and discarding unflushed data waits before it records the discarded entry id as flushed. The wait in a flush observes the flush's cancellation, so a cancelled flush does not hold up the drop or truncate queued behind it; a full truncate and a discard of unflushed data still run their own wait before publishing their frontier. After a backlog was lost, a durability wait for an entry that is not durable fails rather than certifying it, whether the entry was in that backlog or not. In `enqueued` mode `obsolete` also clamps the watermark to the region's durable entry id, since an entry that is not durable yet is replayed after a crash and hiding it would skip it. A manifest watermark can then only ever name a durable entry, and the restart allocation rule under *Entry ids* keeps every new id above it.
**Failure matrix.**
| Situation | Sequence | Waiters of the batch | Store |
| --- | --- | --- | --- |
| create succeeds, or identical retry | advances | acknowledged in sequence order | healthy |
| transient I/O error, `durable` mode | consumed and never reused, nor are the sequences of the other failed batches | this and every batch that is not indexed fail, including those whose objects were created; a create still in flight completes as an orphan | healthy: the next batch links to the last indexed object |
| an object of an earlier epoch holds the sequence, `durable` mode | consumed and never reused | as for a transient I/O error | healthy |
| transient I/O error, `enqueued` mode | unchanged | already acknowledged | healthy: the create is repeated after a short delay |
| transient I/O error, `enqueued` mode after `stop` began | unchanged | already acknowledged | the backlog is dropped and `stop` reports the loss |
| an object of the same or a later epoch holds the sequence, or any conflicting object in the `enqueued` mode | unchanged | every unacknowledged waiter fails, including those of batches whose objects were created but not yet indexed; an `enqueued` acknowledgement already returned cannot be revoked | poisoned |
| encoding or catalog error | unchanged | every unacknowledged waiter fails, including those of batches whose objects were created but not yet indexed; an `enqueued` acknowledgement already returned cannot be revoked | poisoned |
| create succeeds at the last representable sequence | cannot advance | acknowledged | poisoned: no later batch can be allocated a sequence, every later append fails |
| the start object meets an object of an earlier epoch | moves to the next sequence | none admitted yet | construction continues |
| the start object meets an object of the same or a later epoch, or its create has an unknown outcome | unchanged | none admitted yet | construction fails; an unknown outcome is retryable |
A store cannot be constructed on a prefix whose largest object already carries the maximum sequence. A poisoned store fails every `LogStore` operation except `stop` with the same terminal error; in particular `obsolete` can no longer move a watermark. In the `durable` mode transient errors are not retried inside the store, but a failed create is reconciled once: the store immediately reads the key back, accepts the batch if the object exists with identical bytes, treats it as a conflict if the bytes differ, and reports the transient error only when the key is absent or that read fails too. So a lost response alone does not fail the append. When the append does fail, neither the store nor Mito keeps the batch: it surfaces through Mito to the caller of the write, and a retry is a new write by the caller under a new sequence. If the create had in fact succeeded and both its response and the read-back were lost, or its request lands only later, the object is an orphan as soon as a later object is acknowledged. Retrying the batch itself inside the store in the `durable` mode would only hide the latency and would hold writes up for as long as an outage lasts; the `enqueued` mode repeats the create with the same bytes, so a create that succeeded without a response is accepted as an identical retry.
**Stopping.** `stop` is idempotent and awaits the actor. Once stop has begun, queued appends are not admitted, no timer or size triggered seal starts, and appends after stop, including empty ones, fail with the stopped error. In the `durable` mode the open batch and every sealed batch whose create has not started are dropped: their waiters receive the stopped error. A create that is already in flight runs to completion; if it succeeds while its batch is still pending, the batch is indexed in order and acknowledged. A transient failure after stop began fails every batch that is not indexed, as it does while the store runs, and those batches receive the stopped error; a create of such a batch that completes later is an orphan and acknowledges nothing. A create that finds a conflicting object of the same or a later epoch after stop began poisons the store as it would while the store runs, but the waiters of the batches that are not indexed receive the stopped error, the conflict is kept only as the terminal error, and `stop` returns successfully once the creates in flight have completed. In the `enqueued` mode the held-back appends receive the stopped error, the open batch is sealed and the whole backlog is uploaded before `stop` returns; a transient failure in that phase drops the backlog and `stop` returns the error, as described under *Acknowledgement*. A durability wait still pending when the actor exits receives the stopped error, and a delete already in flight runs to completion before `stop` returns.
**Reading.** `read(provider, entry_id)` returns the region's entries with id at least `entry_id`, in order, fetching only that region's segments with range reads and decoding nothing else, and hides entries at or below the region's obsolete watermark. `obsolete` records the watermark in memory, clamped to the durable entry id in the `enqueued` mode; the watermark is re-established after a restart because Mito calls `obsolete` with the manifest's `flushed_entry_id` when it opens a region, so no WAL-owned watermark state is persisted. `obsolete_all` hides the whole region. `latest_entry_id` is not affected by the watermark. Every method taking a provider requires `Provider::ObjectStore` with the store's prefix and the caller's region.
**Metrics.** The store exports metrics under `greptime_logstore_object_store_wal_`. The write path reports `seal_to_durable_seconds`, the time from sealing a batch until its object is durable, `append_ack_seconds`, the time an append waits for its acknowledgement in the `durable` mode, `object_bytes` and `object_entries`, the size and the entry count of every object sealed, and `created_objects_total`, the creates that succeeded, an identical retry included; a create is counted and timed whether or not its batch is still there to be indexed, since an object a poisoned store gave up on is as durable as any other. `indexed_objects` and `indexed_bytes` are the objects and the bytes the catalog holds, set by a recovery that succeeded and moved by every object indexed or unindexed afterwards, so a collection takes the object it deleted off both, and a recovery that is abandoned publishes neither. A recovery that succeeds reports `recovery_seconds` and the objects it read footers from in `recovered_objects_total`, both of the same recoveries, so one that is abandoned at a fetch or rejected by the catalog contributes to neither; a read stream reports `read_seconds` once it has been read to its end, and one that fails or is dropped part way reports nothing. The failure paths are counted: `create_conflicts_total` for a create that found a conflicting object, `create_failures_total` for one the object store did not confirm, `poisoned_total` for a store that recorded a terminal error, `skipped_segments_total` for a segment a read skipped, and `stalled_appends_total` with `stalled_append_seconds` for an append the `enqueued` backlog thresholds held back, timed until it is admitted or refused, a stop, a terminal error or the teardown of the actor releasing it included; the delete counters of a collection are described under *Entry ids*. The histogram buckets are chosen for the default flush interval of 100 ms and the default `max_batch_bytes` of 8 MiB.
A `testing` cargo feature exposes hooks to wait for admitted appends, to seal the open batch, to hold and release creates and to make creates fail, so engine tests can flush deterministically and observe entries that are admitted but not durable.
## Mito integration
Mito maps `WalOptions::ObjectStore { prefix }` to `Provider::object_store_provider(region_id, prefix)`; resolving object store options against a Raft Engine or Kafka log store reports the existing incompatible-provider error. Reads go through the region-scoped reader used for Kafka, so a namespace shared by many regions never leaks entries across regions even if a log store returned them. Object store regions are opened individually rather than grouped by topic. After replay, `topic_latest_entry_id` follows the existing remote-WAL rule: it is taken from the store's latest entry id only when the region's memtables are empty, otherwise it stays at the replay start, because that value feeds pruning and must never advance past unflushed data. Before a flush, a full truncate or a discard of unflushed data publishes a manifest watermark, Mito waits on the durability barrier described under *Acknowledgement*; with the `durable` mode and with every other provider the wait returns at once.
## Datanode wiring and lifecycle
The scalar options are validated before any store or engine exists: the prefix must be non-empty, relative and free of empty or `..` components; `flush_interval` at least 10 ms; `max_batch_bytes`, `max_unpersisted_bytes` and `max_unpersisted_age` positive. `storage_provider` is resolved later, once the object store manager has been built and before the log store and the Mito engine are: empty selects the default store, otherwise the name must match a configured provider. Failures in both stages are `InvalidArguments` naming the field, and both happen before any WAL object is touched. The datanode requires a node id to derive the node prefix it opens the store under, and the standalone bootstrap derives the same prefix for the WAL options it allocates.
`DatanodeBuilder` keeps the store handle from the moment it is created and stops it if a later build step fails. On success the handle moves into `Datanode`, whose shutdown stops the store after the region server. Shutdown of the datanode and of the standalone instance is best-effort and ordered: every step runs, the first error is returned, so a failing region engine does not leave the WAL actor alive.
## Configuration
The options:
```toml
[wal]
provider = "experimental_object_store"
# Name of a configured storage provider; empty selects the default store.
storage_provider = ""
prefix = "wal"
# How long a non-empty batch may stay open before it is sealed; at
# least 10ms. A shorter interval lowers the acknowledgement latency of
# sparse `durable` writes and raises the request count: timer-triggered
# sealing creates at most one object per interval per node, and in the
# `enqueued` mode the backlog thresholds can seal earlier. The batch
# seals on size only when the admitted bytes reach `max_batch_bytes`
# before the next timer tick.
flush_interval = "100ms"
# Estimated batch size at which the open batch is sealed. It is a
# threshold, not an upper bound: a single append larger than this is
# admitted whole, and object framing adds bytes on top. Object stores
# bill per request, and the cost-effective request size is 8 to 16 MiB.
max_batch_bytes = "8MB"
# "durable": append returns after the object is durable (the default).
# "enqueued": append returns on admission. New appends stall once the
# unpersisted backlog reaches either threshold below; the thresholds do
# not bound what a crash during an outage can lose.
ack_mode = "durable"
max_unpersisted_bytes = "64MB"
max_unpersisted_age = "8s"
# What a read does with a segment that still does not decode after a
# second fetch, because its checksum does not match or its content
# disagrees with its footer entry. "skip": the segment is skipped and
# recorded as a WAL hole of its region, counted and logged; the other
# regions of the object are unaffected (the default). "fail": the read
# fails, so the region does not open.
on_corrupted_segment = "skip"
```
`config/config.md` is generated from the example files.
## Compatibility
- `WalOptions::ObjectStore` is a new persisted variant. Existing `raft_engine`, `kafka` and `noop` options decode unchanged; a region's options never change provider implicitly.
- The object format is version 1 and frozen once the provider is released; a version bump after that is a new decoder, not an in-place change. The version 1 header layout was revised before release to carry the epoch, the link, the reclaim boundary and the header CRC. No object in an earlier layout exists, since the provider has not been enabled anywhere.
- The configuration tag carries `experimental_`; the persisted tag does not, so promoting the provider to stable is a configuration rename with an alias, not a metadata migration.
- `LogStore::wait_durable` has a default implementation that returns at once, so Raft Engine, Kafka and the noop provider keep their own acknowledgement and persistence semantics; the durability barrier binds only the object store provider.
- The object-sequence-major id scheme needs no migration of existing objects. A prefix written under the contiguous scheme keeps its objects, and the sequence allocation rule under *Entry ids* keeps new ids above the old ones. Garbage collection reasons from footers and from the reclaim boundary, not from the id encoding, so it applies to objects of either scheme.
- `on_corrupted_segment` is a new configuration option with a default, so existing configurations decode unchanged; it changes what a read does with a damaged segment, not what is written.
## Evidence
In summary, the evidence supports going ahead with the experimental standalone implementation. Targeted tests and a process-level crash gate cover the recovery paths and the durability barrier; a local MinIO run confirms recovery and garbage collection. In same-session comparisons on a laptop the default `durable` mode reaches about 23% to 28% less throughput than Raft Engine with synchronous writes; on S3 from an instance in the bucket's region, against Raft Engine on a local NVMe store, it reaches 31% of Raft Engine's throughput with one writer and 54% with eight, bound by the seal interval rather than by S3, while the `enqueued` mode matches Raft Engine and never stalls. Under sustained load the default interval costs about 600 object creates per minute per node. The pipelined uploader has shown no consistent throughput benefit; in a multi-region experiment with injected object store latency it lowered write latency at the cost of more object requests. Long-running stability and recovery at production scale remain unverified.
The reference implementation was validated on the following evidence; the tests are part of the pull requests that submit it. The first table maps each guarantee of this RFC to what was verified, the list after it names the failures that were driven through the store, and the last list states what the evidence does not cover.
### Guarantees verified
| Guarantee | Verified by | Result |
| --- | --- | --- |
| Acknowledged writes replay exactly once across graceful and ungraceful restarts under `durable` | Deterministic recovery tests on a real store over an in-memory object store: two regions sharing a prefix with only one flushed, an abrupt drop after an object is durable but before any flush, two consecutive restarts with writes and a flush between them. A process-level crash gate: concurrent writers, SIGKILL at a random point of a window, restart on the same bucket and data home, repeated cycles with replay accumulating. A standalone round trip against MinIO with a restart before and one after a flush. | The concrete flushed, replayed, manifest and latest entry ids and row-level scan equality after every restart. In the crash gate every acknowledged row is present exactly once after the restart, and every present row that was not acknowledged was in flight at the kill. |
| The loss of `enqueued` is bounded to the unpersisted backlog and reported | The same recovery tests and the same crash gate run in `enqueued` mode; unit tests of both admission thresholds stalling until an upload completes, of `stop` uploading the backlog, and of `stop` reporting a backlog lost before it was called. | No row is present twice and no present row was never attempted; the crash gate reports the acknowledged rows lost per cycle. |
| A manifest watermark never names an entry that is not durable | Durability barrier tests: a flush held until its entry is durable; a crash in `enqueued` mode after an acknowledgement and a flush attempt but before the object exists, followed by a restart, new writes and a second crash; a drop cancelling a flush that waits; a truncate that waits; a flush after a backlog was lost. | Every durable entry replays; the flush after the lost backlog publishes no frontier; `obsolete` never passes the durable id. |
| A region replays only its own entries | The two-region recovery test above. | Per-region row-level scan equality with the other region unflushed. |
| Recovery never silently loses data | Unit coverage of every corruption class at recovery and on read; footer-only recovery compared with a full decode; a footer longer than the tail window. | Structural damage fails construction of the store; a corrupted segment is skipped, recorded as a hole of its region and counted, or fails the read under `fail`; the footer-only catalog equals the full decode. |
| A region cannot be silently pointed at another WAL | Reopen on a mismatched prefix; reopen on an empty prefix. | The mismatch is rejected with both prefixes named and the region is not opened; the empty prefix opens cleanly with the expected row counts. |
| Garbage collection deletes everything below the persisted reclaim boundary, and recovery stays correct while objects disappear | Unit tests of the boundary computation (a region without a watermark holds it at its first object, a watermark inside an object keeps the object, `obsolete_all` releases it, last indexed plus one when everything is flushed), of stamping and durability (headers carry the boundary current at sealing, a failed create's boundary permits nothing, the start object carries the recovered boundary), of deletion (chain objects, listed orphans and failed batches below the durable boundary, nothing at or above it, a failed delete retried, `stop` waiting for deletes in flight, holes dropped), of recovery (two restarts with a collection between them, a late object of an earlier epoch below the boundary collected, a missing predecessor above the boundary still failing) and of a read that listed an object before it was collected; the Mito flush-then-collect test in both acknowledgement modes; the standalone integration test's assertion after a flush. | Objects below the durable boundary are gone after a flush, the start object included; a restart replays everything above the boundary; exactly one WAL object remains after a flush and one more insert in the integration test. |
| Every row of the *Failure matrix* holds, and every metric moves as documented | A table-driven fault matrix, `test_store_fault_matrix`, that drives every failure the object store access can produce through the store and asserts, per case, which waiters fail and with what error, whether the store is poisoned, whether the sequence is consumed, the indexed gauges at the failure itself, the exact movement of every metric, and what a read of the store sees afterwards; every timer is observed twice around two forced waits whose intervals cannot overlap. | 33 cases: 17 for the ten rows of the matrix and 16 for the failures listed under *Failures driven through the store*. The test fails if a row of the matrix is left uncovered. |
| Recovery cost does not grow with object size | Footer-only recovery over 2,000 objects; recovery concurrency bounded and ordered by sequence. | Construction of the store fell from 3.6 s to 0.9 s. |
### Failures driven through the store
- Conditional create outcomes: success, identical retry, conflicting content from an earlier epoch and from the same or a later one, lost response, transient failure in `durable` mode with later objects already created and with creates still in flight (every batch that is not indexed fails, no sequence is reused, late completions are orphans), transient failure in `enqueued` mode and after `stop` began, completion after a conflict had already poisoned the store, a create at the last representable sequence.
- Object store access: an unwritable store; a listing, a trailer fetch, a footer fetch, a segment fetch and a delete that fail; a delete that fails and is retried by the next collection; an object collected between the listing and its fetch, below and at the tip's reclaim boundary; an object store that answers slowly.
- Store state: an object the catalog rejects, per-region entry id exhaustion, the catalog refusing a next sequence past the maximum, every stop path with several creates in flight that succeed, fail transiently or conflict after stop began, a stop with a delete in flight, and the appends the backlog thresholds were holding back released by admission, by a stop, by a terminal error and by the teardown of the actor.
- Pipelined uploader: concurrent creates acknowledged in sequence order, every sealed batch linked to its pipeline predecessor, the open batch sealing at the threshold while creates are in flight, a transient failure failing every batch that is not indexed, created ones included, a permanent failure poisoning the store.
- Chain recovery: a linear chain, orphans below and above the tip, a broken tip falling back to the latest complete object, a late object of an earlier epoch next to the chain of a later one, a missing predecessor, objects present without a complete chain, orphans raising the next sequence, and the start object on an empty prefix and meeting objects of an earlier and of a later epoch.
### What the evidence does not cover
- The cloud measurement covers one instance type, one region and one bucket, with the Raft Engine control on the instance's local NVMe store; no crash gate ran on the cloud host, and the request cost is the list price, not a bill.
- Loss during writes is measured only by the crash gate, in single samples. The Raft Engine comparison below killed the server after every row was counted, so its replay checks confirm completeness and say nothing about loss.
- The store's own work, encoding, indexing and the actor, is not quantified separately from the object store round trip.
- A create whose response and immediate read-back are both lost is covered only at recovery, by placing such objects next to a chain; no test loses a response and its read-back at the object store.
- The benefit of the pipelined uploader was measured only with injected latency on a local object store.
### Write path measurements
A log workload benchmark was run against the write path on a laptop with a local MinIO server. The numbers calibrate the defaults; the same workload on S3 is reported under *Cloud write path measurements* below. The profile writes 2,000,000 rows of about 900 bytes in 1000-row batches into one region at client concurrency 1, 8 and 32. Raft Engine with `sync_write = true` on the same build and the same storage is the control. Rows per second:
| Write path | 1 | 8 | 32 |
| --- | --- | --- | --- |
| Raft Engine, `sync_write = true` | 3,500 to 4,000 | 19,000 to 25,000 | 20,000 to 28,000 |
| one create in flight, `durable`, 1 s | 998 | 4,001 | 19,029 |
| one create in flight, `durable`, 100 ms | 3,287 | 13,662 | 17,204 |
| one create in flight, `durable`, 10 ms | 3,632 | 19,835 | 19,309 |
| pipelined uploader, `durable`, 1 s | 1,000 | 3,999 | 22,043 |
| pipelined uploader, `durable`, 100 ms | 3,304 | 17,207 | 23,185 |
| pipelined uploader, `enqueued`, 1 s | 3,955 | 24,074 | 22,168 |
| pipelined uploader, `enqueued`, 100 ms | 3,999 | 23,595 | 22,662 |
The rows come from separate runs, and the same build with the same configuration, on an idle host each time, measured 1.4 times apart in runs days apart, so a difference between two rows of the table is not a measurement of the difference between those write paths. The Raft Engine range covers several runs. Both write paths converge on the host's CPU ceiling at concurrency 32, 6 to 7.5 of 10 cores busy with one region, so the last column measures the host rather than the WAL. A serial writer under the `durable` mode is phase-locked to the seal timer: its extra latency over Raft Engine is the interval minus the base insert cost modulo the interval, which is 747 ms at 1 s, 52 ms at 100 ms, and a floor of about 14 ms, one seal plus one PUT, at 10 ms. The pipelined uploader alone was measured by counterbalanced A/B runs of the builds before and after it, run in both orders within one session: at 1 s and concurrency 32 the two builds were within about 1.4% and the two pairs disagreed on which was faster, and at 100 ms and concurrency 8, where both builds sealed about ten objects per second, they were 0.25% apart on the geometric mean of two pairs, while whichever build ran first in a pair was 3.5% to 4% faster, and their object counts and sizes were within 2%. No throughput benefit was measured at either point. Every restart under the `durable` mode replayed exactly the acknowledged rows. In the `enqueued` mode a SIGKILL lost one to two 1000-row batches at 1 s and none at 100 ms, in single samples, always whole batches and never a duplicate. Footer-only recovery cut log store construction from 3.6 s to 0.9 s at 2,000 objects.
The conditions under which the pipelined uploader does help were then examined, again comparing the builds before and after it within one session in both orders at concurrency 8 and 100 ms, with a proxy in front of the object store that delays every request of the WAL while the data files keep a direct connection. In the build before it the actor awaits each create before it admits more entries, and Mito's region worker in turn waits for one append to become durable before it sends the next, so what the uploader can run side by side depends on how many appends wait at once, not only on how slow a create is. With one region, adding about 40 ms or about 160 ms to each create left the two builds within the order effect of these runs, since one region never has a second sealed batch while the first uploads. With a table split into three regions, each served by a different region worker, and about 160 ms added, the pipelined uploader was faster by 13.6% and 2.1%, 7.7% on the geometric mean, still within that margin, and it alone let the regions fall out of step: 27% to 46% of its sampled objects held fewer than three regions, against 0% to 4% before it, because a region whose append became durable while another create was still in flight started the next batch. The clients of those runs wait for each response, which ties every region back to the seal. With the three regions driven at a fixed request rate of 0.7 times that closed-loop throughput instead, the pipelined uploader lowered the median insert latency by 24.0% and 12.2% in the two pairs, 18.3% on the geometric mean, and the 99th percentile in both pairs; at 1.0 times the rate one valid pair showed 23.8%, while the second pair was not completed because the load generator repeatedly fell behind its send schedule. The lower latency is paid for in requests: with about 160 ms added, the pipelined uploader created 19% to 28% more objects with one region, 51% to 67% more with three regions in the closed loop, and 35% to 63% more at 0.7 times the rate in the open loop. These measurements used a local object store with added latency and do not measure a cloud object store.
The comparison with Raft Engine was measured again within one session, with one binary and only the `[wal]` section differing, each comparison run in both orders to reduce the bias of running first or second. Runs on this host still varied by 4% to 8% with their position alone, which limits how small a difference these runs can resolve. Both runs of a pair killed the server only after every row was written and counted, so the replay checks confirm a complete replay and say nothing of what a crash during writes loses. Rows per second:
| Write path | Concurrency 1 | Concurrency 8 |
| --- | --- | --- |
| Raft Engine, `sync_write = true` | 3,817 and 4,503 | 25,124 to 26,880, four runs |
| object store WAL, `durable`, 100 ms | 3,184 and 3,198 | 17,779 to 19,174, four runs |
| object store WAL slower by, geometric mean of the pairs | about 23% | about 28% |
The gap does not close under concurrency; it is slightly wider at 8 than at 1. At concurrency 1 the object store WAL measured the same in both runs, 301 ms at the median, while Raft Engine moved between a median of 237 ms and 221 ms, both of which round up to the same 300 ms seal, so the single-writer figure depends on how fast Raft Engine happens to be. Raft Engine exceeded the upper end of the range in the table above in every one of the four runs at concurrency 8.
Two further configurations of the same binary, run in both orders in two rounds at concurrency 8, locate the gap. With the WAL objects written to local disk through a file object store instead of MinIO, still in the `durable` mode, MinIO was 4.5% slower. Switching the MinIO configuration to the `enqueued` mode was 35% faster, and the `enqueued` configuration was 4.4% slower than Raft Engine. The first and third of these differences are within the run-to-run variation of the host, while the second is far larger, which points to the `durable` mode waiting for the seal and the create before it acknowledges as the main contributor to the gap in this workload. These are differences between whole configurations: they do not measure the MinIO round trip or the store's own work, encoding, indexing and the actor, individually, and the runs leave those costs unquantified. The file object store is not equivalent to a remote one: its conditional create checks, writes a temporary file and renames it rather than creating atomically, and it does not sync the directory after the rename. The `enqueued` mode does not provide the durability of Raft Engine with `sync_write = true`, so this comparison separates costs and is not a comparison of equivalent write paths; against Raft Engine with its default `sync_write = false`, also run in both orders within one session, the `enqueued` mode was 5.1% slower, again within the noise. These local runs do not measure the latency or throughput of a cloud object store.
The default interval is 100 ms because it removes most of the seal-timer latency of a sparse `durable` writer, 52 ms of residual wait against 747 ms at 1 s, at a request cost that stays bounded: timer-triggered sealing creates at most one object per interval per node, ten per second at the default, the `enqueued` mode can seal earlier only when the unpersisted backlog reaches a threshold, and the batch seals on size only when the admitted bytes reach `max_batch_bytes` before the next timer tick, which the measured S3 workload did not reach at about 60 MB/s: every object store cell sealed on the timer. Going to 10 ms buys another 40 ms for ten times the request ceiling. On S3 at the default configuration the seal timer, not the PUT, bounded a serial `durable` writer to one batch per interval, see *Cloud write path measurements*; a shorter interval removes part of that wait but not the create's latency, and was not measured on S3. The measured way to Raft Engine's throughput from one writer is the `enqueued` mode, at the cost of its loss bound. The default acknowledgement mode stays `durable` because that is what a write-ahead log means: an acknowledged write survives a crash. `enqueued` is an explicit choice with a documented loss bound, the unpersisted backlog, and the crash gate reports what it loses.
### Cloud write path measurements
The same workload was run against S3 from an EC2 instance in the bucket's region (`c6id.2xlarge`, 8 vCPU, `ap-southeast-1`, the nightly image of `main` of 2026-10-12) by the workflow of pull request #9536 (https://github.com/GreptimeTeam/greptimedb/pull/9536), which starts GreptimeDB with its data files on S3 and differs between cells only in the `[wal]` section: Raft Engine with `sync_write = true` on the instance's local NVMe store, and the object store WAL in the `durable` and in the `enqueued` mode, both with the defaults. 2,000,000 rows of about 900 bytes in 1000-row batches at concurrency 1 and 8, two rounds with the order of the configurations reversed between them, every cell on a fresh container and a fresh prefix. Rows per second, geometric mean of the two rounds of run 38232084219 (https://github.com/GreptimeTeam/greptimedb/actions/runs/38232084219):
| Write path | 1 | 8 |
| --- | --- | --- |
| Raft Engine, `sync_write = true`, local NVMe | 31,700 | 67,800 |
| object store WAL, `durable`, 100 ms | 9,670 | 36,900 |
| object store WAL, `enqueued`, 100 ms | 32,900 | 67,900 |
A second run of one round at concurrency 8, 38234710697 (https://github.com/GreptimeTeam/greptimedb/actions/runs/38234710697), reproduced the ratios within 1%: the `durable` mode at 0.54 of Raft Engine, the `enqueued` mode at 1.00. The latencies, object counts and histograms below are read from the metrics scraped after every cell, kept in the runs' artifacts.
A conditional create on S3 took about 50 ms at the median and 110 to 130 ms at the 99th percentile from seal to durable, against single-digit milliseconds on the local server, and still a serial `durable` writer is bound by the seal timer, not by S3: every 1000-row batch took about 103 ms, one object per batch, with its acknowledgement at 91 ms at the median, so at the default configuration its throughput was the batch size over the interval, about ten batches per second: the measured 103 ms per batch is one interval plus the admission of the next batch, since the create, the acknowledgement and the next append all complete before the following tick. No shorter interval was measured on S3. Raft Engine on the instance store took 32 ms per batch, hence 0.31. The local ratio of 0.77 was higher only because the local Raft Engine batch took over 220 ms, which rounds up to the next seal with little loss: the faster the control, the lower the ratio at low concurrency under the default interval. At concurrency 8 the `durable` mode still seals on the timer, about four batches per object, and reaches 0.54. The `enqueued` mode matched Raft Engine at both concurrencies and never stalled: at about 60 MB/s of WAL the backlog thresholds were not approached, since one create per interval of 3.5 to 8 MiB drains it. Under sustained load the timer ceiling sets the request count: 550 to 600 creates per minute per node in every cell, which at the S3 standard list price of $0.005 per thousand PUTs is about $0.18 per node-hour of continuous writing, plus about 30 deletes per cell for garbage collection. No create failed, found a conflicting object or poisoned the store in 7,609 creates, the 6,809 of the first run's eight object store cells and the 800 of the second run's two.
# Alternatives
## A generic external provider injected from outside the repository
The backend could stay outside the main crates and be injected through an opaque provider variant plus a resolution hook on the `LogStore` trait. That avoids touching the four enums, but every resolution path then has to special-case an opaque value, distributed rejection can only be enforced by convention, and nothing in the repository can test the backend. Making the provider an explicit variant keeps every match exhaustive and lets the engine tests run against the real store, at the cost of one more arm in each enum.
## Reusing the Kafka provider with an object store "topic"
The Kafka code path assumes a broker assigns offsets and that a topic is shared across datanodes. Neither holds here: entry ids are assigned by the datanode, and one prefix belongs to one datanode. Reusing the Kafka provider would have inherited the topic tickers, the failover preconditions and the metasrv allocation logic, all of which are wrong for this design.
## One object per region per flush
Simpler catalog, but every region then produces its own timer-driven object, so the request rate scales with the number of regions even when they write little, which is the cost trap object stores punish. Sharing one batch across regions removes that per-region floor: at low volume timer-triggered sealing creates one object per interval regardless of region count, with the `enqueued` backlog thresholds able to seal earlier, and at high volume the count is governed by aggregate bytes and `max_batch_bytes`, which is the same request volume a single busy region would produce.
# Drawbacks
- Write latency under the `durable` mode is the time until the batch seals (up to `flush_interval`, 100 ms by default), plus waiting for every earlier object, plus one PUT. A serial writer waits for the next seal, so its extra latency over Raft Engine is the interval minus the base insert cost modulo the interval, and concurrency does not close the gap: measured within one session on a laptop it was about 23% of throughput at concurrency 1 and about 28% at concurrency 8, and on S3 against Raft Engine on a local NVMe store the `durable` mode reached 31% of Raft Engine's throughput at concurrency 1 and 54% at concurrency 8, a serial writer being bound to one batch per interval, while the `enqueued` mode matched it, see *Write path measurements* and *Cloud write path measurements*.
- Every object costs a request. Timer-triggered sealing creates at most one object per interval per node, ten per second at the default; the `enqueued` mode can seal earlier when the unpersisted backlog reaches a threshold, and the batch seals on size only when the admitted bytes reach `max_batch_bytes` before the next tick. Under sustained load on S3 the timer ceiling was reached, about 600 creates per minute per node, about $0.18 per node-hour at the S3 standard list price.
- Garbage collection keeps every object at or above the reclaim boundary, so a region that is written but rarely flushed, or never flushed, holds the boundary at its first object and keeps every later object the prefix shares with other regions; recovery time grows with the number of retained objects. A prefix that receives no appends cannot persist a higher boundary, so an idle prefix is collected only after its next object.
- Every open of the store writes one start object without segments, one extra PUT per open, and every object carries a 54-byte header. Orphans, the objects of failed batches that were stored anyway, hold storage together with the rows Mito wrote again after those failures until garbage collection deletes them.
- Object-sequence-major entry ids consume 20 bits per object for positions, which caps a region at about a million entries per object; the batch seals when the cap is reached. Sequence numbers are 44 bits wide as a result, enough for one object per second for half a million years.
# Alignment with cluster mode
Distributed mode is out of scope, but several decisions were taken so that the cluster design can build on this backend without a metadata migration. Except where a paragraph says otherwise, all of them are *Proposed*.
**Key layout.** The layout is implemented in its standalone form, see *Object format*: the operator configures only the prefix root, the store derives the prefix it writes under as `<prefix>/datanodes/<node_id>/epochs/<generation>`, so objects live at `<prefix>/datanodes/<node_id>/epochs/<generation>/objects/<sequence>.wal`, standalone uses its datanode id and generation zero, and the persisted `WalOptions` of a region carry the full derived prefix, so a region's WAL location is self-describing. *Proposed*: in cluster mode the metasrv assigns both the node id and the generation, and issues a new generation every time it hands a region's write ownership to a node; the self-describing location then lets a node other than the writer open a region's WAL read-only, which today is rejected because a datanode builds its store only under its own derived prefix and the store refuses providers with another prefix. Putting the generation into the key rather than only into the object header means that a writer which lost its lease and keeps writing lands in its old epoch directory, where the new writer never creates objects, so the two can never contend for a sequence number; the generation in the key is the fence. The version 1 header carries no generation; adding one as a read-side check would be a format evolution, a header version bump whose reader keeps decoding version 1 objects, and is not part of this proposal. The header does carry an epoch, which is implemented: the epoch orders the store instances that wrote under one prefix, while the generation separates prefixes, so the two are orthogonal. The generation could later seed the epoch, but the epoch does not depend on it. Despite the name, the `epochs/<generation>` component of the key holds the generation, not the header epoch.
**Planned migration.** A region that moves while its source node is alive follows the existing migration flow, and the cheap path applies only when that flow proves the handoff: the source stops admitting writes, flushes, and the resulting manifest watermark covers the last acknowledged entry. Then the target opens the region from that watermark and starts writing under its own prefix, and no cross-node read is needed. If the final flush fails or times out, the fact that the source process is alive proves nothing about its unflushed tail; the migration then either takes the failover path below, with the source's prefix in the chain, or is aborted.
**Failover.** When the source is gone, the target must replay the region's unflushed tail from the source's prefix. The region's WAL position becomes a chain of segments whose last element is the target's own prefix. Each earlier segment is `(prefix, first sequence, cutover sequence, holes)`: the metasrv appends it to the region's `WalOptions` when it reassigns the region, after the source's lease has expired, and it selects the history under the source prefix at that moment by the rule the store applies at open: the complete object with the largest epoch and, among those, the largest sequence, and every object it links to. It records that chain as the segment's replay history, the cutover sequence and the sequences of the chain's objects, and the record is sealed: a present object outside the chain, whether the orphan of a create the source reported as failed or a stale object of an earlier epoch, is excluded, an object that becomes visible later, because the old writer's upload was still in flight, stays excluded, and an object above the cutover is ignored, so every open of the region replays the same source history. On open the target replays the earlier segments in order, reading only footers and its own region's segments through range reads, and never creates an object under a foreign prefix, so there is nothing to conflict with.
Sealing may only exclude objects whose writes were never acknowledged, which takes two fences. Admission is fenced by the lease mechanism Mito already has: when a region's lease expires the datanode demotes it to a follower, which refuses writes, and the metasrv waits for that deadline before it upgrades the candidate. Completion is fenced by the store: every conditional create runs under an upload deadline, a create that has not completed by its deadline is reported failed to its waiters and is never acknowledged afterwards even if the object turns out to exist, and when a region's lease expires the store fails every waiter of that region that is not yet acknowledged, so that a create still in flight at that moment cannot be acknowledged later. The metasrv seals the segment only after the lease deadline plus the upload deadline have both passed, so an object that appears after sealing belongs to a create the old writer had already reported as failed. Under the durable acknowledgement mode this means a sealed set never excludes an acknowledged write; under the `enqueued` mode the entries in the old writer's unpersisted backlog at that moment are lost, which is that mode's stated recovery point objective. The late-completing create is part of the acceptance criteria for takeover, stated in terms of the sealing instant: an object that was absent at sealing, whether it fills a hole later or lies above the cutover, is excluded on every open of the region, and so is a visible object outside the selected chain; an object on the sealed chain replays on every open even if its write was never acknowledged, which is the same outcome as a crash between an object's creation and its acknowledgement; and in the durable mode every caller whose create had not completed when the lease expired must have been failed, so no acknowledged write is among the excluded objects. The generation in the key is an additional guarantee that such a late object lands in a directory the new writer never reads beyond the sealed set. Object-sequence-major entry ids are what make this cheap: `entry_id >> 20` locates the object under the right prefix without the source node's in-memory catalog.
**Entry id floor across prefixes.** A region that arrives under a new prefix, by migration or by failover, brings a watermark and replayed ids that were assigned under another prefix's sequence. Before the target assigns the region its first new id, the target store raises its next object sequence above `max_id >> 20`, where `max_id` is the largest of the region's inherited watermark and every id it replayed, so that every new id of that region is greater than every id it already has; sequence gaps are tolerated and the sequence space is wide enough for this. The rule is applied on every open from the region's persisted state, in the same place Mito already hands the store the manifest watermark, so it needs no persisted state of its own and survives a crash before the target's first flush: the restart re-derives the floor and the region's acknowledged-but-unflushed entries under the target prefix are above the inherited watermark and replay.
**Acknowledgement modes.** Both are implemented. Cluster mode is expected to run `enqueued` for latency-sensitive workloads; the object layout, the flush durability barrier and the recovery path are identical in both modes.
**Corruption.** Segment-level skipping with the hole recorded per region is implemented, see *Object format*. *Proposed*: cluster mode routes the WAL hole event to the metasrv so that a follower or a takeover knows the region needs a fresh replica rather than a replay.
**Garbage collection across nodes.** Within one prefix, garbage collection is implemented as described under *Entry ids*: watermark from each region's manifest, per-segment footer comparison, a reclaim boundary persisted in every header. A source prefix that still holds segments of regions which have moved away cannot be trimmed by the source alone, because it no longer learns those regions' watermarks. Either the target reports to the metasrv, after each flush, the source sequence below which it no longer needs the region, and the metasrv drives deletion, or a cluster-level janitor collects the watermarks; which of the two is the last of the *Unresolved questions*.
# Future work
1. Probe the next sequence with a GET from the last known sequence instead of listing the whole prefix at recovery; on a prefix with many objects the probe is cheaper than the listing.
2. Distributed mode, which needs metasrv-side allocation of node ids and generations, the takeover chain in `WalOptions`, read-only replay of a foreign prefix, and cross-node garbage collection, as sketched under *Alignment with cluster mode*.
# Unresolved questions
- Whether concurrent writers on one prefix need a fence beyond the epoch conflict. Conditional creates already prevent overwrites, and a concurrent writer is noticed when its epoch meets another at a sequence, which can be after both acknowledged writes.
- Whether garbage collection of a source prefix after a failover is driven by the target reporting its watermark to the metasrv, or by a cluster-level janitor that collects watermarks; the same question covers regions that are closed on a node but still own objects under its prefix.