Commit Graph
188 Commits
Author SHA1 Message Date
Weny Xu beb1f88696 feat: persist bounded WAL releases on region close (#10329)
* feat(log-store): persist bounded region releases in shared WAL batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): release safe WAL history when closing regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover bounded WAL release across close and restart

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): clamp bounded releases to durable entries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(mito): observe close release failures and cover recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: describe bounded WAL release lifecycle

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): make bounded release fence sealing deterministic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): bound queued releases and ignore shutdown hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): cover close release persistence and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): wait for failed close flush cleanup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify bounded WAL release backpressure and shutdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(mito2): clarify staging test retry invariant

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): back off durable release retries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): drive release retry after backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): share and verify WAL retry backoff

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: clarify retained WAL release retry semantics

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-13 12:44:46 +00:00
jeremyhi 6faa9ec862 feat(mito2): replay pending WAL tails in the background (#10342)
* feat(log-store): read the WAL of another node through the log store

`LogStore::read_node` streams one region's entries in `(after, upper]` from
the WAL of another node that shares the log store's storage, without writing
to it. The default returns `None`.

The object store WAL implements it with a registry of read-only readers over
its own object store. The node prefix comes from the configured root and the
store's own generation, and the returned stream holds the reader, so reads of
the same node share its cached object indexes. `WalReadUpperBound` moves to
store-api so the trait can name it; log-store re-exports it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): replay pending WAL tails in the background

A leader that is writable, altering or editing and still has pending tails
replays them in one background task per region and publishes the result in
one manifest edit, which opens the read gate and resumes compaction once no
tail remains.

- The worker starts the task when a lease, a catchup or a role change makes
  the region writable, and cancels it on close, drop, downgrade and when it
  replaces the instance. The task always reports its end, so the worker can
  start another replay or reload a region a conflict fenced.
- Each tail is read from this node's WAL, or through `LogStore::read_node`
  for another node, and replayed with the per-entry code of open-time replay
  into memtables outside the region's version. Chunks of about 64 MiB are
  written with the SST half of the flush task, never across two tails, since
  an SST gives its rows its largest sequence.
- The edit adds the SSTs, clears the tails, and advances the flushed entry id
  to the last entry read and the flushed sequence to the largest chunk
  sequence. It commits outside the edit queue, under the manifest lock, only
  while the region is a writable leader recording exactly the replayed
  tails. A busy region or an unknown result retries the same bytes; a
  conflict keeps the chunks; a certain failure deletes them.
- Another node's listed tail is only a bound. After a catchup made the
  region writable, the edit keeps its unread part until the region receives
  a Leader lease, then a second edit clears it.
- After the commit the task releases the WAL up to the new replay start,
  kept below this node's remaining tails, and the worker raises the region's
  last entry id to the new flushed entry id, so a later truncate cannot pull
  it back.
- WAL pressure no longer flushes a region whose WAL release one of this
  node's tails caps.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): harden tail replay across leases, cancellation and uncertain commits

- A Leader lease counts as received after a catchup only if its heartbeat
  was sent after the catchup promotion: the writable fast path does not
  check the start of a downgrade, so a lease granted before an earlier
  downgrade could otherwise clear the mark. The mark is the promotion time,
  so the fast path still reads one atomic.
- A direct promotion counts as a lease.
- The tail commit also raises the committed sequence, so later writes of the
  region take sequences above the replayed rows.
- Registering a writable region starts a replay, which covers a create that
  reopens an existing region.
- Once an attempt may have persisted the edit, later certain failures retry
  the same content instead of building a new edit.
- WAL reads stop when the task is cancelled, and a cancelled task does not
  start another commit attempt.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that closing a region cancels a stalled tail read

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): hold tail commit attempts by number instead of sleeping

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): cancel tail replays when the worker stops

The worker's cleanup closes its remaining regions without the close handler,
so it now cancels every running tail replay first; otherwise a replay stalled
in a WAL read or waiting for a lease outlives the engine. The stalled-read
test covers an engine stop as well as a close. The docs no longer claim that
every SST gives all its rows its largest sequence.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): shorten two tail replay comments

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): stop tail reads at their upper bound

A tail read over this node's WAL kept reading past the tail's upto until the
log ended, so an unreadable later entry failed a replay that already had every
entry it needed. Stop the read once it reaches upto.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): inline the tail replay notify helper

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that writes during a tail replay win over replayed rows

Whatever records pending tails must raise the region's committed sequence to
the tails' largest sequence, so writes accepted while the replay runs get
larger sequences than the replayed rows. Record this in AGENTS.md and test it
with a write that lands between two commit attempts of the replay.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): bound reads of this node's listed tails

A listed bound may name no entry, so a read of this node's WAL never saw an
entry equal to it and kept polling later objects; one that failed to read
failed the tail. Read every listed tail through `read_node`, which stops at
the bound, and keep reported tails on the node's own WAL.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that rows around a tail replay survive a reopen

The tail commit raises the flushed entry id to the largest entry read, which
can be another node's. Whatever records tails keeps this node's later entry
ids above every tail and leaves none of its unflushed entries below one
outside its own tails. Record this in AGENTS.md, and make the replay test
meet it and reopen the region without a flush afterwards.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject graceful staging entry of a region with pending tails

Entering staging through `set_region_role_state_gracefully` skipped the
pending-tail check that `EnterStaging` runs, so a region with pending tails
could enter staging, end its replay, and return to Leader with nothing
replaying its tails. Run the same check before entering staging there.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): replay tails of a region altered right after promotion

A promotion checked whether it made the region writable after setting the role. If the
worker started an allowed ALTER or edit in between, the check saw Altering or Editing
and asked for no replay, and nothing asked for one later. Check for a leader that a
replay runs on instead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover the WAL release and locked checks of tail commits

Check that a tail commit releases the replayed WAL, that it is refused under the manifest
lock once the region was demoted or its tails changed, that a commit retried while
altering keeps its chunks, and that a replay waiting for a lease does not end and start
again.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): note that read_node also reads this node's WAL

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): keep no region version during a tail replay round

A replay round held the region's version until it ended, and with it the memtables the
region had when the round started. A flush removed them from newer versions, but their
write buffer memory was only freed once the round dropped them, so a slow read, a commit
retry or a lease wait could keep flushed memory charged and stall writes. Copy the tails
and the flushed entry id instead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): count a direct Leader role set as a lease only when it promotes

A direct role set to Leader counted as a Leader lease even when the region already was a
leader. On a region a catchup promoted, it cleared the mark that keeps another node's
unread listed tail until a lease arrives, so the next commit could drop that tail.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): move the tail replay leader check onto MitoRegion

The engine and the replay task both use it, next to the other role checks. Its doc now
names the Writable, Altering and Editing states and why staging and downgrading
regions replay nothing.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay waits for a Leader lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin the pressure skip, staging rejection and remote read error

The WAL pressure test gets a case whose own tail starts above the flushed entry id and
still flushes; the graceful staging case checks the rejection; and a remote tail on a log
store that reads no other node's WAL fails as unsupported and keeps its tails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): list every tail replay trigger and the missing admission limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): rename tail replay helpers and notifications

The names now say what each one checks, records or announces: the
replay predicate is a capability rather than a running replay, the
lease hook records a lease, the writable notice is an event like its
siblings, and the commit notice pairs with the end notice.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): ask whether a Leader lease came after a catchup

`is_writable_by_catchup` was negated at both of its call sites. Its
inverse, `leader_lease_received`, is what the replay reads.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): plan the read of a pending tail in the tail replay

`Wal` no longer interprets `PendingTail` or `UptoKind`. The replay task
decides whether a tail comes from this node's own WAL or up to its bound
through `Wal::read_node`, which now takes plain arguments.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay result goes stale

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): send the writable notice through notify_worker

The tail replay already has one helper that sends a background notice
to a region's worker; the writable notice now uses it too, and so logs
a worker that has stopped.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): note that tail replays skip staging regions and index like flushes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): say why each tail gets its own SSTs and what chunks are for

Ordering between tails needs a tail per SST: an SST can give all its
rows its largest sequence, so an earlier tail sharing an SST with a later
one would take the later tail's sequence and outrank rows that a leader
between the two tails flushed. Within a tail, the 64 MiB chunks only
bound memory: the sequence floor already puts writes accepted during the
replay above every tail.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the tail replay end guard like other drop guards

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): skip a tail replay edit that a sync already superseded

A commit notice that the worker handles after a sync rebuilt the region's version from a later manifest no longer brings back tails or files that manifest removed. The notice carries the tails the replay read, and the worker applies the edit only while the version still records them.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log when a tail replay stops before publishing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover lock waits, busy edits and repeated triggers in tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log every exit that drops a tail replay

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): tell the event listener when a tail replay checks for a lease

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover uncertain commits, the SST hook and lease waits of tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore(mito2): log each tail replay exit once, where it happens

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): reuse the mock region hook and wait for role change tasks in tail replay tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): cover failed-round cleanup, instance replacement and leased first rounds of tail replays

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): make the replacement helper a TestRegion method and fix a lease comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-13 06:32:03 +00:00
jeremyhi 6a202adbe7 fix(log-store): skip objects collected below the boundary in WAL reads (#10343)
* fix(log-store): skip objects collected below the boundary in WAL reads

The read-only WAL reader lists the source prefix and then reads the
header and footer of every listed object. While the store writing the
prefix is still collecting objects, a listed object can be deleted
before its footer is read. The NotFound failed the whole read, even when
the object lay below the reclaim boundary and was off the chain the read
replays, so reads of a large prefix under steady writes failed often.

Apply the rule startup recovery already uses. An object gone below the
reclaim boundary at which the chain of the read stops is skipped. Any
other vanished object, on the chain or off it, still fails the read: it
may be the object that completes a later chain or carries the largest
epoch. Startup recovery and the reader now share one NotFound split and
one boundary check.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): cover unreadable objects in listed WAL reads

A NotFound below the boundary is skipped, but any other error reading a
header or footer must still fail the read. Cover it with a corrupted
header below the boundary in both modes.

Describe the boundary in `ensure_vanished_below` and `read` in terms
that also fit the `LastEntry` mode, where the chain can stop at the
object of `after` instead of a reclaim boundary.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe where WAL reads stop following links

In the `LastEntry` mode a later object can carry a reclaim boundary
above the object of the reported entry. The read then keeps that object
as its tip below the boundary and follows no link from it. Say that the
boundary stops the links to predecessors, and that the tip is the one
object below it a read can need.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): pin the outcomes of vanished-object reader tests

The unreadable-object test now asserts the InvalidWalObject error for
object 1 inside the index cache's wrapper, instead of any error other
than NotFound. The vanished chain-object test states the outcome it
guards against: without object 2 a listed read could select the chain
ending at object 1 and succeed with no entries.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-12 08:02:29 +00:00
Silvio 6dbff8a081 fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown

raft-engine preallocates the active log file and only trims that tail when the
Engine is dropped. GreptimeDB never dropped its engines before exiting, so every
start after a clean shutdown truncated the tail as a broken batch and logged
"Truncating log file due to broken batch".

RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine;
standalone closes the metadata backend last, and the datanode stops the
raft-engine WAL after the region server.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: address review on raft-engine shutdown

- Add ClosableKvBackend to common-meta; the standalone metadata backend creator
  returns one and Instance closes it explicitly instead of downcasting to
  RaftEngineBackend, so wrappers forward the close.
- Record a failed metadata backend close as a shutdown error.
- Sync and drop the engine on a blocking thread.
- Only check tail trimming when raft-engine preallocated, and check reopening
  after close/stop on every platform.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: release raft-engine engines on every shutdown path

- Stop the raft-engine WAL when the datanode build fails after opening it.
- Purge and sync tasks hold the engine weakly, so dropping the last backend or
  log store reference (as the CLI tools do) closes the engine.
- Fix needless borrows reported by clippy.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: drop unused ClosableKvBackend import

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: cancel log store readers on stop; address review

- RaftEngineLogStore tracks the tasks feeding read streams and cancels and
  joins them on stop, so a retained, unconsumed stream no longer keeps the
  engine open past shutdown.
- Tests: stop with a retained read stream; datanode shutdown with a failing
  region engine and after a normal shutdown reopens the raft-engine WAL;
  standalone stop with failing steps reopens the metadata directory.
- entries_to_batch no longer takes self; rename opened to get_open_engine,
  log_files_len to log_files_size and PREALLOCATED_LEN to
  PREALLOCATION_THRESHOLD; mod declarations before imports; rename the
  standalone shutdown closure to record_shutdown_error.

Model: claude-opus-5-5
Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: update Cargo.lock for tokio-util's rt feature

The rt feature makes tokio-util depend on futures-util, which the lock file
has to record for `cargo check --locked`.

Model: claude-opus-5-5
Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: close the raft-engine WAL by ownership instead of a lock

Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again:

- MitoEngine drops its Wal raw entry reader once its workers are stopped,
  so it no longer holds the log store after stop.
- RaftEngineLogStore::close(self) stops the tasks and readers and closes the
  engine. Datanode::shutdown takes the log store after stopping the region
  server and closes it if it holds the last reference; otherwise it logs a
  warning and only stops it.
- A failed build stops the region server it created before closing the log
  store, so the engines release it there too.

Model: claude-opus-5-5
Effort: medium
Tokens: 78, 19161, 107293, 4916570 del commit
Cost: 2.23 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* test: drop the index argument from LogStore::read in the raft-engine tests

#9563 removed it from the trait.

Model: claude-opus-5-5
Effort: medium
Tokens: 16, 3715, 6600, 1706794 del commit
Cost: 0.47 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: end a cancelled WAL read with an error; test a written region

- A raft-engine read cancelled by stop now ends its stream with an
  IllegalState error instead of an end of stream, so a replay cannot take a
  truncated WAL as fully read. The reader flags the cancel instead of sending
  the error, since it is usually blocked on a full channel at that point.
- Datanode test: a raft-engine datanode with a created region and writes
  shuts down cleanly and its WAL reopens, which fails if anything still
  holds the log store.
- Move the doc comment of is_stopped back onto it.

Model: claude-opus-5-5
Effort: medium
Tokens: 42, 10316, 407027, 4108529 del commit
Cost: 4.28 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix(log-store): drop the no-op sync before closing the raft-engine engine

Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and
`Engine::write` returns before the fsync, so the sync in `close_engine` read
as a final flush that never ran. The active log file is trimmed and synced
by `LogFileWriter::close` when the last reference is dropped, so
`close_engine` now only drops it on a blocking thread.

Model: claude-opus-5-5
Effort: xhigh
Tokens: 120, 33613, 182918, 7511982 del commit
Cost: 3.64 USD del commit
Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

---------

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>
2026-11-12 01:09:42 +00:00
Weny Xu d061e08717 feat: flush regions under object store WAL retention pressure (#10309)
* feat(store-api): expose advisory WAL flush candidates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(log-store): select regions blocking WAL object reclamation

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): schedule region flushes under WAL retention pressure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): cover WAL pressure reclamation and recovery

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): prioritize oldest WAL reclamation blockers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito): deterministically exhaust range cache admission budget

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 16:06:17 +00:00
Weny Xu 73698114de feat(log-store): add a read-only object store WAL reader (#10269)
* refactor(log-store): extract shared WAL reading helpers

Move object indexing, chain traversal, history checks, segment prefetching and object I/O out of the writable store. Preserve local recovery and read policies.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(log-store): separate read-only WAL object access

Introduce WalObjectRead as the read capability shared by indexing and prefetching. Keep writes and startup listing on WalObjectIo, and adapt writable-store test doubles without changing their behavior.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(log-store): add a read-only object store WAL reader

Validate source identity with NodeWalPrefix and share readers by source. Read known-last-entry or listed ranges using fresh listings and bounded footer concurrency, without holding the cache lock across I/O.

Evict indexes outside the current listing, keep release extraction in startup recovery, and cover bounds, history, cache lifetime, prefix validation and concurrent reads.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): coalesce remote WAL footer loads with shared cache

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): report missing source WAL history separately

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): cover remote WAL chain and listing boundaries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(log-store): wait for asynchronous WAL index release

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test(mito2): align WAL recovery assertions with shared history messages

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 16:14:20 +00:00
jeremyhi 8e16710425 fix(log-store): accept a retried first create of the conditional create probe (#10034)
* fix(log-store): accept a retried first create of the conditional create probe

The startup probe of the object store WAL wrote its first create by hand
and failed on any error. When that create is stored but its response is
lost, the retry layer repeats it, the repeat meets the object, and the
probe reported a failed write, so the WAL refused to open.

Move the create of put_if_absent, which reads the object back and accepts
identical content, into a path-based core and use it for the probe's
first create too. The second create, which the store must refuse, is
unchanged.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): reuse LosingWriter for the retried probe create test

Move LosingWriter from the fault matrix into the test module and use it,
armed with the lose_next pattern, in place of RetriedWriter. Say why the
probe's second create bypasses put_path_if_absent: a read-back would find
the first create's object and let an overwriting store pass.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-28 13:33:39 +00:00
Dragon Roar 5f0e22171e fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)
* fix(log-store): skip records with unparsable keys instead of aborting the read

Signed-off-by: lczllx <2181719471@qq.com>

* fix(log-store): keep undecodable RecordMeta keys fatal

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 06:55:55 +00:00
Yvan Wang 180e2716fc fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715)
* fix(log-store): make raft-engine periodic WAL sync actually fsync

On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and
write() returns before fsync (tikv/raft-engine#395). The periodic
sync task therefore never durable-flushed acknowledged WAL data.

Work around by writing a minimal non-empty marker with sync=true so
the fsync path is reached until a crates.io release includes
tikv/raft-engine#400.

Fixes #9712

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* perf(log-store): reuse LogBatch across periodic WAL syncs

Keep the periodic sync task's LogBatch across ticks instead of
allocating a fresh one (4 KiB encode buffer plus item vec) every
period. A successful Engine::write drains the batch but retains its
buffers; after a failed write or a panicked blocking task the batch is
discarded and recreated on the next tick, since it may be left
partially encoded.

Keep the marker put rather than deleting an empty key: the put
overwrites a single key, whereas every delete would become a recovery
tombstone that raft-engine keeps in memory while replaying append
files.

Also document that the marker value must not decode as a NamespaceImpl
(the marker key lies inside the range scanned by list_namespaces), add
a test covering batch reuse and namespace listing, and apply rustfmt.

Refs #9712

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

---------

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
2026-10-28 02:34:56 +00:00
jeremyhi 1e97a0e7c3 fix(log-store): refuse an object store that ignores conditional create (#9909)
* fix(log-store): refuse an object store that ignores conditional create

The object store WAL checks only the write_with_if_not_exists capability
before it opens, and OpenDAL declares that capability per service. A store
behind the S3 service can accept If-None-Match and overwrite the object
anyway, so the WAL could overwrite its own objects and lose acknowledged
writes without an error.

Before recovery, the store now creates a probe key, unique to the open,
twice under <node prefix>/probes/, outside the objects it lists, and
deletes it. The second create must be refused with a condition failure,
otherwise the open fails with an error naming the object store. The
probe costs two writes and one delete per open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): drop a comment that restates the keys helper

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): accept AlreadyExists from the second probe create

A backend may report an existing conditional-create target as AlreadyExists. The first create stored the probe key, so that error can only refer to it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-26 09:08:13 +00:00
jeremyhi 1c5d15fd36 perf(log-store): read object store WAL footers while the recovery listing goes on (#9889)
* perf(log-store): read object store WAL footers while the recovery listing goes on

Recovery listed the whole prefix before it read any footer, so its LIST
pages, about 80-120 ms each on S3 and one per 1000 objects, added to the
footer reads. The listing now runs in a task of its own and feeds the
footer reads as its pages arrive; a listing that fails part-way still
fails recovery, and a failed footer read stops the listing.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): fail object store WAL recovery as soon as its listing fails

A listing that failed part-way was reported only after every object it had
already listed was read, which on S3 can be tens of thousands of reads. The
stream now settles a finished listing before it yields another object, so
its failure goes first. Tests cover a large listed backlog and a listing
task that panics; the counting and parking test object accesses forward
list_ahead.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): hold the listing until the first footer reads wait in the backlog test

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): check a failed listing settles before buffered objects with a finished task

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): bound the objects the recovery listing holds ahead of the reads

The listing fed the footer reads through an unbounded channel, so a listing
much faster than the reads could hold most of the prefix in memory. It now
holds at most 4096 objects, four LIST pages of S3, and waits for the reads
while that many are queued.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): poll the recovery listing directly to show it waits on a full backlog

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): check the bound with the channel the recovery listing uses

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-23 14:27:39 +00:00
jeremyhi baa5630119 perf(log-store): keep the object store WAL reclaim boundary independent of the region count (#9644)
* perf(log-store): keep the object store WAL reclaim boundary independent of the region count

The WAL actor recomputed the reclaim boundary after every watermark
update, release and indexed object by scanning the footers of the objects
from the previous boundary, checking each segment against the obsolete
watermarks. Each scan started at the first segment of the boundary object,
so when one object holds segments of many regions whose watermarks advance
one region at a time, every update inspected the segments already known to
be obsolete again, about R^2/2 footer entries in total for R regions.

The catalog now keeps the obsolete watermarks itself and counts the live
segments of every object. A watermark move visits only the segments of its
region that it newly reaches, through the per-region index, and the
boundary is the first object at or above the floor that still holds a live
segment. The boundary is unchanged: segments of release records never hold
it, a region without a watermark does, a watermark of EntryId::MAX releases
every segment of its region, and it never moves below the floor. The store
records each watermark in the catalog together with the one reads use.

The per-region index is now keyed by the largest entry id of each segment,
which is also object sequence order because the entry ranges of a region
increase strictly with the sequence. Indexing an object already enforced
that; it now checks the neighbours by entry id instead of by sequence,
which accepts and rejects the same objects.

A property test checks the boundary against a scan of the footers over
random appends, watermark moves, releases, drops, collections and
recoveries, and ignored measurement tests time watermark updates and tail
reads through the store.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): seek the object store WAL objects of a region by entry id

A read found the objects of its region by filtering every object of the
region that the catalog retains, so a read of the last few objects cost
time in proportion to the whole retained history of the region, under the
catalog read lock.

The per-region index is keyed by the largest entry id of each segment, and
the entry ranges of a region increase strictly with the object sequence,
which the catalog enforces when it indexes an object, whatever entry ids
the objects carry. So the lookup seeks to the first segment that ends at
or after the start of the range and stops at the first that starts after
its end, and the result is still in sequence order.

The property test of the catalog now also checks range lookups against a
scan of the footers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-16 14:27:04 +00:00
jeremyhi e68e8c46d8 perf(log-store): reuse the release records the object store WAL recovery already read (#9645)
* perf(log-store): reuse the release records the object store WAL recovery already read

Recovery reads every object it lists to verify its header and footer: an
object of at most 64 KiB whole, a larger one as its header and the last
64 KiB. It then read the segment of release records of every indexed
object that holds one again with a range read, although the segment, the
last one before the footer, is almost always among the bytes already read.
A prefix of many dropped regions, each released in an object of its own,
paid two reads per such object.

The footer read now keeps a copy of that segment when the bytes it read
hold it, and recovery decodes the copy, or reads the segment as before when
the copy is missing, only for the objects on the selected chain. The
segment is copied so that the read window is not held until the releases
are decoded. Chain selection, the merge of the largest released entry id
per region, and the failure of recovery on a release segment that cannot
be fetched or decoded, whatever on_corrupted_segment says, are unchanged.

Tests cover the reads of a recovery whose release segments are all in the
read bytes, a release segment that starts before the tail window, and that
a damaged release segment fails recovery on the chain and is never decoded
off it. An ignored measurement test counts the reads and times recovery
over a prefix of dropped regions.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): bound the release segments object store WAL recovery keeps

Recovery keeps the segment of release records of every object it fetches
before it selects the chain, so an object off the chain could hold up to
the tail window until recovery ends. Keep a segment only up to 4 KiB, far
above the records of the few regions a release seals at once, and read a
larger one again only when its object is on the chain, as before.

Tests cover a release segment inside the tail window of an object read as
its header and its tail, which exercises the offset into the window, and a
segment above the kept size in an object read whole.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-16 12:43:27 +00:00
jeremyhi deae75ffee fix(log-store): refuse to open object store WAL regions whose history is missing (#9561)
* fix(log-store): refuse to open object store WAL regions whose history is missing

A region persists only its WAL prefix, not the object store that holds it.
After `[wal].storage_provider`, or the bucket or root of the provider it
names, changes, or after the objects under the prefix are deleted, every
region reopened against a prefix without its history, replayed nothing and
silently lost its acknowledged writes that were not flushed.

Add `LogStore::check_region_history` and `LogStore::start_region_history`,
whose defaults accept and do nothing, so Raft Engine and Kafka are
unchanged. The object store WAL reports the history of a region as missing
when recovery found no object under the prefix, or when the region's
watermark names an object above the newest object holding entries that
recovery indexed, unless the catalog holds the region's entries through the
watermark or the region's history starts in the store. Both rest on
invariants of the prefix: an opened prefix is never empty, and a persisted
watermark names an object at or below the newest object holding entries,
which collection never deletes. The segment of release records counts as
entries, since an object holding only release records can be the one
collection keeps; start objects hold no segment and do not move the bound.

When a store that found its prefix empty refuses a region, the actor gives
up its start object, so the next start refuses the region again. It agrees
only while nothing was handed an id, no batch, append, release or create
waits, no failed release is kept, no other object was listed, the catalog
holds only the start object and no region's history started in the store,
deciding under the lock region registration takes. It then records a
terminal error, deletes the object as one of its tracked deletes, which
`stop` waits for, and answers every refusal once that delete settled.

Mito runs the check before it replays an existing region on open, on
catch-up and when a create finds the region present, with the larger of
the flushed and truncated entry ids, and records the history of a region
it creates; regions that skip the WAL and noop regions are not checked. A
missing history fails the open with an error naming the region, the
prefix, the configured storage and how to move the WAL by copying its
objects. The new `[wal].accept_missing_history` option, off by default,
opens such a region without its unflushed writes instead, with a warning
and the counter
`greptime_logstore_object_store_wal_missing_history_accepted_total`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the object store WAL history check

Describe the open-time history check, the prefix invariants it relies on,
how release records bound the history, the start object withdrawal of a
refused open and the window that remains, the accept_missing_history
option and how to move a WAL by copying its objects, and update the
metrics, Mito integration, evidence, compatibility and cluster alignment
sections.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): unindex the start object a refused open withdrew

A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): separate the withdrawal windows of start objects and entries

An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep a successor's chain complete across a late withdrawal

A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): record a created region's WAL history before its manifest

A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the start boundary of a chain without segments and the create order

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-15 12:38:01 +00:00
jeremyhi 60655a933b fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases

A region's release lived only in memory: a restart after DROP TABLE and
before the next object never re-established the dropped region's
watermark, so its segments counted as live and pinned garbage collection
of the whole prefix.

A release now writes a release record (region id and the entry id the
region is released through) into an object sealed at once, and returns
once that object is durable and indexed, in both acknowledgement modes.
Recovery seeds the obsolete watermarks from the release records of the
indexed objects before it computes any reclaim boundary. obsolete_all
releases in full; the new LogStore::release_region, a no-op by default,
releases a dropped region through the largest entry id the store handed
out for it, and Mito calls it after obsolete when it drops a region.

Objects with release records use format version 2, the version 1 layout
with the records at the end of the footer; every other object is still
written as version 1. A counter of release records made durable is added.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe persisted object store WAL region releases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): carry a failed region release into the next object

A release whose batch failed left its record only in an orphan object,
so a drop that failed there could still leave the region unreleased
after a restart. The store now keeps the release records of failed
batches and seals them with its next batch, on a timer tick when no
entry is admitted, so the release becomes durable once a create
succeeds again.

The version 1 and version 2 object fixtures are frozen under
tests/compatibility/wal/object_store and loaded by the format tests,
and the RFC no longer says that state is persisted only in headers.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): complete a region drop whose WAL release fails

A drop request is not sent again once the region was removed, so a drop
whose WAL release failed used to leave the region half dropped. The
drop now completes, cleaning up the region's runtime state and
scheduling its files for deletion, and reports the release failure.

In the durable mode the object store WAL now creates one more object at
stop with the release records of failed and dropped batches, so a
graceful stop does not lose a release that failed. The release record
counter is described as counting the records of indexed objects.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe the stop-time release create of the WAL actor

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): state the accepted window of unreleased dropped regions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe release waiters in the WAL actor's outcome table

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): fail region releases while the object store WAL is failing

A failing store keeps its backlog and rejects appends until a create
succeeds, so a release would otherwise wait for an upload that is not
coming. A release that arrives in that state now fails at once with the
retryable error, and one whose batch is still pending when the store
enters it fails as well. Neither poisons the store: the record of the
first waits for the next sealed batch, like that of a batch that failed,
and the record of the second stays in the backlog, so both become durable
once a create succeeds.

Rejected releases count in rejected_appends_total.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe region releases while the object store WAL is failing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): place the drop crash window in the drop procedure

The crash window between the deletion of a table's metadata and a
durable release record is a gap of the drop procedure, not of the WAL.
Describe it as far as the code shows, name the remedy that belongs in
the procedure, cleaning up offline a region a resumed region drop finds
not open, and list it under future work.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep rejected release records coalesced while failing

A release rejected while the store is failing kept its record for the
next sealed batch, but every timer tick then sealed that record alone
behind the refused backlog, so a release repeated across ticks grew the
backlog without bound. A failing store now keeps such records coalesced
and seals them on the first tick after a create succeeds.

The fault-matrix case of a release while the backlog is refused now runs
with a 10 ms flush interval, repeats the release, and checks that the
backlog does not grow and that recovery reads the record back.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): keep rejected release records unsealed while failing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): cite the drop procedure issue for the crash window

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): let a stop seal rejected release records of a failing store

Keeping the records of rejected releases out of sealed batches while the
store is failing also kept them out of the seal a stop requests, so a
stop whose last attempt of the backlog succeeded dropped them and the
released region pinned collection after the restart. A stop now seals
them with the backlog.

A fault-matrix case rejects a release while the store is failing, stops,
lets both creates succeed, and checks the record after a reopen.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe the stop of a failing store with rejected releases

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep release records readable by older object store WAL readers

Release records were written as a version 2 footer, which a build without
it refuses to read, so rolling a node back after a drop failed to open
the whole WAL prefix. They are now the single entry of a segment of the
reserved region u64::MAX at the first position of a version 1 object:
an older reader indexes that segment as a region it never opens, and
recovery reads the segment back with one range read per object that
holds one. The segment never holds the reclaim boundary and names no
namespace.

A durable stop whose last create of release records fails now reports
the error instead of returning Ok and losing the records silently.

The tests are trimmed: the release unit tests and fault-matrix cases
that repeated each other are merged, and the Mito drop tests share one
parameterized case.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe release records as a segment of a reserved region

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): report release records a poisoning stop drops

A durable stop whose last create of release records met a conflict
poisoned the store and returned Ok, losing the records silently. Stop
now reports the error whenever release records are dropped after it
began, by a rollback or by poisoning. The test of a failed release
covers that conflict, and seals its record on a tick after a one-shot
create failure, so the next tick cannot fail too.

The documentation now says that recovery reads the segments of release
records, and that one that does not decode fails recovery.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe how recovery reads segments of release records

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): say that stop reports release records a conflict drops

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): restore the transiently failing release create in enqueued mode

The fault matrix again covers a release whose create fails transiently in
the enqueued mode: the release waits for the repeat under the same
sequence and returns once it succeeds. The encode_batch doc now says the
release segment is the last segment of the object.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): place the release segment and count the restored case

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): do not fail a region drop on a failed WAL release

Returning the release error from the drop left a metric physical region
half dropped: the metric engine stops after the data region and never
drops the metadata region or forgets the physical region, and a retried
drop finds the data region gone and counts it as done. The release error
gave the caller nothing to retry, since the region is already removed
and the log store keeps the record for its next object. The drop now
logs the error and succeeds.

A metric engine test drops a physical region while the WAL releases of
both mito regions fail, and checks that both regions and the physical
region state are gone.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): a failed release no longer fails the drop

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): do not fail a region drop on a failed WAL obsolete

A store that the release of another region poisoned fails obsolete as
well, which left the metric engine with its metadata region undropped.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(metric-engine): drop a physical region whose WAL obsolete or release fails

Cover a failed data release, two failed releases and a conflicting
release that poisons the store, after the regions wrote to the WAL, and
check that a store opened later keeps no segment of either region.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): a failed obsolete does not fail the drop either

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): fail the drop on an obsolete error of other WAL providers

Only the object store WAL is shared by the regions of a store that the
release of one can poison; Raft Engine and Kafka keep failing the drop
when obsolete fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): a stopped or poisoned store keeps no failed release

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-15 03:27:43 +00:00
jeremyhi a08e5bcceb fix(log-store): keep the enqueued backlog on permanent object store errors (#9558)
* fix(log-store): keep the enqueued backlog on permanent object store errors

On object storage the WAL's object store sits behind OpenDAL's retry
layer, which marks every error it returns as persistent. The enqueued
mode classified create failures by that status, so a 403 or a missing
bucket was repeated every 100 ms on every create in flight, logged
nothing, and made appends hang once the sealed batches were full.

Object store errors are now classified by kind: denied access, an
invalid configuration (how OpenDAL reports a missing S3 bucket) and a
write to a missing location are non-retriable. In the enqueued mode such
an error makes the store failing instead of poisoning it: the
acknowledged backlog is kept under its sequences and bytes, its lowest
pending batch is attempted alone with a delay that doubles from 100 ms
to 30 s, appends are rejected at once with a retryable error that
carries the last object store error, and the first create that succeeds
ends the state. Stop uploads the backlog at once and reports a failure
as before. The state is logged on entry, at most once a minute while it
lasts and on exit, and exported as the `failing` gauge and the
`rejected_appends_total` counter. Conflicts, encoding and catalog errors
still poison the store; the durable mode is unchanged.

An append held back because the sealed batches are full is now counted
and timed by the stall metrics in both modes.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(rfc): describe how the enqueued object store WAL handles refused creates

Describe the enqueued failure contract as the store now implements it:
non-retriable object store errors are told apart by kind because
OpenDAL's retry layer under the store marks every error persistent, a
failing store keeps its backlog and attempts it with backoff while it
rejects appends, and stop uploads the backlog at once. Add the failure
matrix row, the failing gauge and the rejected appends counter, the
sealed-batch admission limit and the stall metrics that now count it,
and the fault matrix case counts.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): time the failing attempts where they start and fix two statements

The test of the attempts of a failing store now records when each
refused write starts and checks the gaps between them, instead of
sampling a counter after a sleep of the test task. The delay comment and
the RFC describe 30 seconds as the longest delay between two attempts,
not as a bound on when appends resume, and the RFC states that the next
append is held once seven sealed batches wait.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): drop the wait_until helper that main now defines

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): name errors that need an operator and trim the RFC contract

Rename is_non_retriable to requires_intervention: such a create is still
repeated, only slowly, until someone fixes the credentials, configuration
or bucket. Name the backoff test after what it asserts, cut the test of
the lowest pending batch to two failed rounds since the cadence is
covered by the backoff test, and shorten the RFC failure contract to
the contract and its rationale, leaving error mappings, request counts
and logging cadence to the code comments.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): tie the slow repeat to errors requiring intervention

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-14 06:10:26 +00:00
Weny Xu f690bbb204 refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): update new WAL replay tests after index removal

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-13 15:31:04 +00:00
jeremyhi 55d753ca58 perf(log-store): fetch object store WAL replay segments concurrently (#9555)
* perf(log-store): fetch object store WAL replay segments concurrently

A region read fetched each segment only after the previous object's
entries were yielded, so a replay paid the full request latency once per
object. Fetch up to READ_PREFETCH_DEPTH (8) segments ahead, each on its
own task, and handle their outcomes in object sequence order: entries,
collected-object skips, corrupted-segment holes and errors surface
exactly as before, and dropping the stream aborts the fetches in flight.
A read holds at most eight decoded segments besides the one it yields.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): raise object store WAL recovery concurrency to 64

On S3, recovery reads the header and tail of every object, about 57 to
91 ms per object per lane on 4, 8 and 16 vCPUs alike, and its time
scales with objects divided by concurrency up to 128 lanes without
throttling: about 9,500 objects recovered in 99 s at 8 and 10.5 s at
64. At 64 recovery issues about 1,800 reads per second, a third of
S3's request rate per prefix, and holds one 64 KiB tail window per
lane, 4 MiB in all.

The recovery concurrency tests now scale with the constant.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): share one prefetch byte budget among object store WAL reads

Each read fetched up to eight segments ahead on its own, so regions
opened in parallel (16 by default) could together hold 128 segments of
up to max_batch_bytes each. A read now always fetches the segment it
yields next, as before, and fetches a later one only while its length
fits in a 64 MiB budget shared by every read of the store. Regions
replayed in parallel thus hold one segment each plus at most 64 MiB,
however many open at once. Nothing waits on the budget: a read that
cannot reserve stops fetching ahead until its next poll.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): fail an object store WAL read whose fetch task panics

A fetch task that panicked was re-raised in the reader with
resume_unwind, which unwinds through the caller, such as Mito's region
opener task, past its cleanup. Return it as a WalSegmentFetch error
instead, so the read ends at that object like any other failed fetch.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): bound replay prefetch by decoded size held by fetch tasks

The prefetch reservation sat in the read's queue, so dropping a read
released it at once while an aborted fetch could still hold its
buffers until it next yielded. The fetch task now holds the reservation
and returns it with its outcome, so it is released with the buffers.

A reservation also counted only the encoded segment length, while a
fetched segment is held decoded. It now counts the length plus one
Record per entry, and the comment states the whole bound.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): round the measured figures in the recovery concurrency comment

Keep the magnitudes in the comment; the exact S3 figures belong to the
measurement run rather than the code.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): abort replay fetch tasks with tokio-util's AbortOnDropHandle

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): rename the replay prefetch reservation to try_reserve_prefetch_bytes

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): share the bounded polling helper across the object store WAL tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-13 12:26:16 +00:00
jeremyhi 29d5854e6a test(log-store): drive recovery, read, delete and stall failures through the fault matrix (#9488)
* test(log-store): drive recovery, read, delete and stall failures through the fault matrix

Add the second half of the object store WAL fault matrix: the failures of
the object store access and of the store state that the conditional-create
rows did not drive. Each case asserts the failed waiters and their error,
whether the store is poisoned, the sequence, both indexed gauges at the
failure, the exact movement of every counter and what a read sees after.

Recovery: a listing, a trailer fetch and a footer fetch that fail, the
catalog rejecting an object after every footer was read, and an object
gone by the time it is fetched, below the boundary of the tip (ignored)
and at it (construction fails); each leaves the gauges and the recovery
metrics untouched. Reads: a segment fetch that fails, without a hole or a
read timer sample, and a later read of the region that succeeds.
Collection: a delete that fails and is attempted again at the next
collection, and a stop that waits for a delete in flight, with the catalog
agreeing with the listing in both outcomes. Store state: per-region entry
id exhaustion refusing the append without touching the batch it sealed,
and the durable stop paths with several creates in flight that succeed,
fail transiently or conflict after stop began. Stalled appends released by
admission, by a stop, by a terminal error and by the teardown of the actor,
the first two timed as a pair. The recovery and read timers observed around
parked fetches, twice each, with holds that cannot overlap.

The cases use the existing doubles, extended with a failing listing and a
failing range read at an offset on RecordingIo and parked range reads on
ParkedIo. Case::row is now optional, and the assertion that every row of
the failure matrix has a case is unchanged.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): lose a wide object at its header and footer reads in the fault matrix

The vanished-object case now runs three ways: a short object read whole,
and a wide object deleted before its header and tail window reads or
before its separate footer read, each below and at the boundary of the
tip. The entry id exhaustion case reads the other region of the sealed
batch back, the teardown case asserts the indexed gauges before and after
the actor is gone, and two comments that restated their names are gone.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): prove the stop-path ordering and share the wide-footer fixture in the fault matrix

The case of creates that succeed after stop began keeps object 1 parked
while objects 4 to 2 complete, round-trips a command through the actor and
asserts that no append is acknowledged and nothing is indexed before
object 1 is released. The stop-with-delete case replaces its yields with
the same command barrier, queued behind the stop command. The wide-footer
fixture moves to the tests module and is shared with the recovery test
that fetches a footer longer than the tail window. The gauge helper is
named for its side effect and the vanished-object stage enum for what it
names.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): keep the four-create scenario doc on its fault matrix case

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-12 07:17:44 +00:00
jeremyhi c8ea00b13f feat(log-store): add object store WAL metrics and the conditional-create fault matrix (#9475)
* feat(log-store): add object store WAL metrics and the conditional-create fault matrix

Export the remaining fifteen metrics of the object store WAL under
greptime_logstore_object_store_wal_: the write path timers and object
shape histograms, created objects, the indexed object and byte gauges,
recovery and read timers, recovered objects, create conflicts and
failures, poisoned stores, and stalled appends with their wait.

Add a table-driven fault matrix that drives every outcome of a
conditional create and of the start object through the store and asserts
the failed waiters, the poison, the sequence, both indexed gauges at the
failure, the movement of every metric, and what a read sees afterwards.
The test fails if a row of the failure matrix has no case. Every test of
the store module now holds one lock, since the metrics are process-wide.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): count every conditional create of the object store WAL

Count the outcome of the start object's create, and count a create
that met a different object as a conflict before the epoch of that
object is read, so a failed epoch read no longer hides the conflict.

The fault matrix waits until no store actor of an earlier test is
alive, checks that a poisoned read returns the recorded terminal error,
and covers a conflicting object whose header cannot be read.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-09 06:07:27 +00:00
jeremyhi e4872ec1a2 feat(log-store): collect object store WAL objects below a persisted reclaim boundary (#9471)
* feat(log-store): collect object store WAL objects below a persisted reclaim boundary

Every object header carries a reclaim boundary: every object below it was
fully flushed when the object was sealed. The writer computes the boundary
from the catalog and the obsolete watermarks after every obsolete and every
indexed object, and stamps it on every sealed object and on the start
object. The boundary of the last indexed object permits deletion: every
present object below it, indexed, listed by recovery off the chain, or
left by a failed batch, is deleted in the background, at most four at a
time.

Recovery judges every candidate with its own boundary: a link to a
predecessor below it holds whether or not the predecessor is present, and
the chain stops there. A read that listed an object before it was
collected skips it instead of failing.

The header grows from 46 to 54 bytes; the format version stays 1.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): scan the reclaim boundary from the previous boundary

The boundary was recomputed by scanning every region's indexed objects
from the first one, so the flushed objects of busy regions were scanned
again on every indexed object while another region held the boundary.
Scan the objects in sequence order from the previous boundary instead
and stop at the first one holding a live segment. Watermarks never move
down and a region first appears above the boundary, so the result is
the same and an object is not scanned again once the boundary passed it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): name the parked operation helpers after both their uses

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-08 13:16:14 +00:00
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
jeremyhi f9ab3c296b feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region

A segment that does not decode, because its checksum does not match or
its content disagrees with its footer entry, is fetched once more. If it
still does not decode, the read follows `on_corrupted_segment`: `skip`
records a WAL hole for the region with the object key, sequence and the
footer's entry id range, increments
`greptime_logstore_object_store_wal_skipped_segments_total`, logs a
warning and continues with the next object; `fail` fails the read. A
failed fetch fails the read in both modes. Holes are kept in memory per
region and can be listed from the store.

The standalone test builder allocates WAL options through
`build_standalone_wal_provider`, and integration tests run a standalone
instance on the object store WAL across restarts, on S3 when the
`GT_S3_*` variables are set. The standalone shutdown runs every step and
returns the first error.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(cmd): cover the best-effort standalone stop

Build a standalone instance whose leader services fail to stop and whose
datanode servers fail afterwards, and check that `Instance::stop` still
stops the object store WAL and returns the leader services error.

Rename `skip_segment` to `record_skipped_segment` and the file store
integration test to `test_standalone_object_store_wal_survives_restarts_on_file`.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-08 06:33:40 +00:00
jeremyhi 8553db3f27 feat(mito): wait for WAL durability before publishing a manifest watermark (#9410)
* feat(mito): wait for WAL durability before publishing a manifest watermark

A log store that acknowledges appends before their entries are durable can
hand out entry ids that a crash loses. Before a flush, a full truncate or a
discard of unflushed data records an entry id in the manifest, Mito now waits
on `LogStore::wait_durable` for that id. The flush waits after its SSTs are
written and observes its cancellation, so a drop or truncate queued behind it
is not held up by the upload.

Add engine tests of the object store WAL across restarts and of the barrier,
and a log store testing hook that stores an object and then reports its create
as failed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito): synchronize the WAL barrier tests on store signals

The barrier tests waited with fixed sleeps, which do not establish that a
flush or truncate has reached its durability wait, and the lost backlog test
could stop the store before the seal was handled. The object store WAL gets
testing hooks that observe parked creates and durability waits, and one that
ends the actor as a crash would; the tests wait on them, crash the store and
the engine before a reopen until nothing holds the old store, and cover a
truncate or discard of unflushed data that completes once its entry is
durable.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito): rename the recovery state and the crash test in the WAL recovery tests

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito): drop a comment that restates the empty recovery state

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-30 05:20:59 +00:00
jeremyhi a99cb56d71 feat(datanode): wire the object store WAL into the datanode and standalone (#9386)
* feat(datanode): wire the object store WAL into the datanode and standalone

A datanode configured with `provider = "experimental_object_store"` now
builds an `ObjectStoreLogStore` instead of failing with the placeholder
"not supported yet" error. The builder resolves `storage_provider`
against the object store manager (empty selects the default store, an
unknown name is rejected as an invalid `storage_provider`), opens the
store under the node id and the standalone generation, stops it if a
later build step fails, and hands it to `Datanode`, whose shutdown stops
it after the region server. Datanode shutdown is now best-effort: every
step runs and the first error is returned. A datanode with a metasrv
client still rejects the provider.

The standalone bootstrap builds `WalProvider::ObjectStore` with the same
derived node prefix before the metasrv WAL conversion, so the WAL
options it allocates match the prefix the store runs under.

`log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's
test utilities can build an engine on the object store WAL over an
in-memory object store, and new engine tests cover create, reopen,
replay isolation between regions, batch open, drop and offline cleanup,
the latest entry id after replay, and the rejection of object store WAL
options on a Raft Engine log store.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(datanode): stop every region engine before the object store WAL

The region server returned at the first engine whose stop failed, so the
engines after it in map order, Mito included, were never stopped before
the datanode stopped the object store WAL. It now attempts every engine,
returns the first error and logs the later ones.

`ObjectStoreLogStore` is also exported from the `log-store` crate root,
and its callers import it from there.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-29 13:17:42 +00:00
jeremyhi 03823a9a01 feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL

Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog
thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the
object store WAL config, validated by the datanode and the store.

In the `enqueued` mode `append_batch` returns on admission with the
entry ids assigned and the object is created in the background. At a
backlog threshold the next append is held back until an upload
completes. A transient create failure is repeated under the same
sequence with the same bytes; any conflicting object poisons the
store. `stop` uploads the backlog, or returns the error that dropped
it once stop began. `obsolete` clamps the watermark to the durable
entry id, and an id the store handed out needs no sequence floor.

Add `LogStore::wait_durable` with a default that returns at once. The
object store WAL answers it once the region is durable and indexed
through the entry id, and fails it after a backlog was lost.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): poison an enqueued store on permanent create failures

Repeat a failed create in the enqueued mode only when the storage error
is retryable; any other storage error poisons the store. A conflicting
object poisons an enqueued store without reading its epoch, so a failed
header read cannot turn the conflict into a retry.

A durability wait for an id above the highest id the store handed out
now waits for the handed-out ids of the region below it instead of
returning at once.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): poison on permanent create failures after stop begins

A create that fails with a storage error that is not retryable poisons
an enqueued store even after stop began; only a transient failure drops
the backlog without poisoning. Durability waiters whose callers stopped
waiting are pruned before a new waiter is queued. The backlog age test
no longer depends on a follow-up append finishing within the threshold.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): answer a durability wait once no earlier entry is pending

A durability wait now returns once the region holds no entry at or
below the target that is handed out but not durable, instead of waiting
for the largest id handed out to the region. A later object of the
region that is still being created no longer holds back a wait whose
target it does not cover.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): order the pending durability wait check after the actor

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(store-api): state that the default wait_durable keeps each store's guarantee

The default `LogStore::wait_durable` returns at once, which keeps each
log store's own acknowledgement guarantee; Raft Engine with
`sync_write = false` acknowledges before its periodic sync, so the
documentation no longer claims that every entry id a caller holds is
durable. The object store WAL configuration test now also serializes
the new options and reads them back.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): limit the acknowledgement guarantees to the durable mode

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): repeat enqueued creates that a retry layer marks persistent

An object store wrapped in the OpenDAL retry layer reports a temporary
error that outlasted its retries as persistent rather than temporary.
The enqueued mode now repeats a create after any storage error that is
not permanent, so a transient outage behind the retry layer no longer
poisons the store and drops the acknowledged backlog; after stop began
such a failure still drops the backlog without poisoning.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): cover a persistent create failure after stop begins

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-28 09:17:45 +00:00
jeremyhi 194bc2fb3c feat(log-store): add the object store WAL durable write path (#9320)
* feat(log-store): add the object store WAL durable write path

ObjectStoreLogStore can now write. append_batch admits entries into one
open batch and assigns object-sequence-major entry ids at admission. The
batch is sealed by size, by the flush interval, or before a region would
run past the position range, and sealed batches are uploaded with at most
four conditional creates in flight, started in sequence order. Created
objects are indexed in sequence order, and an append is acknowledged only
once its object is durable and indexed.

A transient create failure rolls back the failed batch and every later
batch unless a later object is already durable, in which case the store
poisons itself with a history-gap error. A conflicting object, an
encoding or catalog error, or taking the last representable sequence
poisons the store.

obsolete now goes through the actor and raises the sequence floor
together with the watermark, refusing with a retryable error while the
next sequence is not settled. stop drops the open batch and the batches
whose create has not started, and lets creates in flight finish.

Only the durable acknowledgement mode exists. A testing feature exposes
hooks to wait for admissions, seal the open batch, and hold or fail
creates.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): synchronize object store WAL write tests with the actor

Tests that assert nothing happened round-trip a command through the actor
instead of yielding the test task, the conflict test waits for the object
it expects, and the obsolete-behind-stop test holds the command channel
itself so the unanswered command is deterministic.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): bound admitted WAL appends and never reuse a failed sequence

A create that reports an error may still have written its object, so the
sequences of failed batches are no longer handed out again: the next
batch keeps the sequence after the last sealed one and retries are
assigned new ids. A retry batched differently can no longer conflict
with that object. As the next sequence never moves back, the sequence
floor of obsolete only waits for an open batch that has handed out ids.

Appends now arrive on their own bounded channel, which the actor stops
reading while MAX_SEALED_BATCHES batches wait to become durable, so a
stalled object store holds callers back instead of growing the backlog,
while stop and obsolete are still handled.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): reserve room for two seals before admitting a WAL append

One admission can seal the open batch before an append that would exhaust
its positions and then the append's own batch, so the actor takes an
append only while two more sealed batches fit under MAX_SEALED_BATCHES.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(log-store): chain object store WAL objects and recover the latest chain

A conditional create can fail with an unknown outcome while its object is
stored, or still lands later, and Mito reuses the row sequences of a
failed append. Replaying such an object next to a later acknowledged one
can let the unacknowledged rows win after a restart.

Format version 2 gives every object header its writer's epoch, a link
to the object it extends (sequence and writer instance) and a header
CRC32, and allows objects without segments. Recovery replays only the
chain ending at the complete object with the largest epoch and sequence:
a link holds when its predecessor is present with the recorded writer
instance, or is missing below every present object. Objects off the
chain are orphans that are never replayed but keep their sequences.

Each open writes an empty object that starts an epoch above every
present object, linked to the recovered tip, before it accepts writes,
so a late object of an earlier instance never ends the chain. A start
object that meets an object of an earlier epoch moves to the next
sequence; one of an equal or later epoch fails the open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): break the WAL chain at every missing predecessor

Accepting a missing predecessor below every present object lets a late
object that lands below the chain change which links hold. Nothing
collects objects yet, so a missing predecessor now always breaks the
link, and recovery fails when objects are present but none completes a
chain.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep the object store WAL format at version 1

The object store WAL has not been enabled anywhere, so no object in the
previous layout exists and the chained header can stay version 1.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): link object store WAL objects by epoch

Only one store instance writes under an epoch, since every open starts an
epoch above every present object and a start object that meets the same
or a later epoch fails the open. The epoch therefore identifies the
instance, and the random writer instance id is dropped from the header.
A link now records the sequence and the epoch of the object it extends,
and holds when the predecessor carries that epoch. The header shrinks to
46 bytes, and the store logs its epoch when it opens.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): fail an open whose start object is already present

Without a random writer instance, two opens that recover the same objects
encode byte-identical start objects, and a conditional create treats the
same bytes as its own retry. A start object that is already present
therefore fails the open with a retryable error instead of letting both
opens claim the epoch; the next open counts the object and starts a
later epoch.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): derive the WAL epoch from the claimed start sequence

Two opens can recover different listings when a late object lands across
a sequence gap between them, pick the same largest epoch plus one, and
both create their start objects under different sequences. The epoch of
an instance is now one above the sequence its start object claims, so a
successful create decides the epoch and no two instances share one. It
stays above every epoch recovery listed, and an object that carries an
epoch above the next sequence fails the open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): never run a held WAL create after the store is dropped

The create test hook ignored the closed hold channel, so a create parked
when the store was dropped could still run. It now returns without
creating. Drop the per-admission bookkeeping of issued entry ids, which
nothing reads, and move the parked I/O documentation to its helper.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): cover a WAL start object stored with an unknown outcome

Add a fault-injection test in which the create of the start object
stores the object but reports an error: the open fails without moving
to another sequence, and the next open counts the stored object and
claims a later epoch. Rename the test helper that writes a whole object
from a given header to put_object_with_header, and drop a needless
clone.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): make the held-create drop test deterministic

Keep the actor running while the store drops the hold sender, so the
parked create always completes on the closed channel instead of racing
the actor's exit. Drop a comment that restates epoch_of.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-24 11:07:29 +00:00
jeremyhi 822042c7bb feat(log-store): chain object store WAL objects and recover the latest chain (#9334)
* feat(log-store): chain object store WAL objects and recover the latest chain

A conditional create can fail with an unknown outcome while its object is
stored, or still lands later, and Mito reuses the row sequences of a
failed append. Replaying such an object next to a later acknowledged one
can let the unacknowledged rows win after a restart.

Format version 2 gives every object header its writer's epoch, a link
to the object it extends (sequence and writer instance) and a header
CRC32, and allows objects without segments. Recovery replays only the
chain ending at the complete object with the largest epoch and sequence:
a link holds when its predecessor is present with the recorded writer
instance, or is missing below every present object. Objects off the
chain are orphans that are never replayed but keep their sequences.

Each open writes an empty object that starts an epoch above every
present object, linked to the recovered tip, before it accepts writes,
so a late object of an earlier instance never ends the chain. A start
object that meets an object of an earlier epoch moves to the next
sequence; one of an equal or later epoch fails the open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): break the WAL chain at every missing predecessor

Accepting a missing predecessor below every present object lets a late
object that lands below the chain change which links hold. Nothing
collects objects yet, so a missing predecessor now always breaks the
link, and recovery fails when objects are present but none completes a
chain.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep the object store WAL format at version 1

The object store WAL has not been enabled anywhere, so no object in the
previous layout exists and the chained header can stay version 1.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): link object store WAL objects by epoch

Only one store instance writes under an epoch, since every open starts an
epoch above every present object and a start object that meets the same
or a later epoch fails the open. The epoch therefore identifies the
instance, and the random writer instance id is dropped from the header.
A link now records the sequence and the epoch of the object it extends,
and holds when the predecessor carries that epoch. The header shrinks to
46 bytes, and the store logs its epoch when it opens.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): fail an open whose start object is already present

Without a random writer instance, two opens that recover the same objects
encode byte-identical start objects, and a conditional create treats the
same bytes as its own retry. A start object that is already present
therefore fails the open with a retryable error instead of letting both
opens claim the epoch; the next open counts the object and starts a
later epoch.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): derive the WAL epoch from the claimed start sequence

Two opens can recover different listings when a late object lands across
a sequence gap between them, pick the same largest epoch plus one, and
both create their start objects under different sequences. The epoch of
an instance is now one above the sequence its start object claims, so a
successful create decides the epoch and no two instances share one. It
stays above every epoch recovery listed, and an object that carries an
epoch above the next sequence fails the open.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): cover a WAL start object stored with an unknown outcome

Add a fault-injection test in which the create of the start object
stores the object but reports an error: the open fails without moving
to another sequence, and the next open counts the stored object and
claims a later epoch. Rename the test helper that writes a whole object
from a given header to put_object_with_header, and drop a needless
clone.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-24 08:41:56 +00:00
1d8d95c12a chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks

cargo-udeps requires a nightly toolchain and its pinned version (0.1.61)
no longer detects unused dependencies against current cargo internals —
unused deps have landed on main undetected (e.g. humantime in
common-frontend since #6689). cargo-shear is a standalone static analyzer
that runs on any toolchain.

- Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' /
  'cargo shear --fix' and retire scripts/fix-udeps.py
- CI: install cargo-shear in the check-udeps job; drop the build cache
  and protoc steps (cargo-shear never compiles)
- Remove ~150 unused dependency declarations found by cargo-shear, move
  misplaced deps to the correct sections, drop orphaned
  [workspace.dependencies] entries (arrow-cast, rustc-hash)
- Add [package.metadata.cargo-shear] ignored entries with explanations
  for dependencies that are structurally required despite no textual
  reference: sqlparser (required by sqlparser_derive expansions in
  datatypes, common-query), common-error (required by common-macro's
  stack_trace_debug expansions in session, tests-fuzz), k8s-openapi
  (feature-pinning for the transitive kube dependency in tests-fuzz),
  tikv-jemalloc-sys (link-only, enables jemalloc profiling features in
  common-mem-prof), protobuf (required by build.rs-generated bindings in
  log-store)
- Drop the obsolete [package.metadata.cargo-udeps.ignore] sections

Part of #9289

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix(meta): populate physical metric table column ids (#9286)

* fix(meta): populate physical metric table column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* test(meta): verify physical metric column ids

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

---------

Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>

* fix(postgres): return empty responses for comment-only SQL (#9295)

fix(postgres): handle parsed empty queries in both protocols

Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>

* ci: create docs follow-up issue on PR merge instead of on label (#9237)

* ci: create docs follow-up issue on PR merge instead of on label

The docbot workflow previously created a docs-repo issue as soon as the
'docs-required' condition was detected (PR opened/edited with the docs
checkbox ticked), even if the PR was never merged.

Now the workflow also triggers on PR 'closed':
- opened/edited: only manage the docs-required/docs-not-required labels
- closed: create the docs issue only when the PR was actually merged and
  carries the docs-required label

This also lets maintainers control issue creation by manually adding or
removing the docs-required label before merging.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: address review comments on docs issue creation timing

- Only touch docs labels when the docs checkbox state actually changed
  in an edit. Previously, editing any other part of the PR body while
  the checkbox stayed checked removed the docs-required label, silently
  dropping the docs follow-up now that issue creation happens at merge.
  Unchanged checkbox now leaves labels untouched, which also preserves
  manual label overrides.
- Do not trust the closed event's stale label snapshot at merge time:
  re-read the live PR via the API and create the docs issue if the
  docs-required label is present OR the checkbox is ticked in the
  current body.
- Make the workflow concurrency group action-aware so a merge run does
  not cancel an in-flight label update from an edit run.

Signed-off-by: Ning Sun <sunning@greptime.com>

* fix: make docs-required label the single source of truth at merge

The label-OR-checkbox merge condition could not distinguish an
intentional opt-out from an unfinished label update: removing
docs-required while the checkbox stayed checked still produced an
issue, and unchecking the box could still produce one if the merge
read the stale label before the edit run removed it.

At merge time, wait for any pending docbot runs on the PR head SHA to
finish their label updates (bounded to 5 minutes), then decide solely
by the live docs-required label. Adds actions: read permission for
listing workflow runs.

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>

* perf(promql): push label filters into grouped join inputs (#9280)

* perf(promql): propagate matching filters through grouped joins

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): check matcher safety on the receiving operand

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): spell out the shapes a filter may cross

`preserves_filter` ended in `_ => true`, which was only sound because
`selector_matchers` independently rejects label rewriting, `count_values`,
subqueries and non-rollup calls on the same operand. Loosening the latter
alone would have silently pushed a matcher below a label rewrite. List the
shapes that carry a scan filter instead and default to `false`.

Cite #9207 for the result labels the grouped cases record: the join
projects the right operand's tag set, so `zone` is missing wherever the
right side aggregates it away.

No behavior change.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): assert the new pushdowns reach the scan

The grouped-join unit tests feed tag columns by hand and the SQLness case
only checks results, which are identical whether or not the rewrite fires.
Nothing would have failed if scalar arithmetic, ranking or grouped
matching stopped propagating. Assert through the planner that the matcher
reaches both scans, with a global topk one-side as the counter-example.

Also state that the duplicate-one-side cases record a cross product
Prometheus rejects (#9209), so the baseline is not read as intended
semantics.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(ci): build tests-integration lib with meta-srv/mock (#9299)

* fix(ci): build tests-integration lib with meta-srv/mock

tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but
the dependency carrying the mock feature sits in [dev-dependencies].
Builds that only touch the lib, such as the apidoc job's cargo doc
--workspace, resolve meta-srv without mock and fail with E0432.
--all-targets builds unify dev-dependency features, which is why check,
clippy and nextest stayed green.

Move the mock-enabled meta-srv entry back to [dependencies]. The other
testing features moved out in #9072 are not needed by the lib and stay
in [dev-dependencies].

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(repartition): split per-case repartition tests

test_repartition_metric ran four format/primary-key-encoding cases in a
single test function, and test_repartition_mito ran two format cases.
Each case builds its own 3-datanode cluster and runs a full repartition
plus GC cycle, so on S3 the metric test took 165-178s against the 180s
nextest terminate-after. Merge queue runs failed on it at random.

Split each case into its own test. Cases were already independent, so
they now run in parallel and each stays far inside the timeout, and a
failure points at one encoding instead of four.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat(json2): support altering JSON2 settings (#9029)

* feat(sql): support alter syntax for JSON2 columns

Signed-off-by: fys <fengys1996@gmail.com>

* fix(json2): preserve rows on type hint mismatch during compaction

* refactor(json2): simplify alter settings handling

* fix(json2): preserve coerced values during compaction

* chore: remove unnecessary clone

* chor: reduce memory allocations

* fix: cargo clippy

* chore: update greptime-proto to main branch

* refactor(datatypes): unify string handling with other JSON type hints

* fix: cr

---------

Signed-off-by: fys <fengys1996@gmail.com>

* fix: keep compaction pruning, metadata, and index work on compact runtime (#9304)

* fix: run compaction pruner tasks on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: keep compaction metadata and index work on compact runtime

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add AI matching, classification, and scoring functions (#9300)

* feat: return matching scores from jev

Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: add Jev choice and score functions

Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: use generic AI SQL function names

Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: share constant AI criteria within each batch

Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: preserve AI score uncertainty in JSONB results

Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs: explain reuse of volatile AI evaluations

Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: share logical table batching with OTLP metrics (#9288)

* feat: share logical table batching with OTLP metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: unify pending rows batch acknowledgement policy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align logical batcher example configuration expectations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: align batcher worker channel defaults to 65536

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): lazily decode dense primary key columns (#9226)

* perf(mito2): lazily decode dense primary key columns

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): bypass lazy decoding for full primary keys

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito-codec): preserve prefix decoding errors

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): align encoded length helper naming

Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): clarify conditional dense key decoding

Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito-codec): share string framing in row converter

Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: sync lock

* fix: shear and check issues

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com>
Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Signed-off-by: fys <fengys1996@gmail.com>
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com>
Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
Co-authored-by: Weny Xu <wenymedia@gmail.com>
2026-09-23 04:56:44 +00:00
jeremyhi 2a63ea2660 feat(log-store): add object store WAL reads and obsolete watermarks (#9248)
* feat(log-store): add object store WAL reads and obsolete watermarks

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* perf(log-store): reuse decoded WAL payload allocations

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): clarify WAL region mismatch error

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-22 04:01:31 +00:00
jeremyhi 5ccc29ce77 feat(log-store): add object store WAL store construction and recovery (#9238)
* feat(log-store): add object store WAL store construction and recovery

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): derive WAL store prefix from node and generation

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(log-store): describe unsupported operation boundary

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-19 02:12:59 +00:00
jeremyhi 9c2933de0b feat(log-store): add the object store WAL batch, catalog and I/O (#9216)
* feat(log-store): add the object store WAL batch, catalog and I/O

Add the three modules between the object format and the store of the
object store WAL:

- batch: the open batch that accumulates admitted entries into the next
  object and assigns entry ids at admission. Ids are object-sequence-major,
  `(object_seq << 20) | position`, with positions starting at one per
  region per object, so a batch that is rolled back and admitted again
  under the same sequence hands out the same ids. The time the first entry
  was admitted is kept for the store's age-based sealing; an empty
  admission does not start it.
- catalog: the in-memory index over object footers, by sequence and per
  region. Insertion is atomic and rejects an empty footer, duplicate
  region segments, invalid entry ranges, an already indexed sequence and
  entry ranges that are not strictly increasing across objects. The next
  object sequence continues after the largest indexed one and is raised
  above the largest entry id of any region, and is rejected once it no
  longer fits an entry id.
- io: object store access under `<prefix>/objects/`: a conditional create
  whose retry with identical content is a no-op and whose conflicting
  content is rejected, whole and range reads, and a listing of well-formed
  keys in sequence order. A prefix with a `.` or `..` component is
  rejected. When the store reports that the object exists but the read
  that compares its content fails, the read failure is returned with its
  retry hint instead of the create failure.

The modules have no callers until the store lands, so they are declared
with `#[allow(dead_code)]`. Only `entry_id` is exported.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): drop comments that restate the batch test assertions

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): use pub(crate) in the object store WAL batch, catalog and I/O

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): keep entry_id crate-private and narrow the create collision check

Re-export `entry_id` as `pub(crate)`: nothing outside `log-store` uses it
yet, and exporting it would freeze the raw `(object_seq, position)`
encoding before the store owns id allocation.

Treat only `ConditionNotMatch` as the sign that a conditional create
collided with an existing object. A store may report `AlreadyExists` for
an unrelated path, for example when a parent of the object is a file, and
that failure must come back as the write failure rather than as the error
of the read that would compare content.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): rename the catalog's out_of_order helper to out_of_order_reason

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-18 07:28:40 +00:00
jeremyhi d45f5d6eaf feat(log-store): add the object store WAL object format (#9200)
* feat(log-store): add the object store WAL object format

Add the byte format of a single object store WAL object: a header with
the GTWALOBJ magic, format version 1, the object sequence and the writer
instance id; one segment per region ordered by region id with entries
ordered by entry id; a footer that records each segment's region id,
entry id range, entry count, byte range and CRC32; and a fixed trailer
with the GTWALTRL magic, the footer location, the footer CRC32 and the
whole-object CRC32.

The module encodes objects deterministically and decodes the header,
trailer, footer and segments separately, with structural checks on
footer ranges and segment tiling. It has no callers yet; the store that
writes and reads objects follows in later changes.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(log-store): pin the object store WAL format with a byte fixture

Add a fixed version 1 object with two regions and five entries as a hex
literal. The test decodes it and checks the exact header, trailer,
footer entries and records, and checks that encoding the same records,
in either input order, reproduces the fixture byte for byte.

Round-trip tests alone pass when a refactor changes field order,
endianness or checksum coverage in both the encoder and the decoder.
The fixture bytes were derived from the documented layout rather than
from the encoder, so such a change now fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(log-store): reject an empty footer when decoding a WAL object

The encoder never writes an object without records, but decode_footer
accepted a footer that declares zero segments, and
verify_segment_ranges accepts an empty footer too. Only the test-only
decode_object rejected it, so a checksum-valid empty object would pass
the header, trailer and footer checks that recovery runs.

Reject a zero entry count in decode_footer and drop the now unreachable
check in decode_object. Add a test that builds a checksum-valid object
with an empty footer and checks that decode_footer and decode_object
reject it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(log-store): use pub(crate) for the WAL object format API

Other log-store modules use pub(crate) for items shared across module
boundaries. Switch the format module from pub(super) to pub(crate) to
follow that convention. No behavior change.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-09-17 09:14:22 +00:00
XuanwoandWenyXu 4c12ea1aba chore(deps): bump opendal to 0.58.1 (#8742)
* chore(deps): bump opendal to 0.58.1

Upgrade direct opendal dependency and workspace object_store_opendal pin
from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal
0.58.0). Adapt to OpenDAL 0.58 composition API:

- Operator::new returns a finished operator; drop .finish() call sites
- Replace HttpClientLayer / raw::HttpClient with OperationContext +
  HttpTransporter (ReqwestTransport)
- Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to
  Service + Layer::apply_service
- Rewrite SecureFs reader/writer/lister for sync factories and StreamRead
- Use OperatorInfo::capability() instead of removed native_capability()

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: retrigger CI after udeps runner segfault

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(object-store): restore suffix read simulation for secure fs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: adapt remaining callers to OpenDAL 0.58

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: Xuanwo <github@xuanwo.io>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: WenyXu <wenymedia@gmail.com>
2026-09-07 08:16:47 +00:00
Weny Xu a932433d21 fix(wal): bound Kafka requests and extend latency buckets (#9026)
* fix(wal): bound Kafka requests and extend latency buckets

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore(wal): update rskafka request timeout revision

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs(config): document Kafka WAL timeouts in MetaSrv

Signed-off-by: WenyXu <wenymedia@gmail.com>

* style: sort common-wal dev dependencies

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-09-04 09:30:02 +00:00
shuiyisongandfys d4af650ec0 perf: reduce cold workspace compile time (#8801)
* refactor: remove datanode and meta-srv dep from frontend

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: use on-device protoc if possible

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor: remove unused dep

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: version and docs

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update logs

Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com>
2026-08-10 07:18:16 +00:00
Weny Xu 912db22417 fix(log-store): deduplicate Kafka WAL multipart records (#8514)
* fix(log-store): deduplicate Kafka WAL multipart records

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): preserve delayed Kafka WAL entry offsets

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): reject conflicting Kafka WAL last records

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): emit Kafka WAL entries on last record

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(log-store): discard duplicate Kafka WAL first records

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-07-15 03:04:11 +00:00
Lei, HUANG 56e9158819 feat: prepare soft-drop WAL retirement (#8475)
* fix: flush soft-dropped regions on close

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): handle flush-on-close race with concurrent in-flight flush

When a region close with `flush_on_close: true` races with an
already-running flush, pass the actual close request (including the
flush_on_close flag) to the DDL handler instead of a default request
so the pending flush is correctly awaited.

Files: `src/mito2/src/worker/handle_close.rs`

Also adds a test verifying that closing with flush-on-close while a
flush is in progress still persists all written data correctly.

Files: `src/mito2/src/engine/close_test.rs`
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat: support full WAL retirement

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: complete close request migration

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: finish close request callsites

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: guard Kafka provider setup behind index collector check

Move Kafka provider initialization and `get_or_insert` inside the
existing `if let Some(collector)` block so these operations are
skipped when no global index collector is configured.

Affected file:
- `src/log-store/src/kafka/log_store.rs`

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: avoid to_vec

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: replace imperative close-region loop with functional combinators

Transform the region close dispatch in `DropTableExecutor` from mutable
`Vec` and `push` loops to iterator chains with `join_all`, improving
idiomatic Rust style and readability.

- `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode
  region-close logic to use `peers.map()` and nested `join_all`, moving
  `node_manager.datanode()` inside the closure to align with the new
  structure

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: decouple Kafka client from WAL checkpoint

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: merge Kafka WAL index checkpoints

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: delegate Kafka WAL retirement to metasrv

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: rebase main and resolve conflicts

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix: license header

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* chore: bump proto to commits on main

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor: remove Kafka obsolete-all index changes

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-07-13 12:32:46 +00:00
Weny Xu fbdc35a109 ci: add remote WAL logical pruning fuzz (#8307)
* feat: configure remote wal checkpoint intervals

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add logical pruning fuzz target

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: add remote WAL logical pruning fuzz

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-18 08:24:47 +00:00
Weny Xu e520ff8300 feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: propagate retry hints in core errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: propagate retry hint through RPC metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fallback retry hints to status codes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: add explicit retry hints for retryable errors

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: remove status code retry fallback

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: classify io retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve retry hints across error wrappers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minior

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve datasource retry hints in query

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve metric engine retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: preserve flow and frontend retry hints

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: share rskafka retry hint mapping

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-06-17 08:27:53 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00
Weny Xu 4d66bd96b8 feat: make distributed time constants and client timeouts configurable (#7433)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2025-12-19 02:23:20 +00:00
Weny XuandLei, HUANG cb0f1afb01 fix: improve network failure detection (#7382)
* fix(meta): add default etcd client options with keep-alive settings (#7363)

* fix: improve network failure detection (#7367)

* Update src/meta-srv/src/handler.rs

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>

---------

Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com>
2025-12-10 09:48:36 +00:00
Weny Xu 3119464ff9 feat: introduce the Noop WAL provider for datanode (#7105)
* feat: introduce noop log store

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update config example

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: add noop wal tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2025-10-20 06:13:27 +00:00
LFCandNing Sun 8fe17d43d5 chore: update rust to nightly 2025-10-01 (#7069)
* chore: update rust to nightly 2025-10-01

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: nix update

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
2025-10-11 07:30:52 +00:00
Ruihang Xia c9377e7c5a build: bump rust edition to 2024 (#6920)
* bump edition

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* gen keyword

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime and env var

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* one more gen fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* lifetime of temporaries in tail expressions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* format again

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy nested if

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* clippy let and return

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-09-08 02:37:18 +00:00
Weny Xu 7cf47ccf54 fix: initialize remote WAL regions with correct flushed entry IDs (#6856)
* fix: initialize remote WAL regions with correct flushed entry IDs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: correct latest offset

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: update sqlness

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add replay checkpoint to catchup request

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2025-09-03 04:00:02 +00:00
LFC d57c0db9e6 fix: no need to early lookup DNS for kafka broker (#6845)
Signed-off-by: luofucong <luofc@foxmail.com>
2025-09-01 03:03:33 +00:00
Weny Xu 566a647ec7 feat: add replay checkpoint to reduce overhead for remote WAL (#6816)
* feat: introduce `TopicRegionValue`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: persist region replay checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: udpate config.md

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor refactor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: send open region instructions with reply checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: use usize

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add topic name pattern

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: enable wal prune by default

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2025-08-27 07:24:33 +00:00
8894cb5406 feat: resolve unused dependencies with cargo-udeps (#6578) (#6619)
* feat:resolve unused dependencies with cargo-udeps (#6578)

Signed-off-by: Arshdeep54 <balarsh535@gmail.com>

* Apply suggestion from @zyy17

Co-authored-by: zyy17 <zyylsxm@gmail.com>

* Apply suggestion from @zyy17

Co-authored-by: zyy17 <zyylsxm@gmail.com>

---------

Signed-off-by: Arshdeep54 <balarsh535@gmail.com>
Co-authored-by: Ning Sun <classicning@gmail.com>
Co-authored-by: zyy17 <zyylsxm@gmail.com>
2025-08-26 10:22:53 +00:00