Files
greptimedb/src/standalone/src
Silvio 6dbff8a081 fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown

raft-engine preallocates the active log file and only trims that tail when the
Engine is dropped. GreptimeDB never dropped its engines before exiting, so every
start after a clean shutdown truncated the tail as a broken batch and logged
"Truncating log file due to broken batch".

RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine;
standalone closes the metadata backend last, and the datanode stops the
raft-engine WAL after the region server.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: address review on raft-engine shutdown

- Add ClosableKvBackend to common-meta; the standalone metadata backend creator
  returns one and Instance closes it explicitly instead of downcasting to
  RaftEngineBackend, so wrappers forward the close.
- Record a failed metadata backend close as a shutdown error.
- Sync and drop the engine on a blocking thread.
- Only check tail trimming when raft-engine preallocated, and check reopening
  after close/stop on every platform.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: release raft-engine engines on every shutdown path

- Stop the raft-engine WAL when the datanode build fails after opening it.
- Purge and sync tasks hold the engine weakly, so dropping the last backend or
  log store reference (as the CLI tools do) closes the engine.
- Fix needless borrows reported by clippy.

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: drop unused ClosableKvBackend import

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: cancel log store readers on stop; address review

- RaftEngineLogStore tracks the tasks feeding read streams and cancels and
  joins them on stop, so a retained, unconsumed stream no longer keeps the
  engine open past shutdown.
- Tests: stop with a retained read stream; datanode shutdown with a failing
  region engine and after a normal shutdown reopens the raft-engine WAL;
  standalone stop with failing steps reopens the metadata directory.
- entries_to_batch no longer takes self; rename opened to get_open_engine,
  log_files_len to log_files_size and PREALLOCATED_LEN to
  PREALLOCATION_THRESHOLD; mod declarations before imports; rename the
  standalone shutdown closure to record_shutdown_error.

Model: claude-opus-5-5
Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: update Cargo.lock for tokio-util's rt feature

The rt feature makes tokio-util depend on futures-util, which the lock file
has to record for `cargo check --locked`.

Model: claude-opus-5-5
Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* refactor: close the raft-engine WAL by ownership instead of a lock

Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again:

- MitoEngine drops its Wal raw entry reader once its workers are stopped,
  so it no longer holds the log store after stop.
- RaftEngineLogStore::close(self) stops the tasks and readers and closes the
  engine. Datanode::shutdown takes the log store after stopping the region
  server and closes it if it holds the last reference; otherwise it logs a
  warning and only stops it.
- A failed build stops the region server it created before closing the log
  store, so the engines release it there too.

Model: claude-opus-5-5
Effort: medium
Tokens: 78, 19161, 107293, 4916570 del commit
Cost: 2.23 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* test: drop the index argument from LogStore::read in the raft-engine tests

#9563 removed it from the trait.

Model: claude-opus-5-5
Effort: medium
Tokens: 16, 3715, 6600, 1706794 del commit
Cost: 0.47 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix: end a cancelled WAL read with an error; test a written region

- A raft-engine read cancelled by stop now ends its stream with an
  IllegalState error instead of an end of stream, so a replay cannot take a
  truncated WAL as fully read. The reader flags the cancel instead of sending
  the error, since it is usually blocked on a full channel at that point.
- Datanode test: a raft-engine datanode with a created region and writes
  shuts down cleanly and its WAL reopens, which fails if anything still
  holds the log store.
- Move the doc comment of is_stopped back onto it.

Model: claude-opus-5-5
Effort: medium
Tokens: 42, 10316, 407027, 4108529 del commit
Cost: 4.28 USD del commit
Session: 82696960-1984-4432-bcf1-33bfece8b526
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

* fix(log-store): drop the no-op sync before closing the raft-engine engine

Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and
`Engine::write` returns before the fsync, so the sync in `close_engine` read
as a final flush that never ran. The active log file is trimmed and synced
by `LogFileWriter::close` when the last reference is dropped, so
`close_engine` now only drops it on a blocking thread.

Model: claude-opus-5-5
Effort: xhigh
Tokens: 120, 33613, 182918, 7511982 del commit
Cost: 3.64 USD del commit
Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f
Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>

---------

Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com>
2026-11-12 01:09:42 +00:00
..