mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-20 17:54:12 +00:00
48bb6beadde54f87de8f5ceab96e0fe77fea04be
188
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
beb1f88696 |
feat: persist bounded WAL releases on region close (#10329)
* feat(log-store): persist bounded region releases in shared WAL batches Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): release safe WAL history when closing regions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): cover bounded WAL release across close and restart Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): clamp bounded releases to durable entries Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito): observe close release failures and cover recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: describe bounded WAL release lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): make bounded release fence sealing deterministic Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): bound queued releases and ignore shutdown hints Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover close release persistence and shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): wait for failed close flush cleanup Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: clarify bounded WAL release backpressure and shutdown Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(mito2): clarify staging test retry invariant Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): back off durable release retries Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): drive release retry after backoff Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): share and verify WAL retry backoff Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: clarify retained WAL release retry semantics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6faa9ec862 |
feat(mito2): replay pending WAL tails in the background (#10342)
* feat(log-store): read the WAL of another node through the log store `LogStore::read_node` streams one region's entries in `(after, upper]` from the WAL of another node that shares the log store's storage, without writing to it. The default returns `None`. The object store WAL implements it with a registry of read-only readers over its own object store. The node prefix comes from the configured root and the store's own generation, and the returned stream holds the reader, so reads of the same node share its cached object indexes. `WalReadUpperBound` moves to store-api so the trait can name it; log-store re-exports it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(mito2): replay pending WAL tails in the background A leader that is writable, altering or editing and still has pending tails replays them in one background task per region and publishes the result in one manifest edit, which opens the read gate and resumes compaction once no tail remains. - The worker starts the task when a lease, a catchup or a role change makes the region writable, and cancels it on close, drop, downgrade and when it replaces the instance. The task always reports its end, so the worker can start another replay or reload a region a conflict fenced. - Each tail is read from this node's WAL, or through `LogStore::read_node` for another node, and replayed with the per-entry code of open-time replay into memtables outside the region's version. Chunks of about 64 MiB are written with the SST half of the flush task, never across two tails, since an SST gives its rows its largest sequence. - The edit adds the SSTs, clears the tails, and advances the flushed entry id to the last entry read and the flushed sequence to the largest chunk sequence. It commits outside the edit queue, under the manifest lock, only while the region is a writable leader recording exactly the replayed tails. A busy region or an unknown result retries the same bytes; a conflict keeps the chunks; a certain failure deletes them. - Another node's listed tail is only a bound. After a catchup made the region writable, the edit keeps its unread part until the region receives a Leader lease, then a second edit clears it. - After the commit the task releases the WAL up to the new replay start, kept below this node's remaining tails, and the worker raises the region's last entry id to the new flushed entry id, so a later truncate cannot pull it back. - WAL pressure no longer flushes a region whose WAL release one of this node's tails caps. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): harden tail replay across leases, cancellation and uncertain commits - A Leader lease counts as received after a catchup only if its heartbeat was sent after the catchup promotion: the writable fast path does not check the start of a downgrade, so a lease granted before an earlier downgrade could otherwise clear the mark. The mark is the promotion time, so the fast path still reads one atomic. - A direct promotion counts as a lease. - The tail commit also raises the committed sequence, so later writes of the region take sequences above the replayed rows. - Registering a writable region starts a replay, which covers a create that reopens an existing region. - Once an attempt may have persisted the edit, later certain failures retry the same content instead of building a new edit. - WAL reads stop when the task is cancelled, and a cancelled task does not start another commit attempt. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that closing a region cancels a stalled tail read Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): hold tail commit attempts by number instead of sleeping Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): cancel tail replays when the worker stops The worker's cleanup closes its remaining regions without the close handler, so it now cancels every running tail replay first; otherwise a replay stalled in a WAL read or waiting for a lease outlives the engine. The stalled-read test covers an engine stop as well as a close. The docs no longer claim that every SST gives all its rows its largest sequence. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): shorten two tail replay comments Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): stop tail reads at their upper bound A tail read over this node's WAL kept reading past the tail's upto until the log ended, so an unreadable later entry failed a replay that already had every entry it needed. Stop the read once it reaches upto. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): inline the tail replay notify helper Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that writes during a tail replay win over replayed rows Whatever records pending tails must raise the region's committed sequence to the tails' largest sequence, so writes accepted while the replay runs get larger sequences than the replayed rows. Record this in AGENTS.md and test it with a write that lands between two commit attempts of the replay. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): bound reads of this node's listed tails A listed bound may name no entry, so a read of this node's WAL never saw an entry equal to it and kept polling later objects; one that failed to read failed the tail. Read every listed tail through `read_node`, which stops at the bound, and keep reported tails on the node's own WAL. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that rows around a tail replay survive a reopen The tail commit raises the flushed entry id to the largest entry read, which can be another node's. Whatever records tails keeps this node's later entry ids above every tail and leaves none of its unflushed entries below one outside its own tails. Record this in AGENTS.md, and make the replay test meet it and reopen the region without a flush afterwards. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject graceful staging entry of a region with pending tails Entering staging through `set_region_role_state_gracefully` skipped the pending-tail check that `EnterStaging` runs, so a region with pending tails could enter staging, end its replay, and return to Leader with nothing replaying its tails. Run the same check before entering staging there. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): replay tails of a region altered right after promotion A promotion checked whether it made the region writable after setting the role. If the worker started an allowed ALTER or edit in between, the check saw Altering or Editing and asked for no replay, and nothing asked for one later. Check for a leader that a replay runs on instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover the WAL release and locked checks of tail commits Check that a tail commit releases the replayed WAL, that it is refused under the manifest lock once the region was demoted or its tails changed, that a commit retried while altering keeps its chunks, and that a replay waiting for a lease does not end and start again. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): note that read_node also reads this node's WAL Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): keep no region version during a tail replay round A replay round held the region's version until it ended, and with it the memtables the region had when the round started. A flush removed them from newer versions, but their write buffer memory was only freed once the round dropped them, so a slow read, a commit retry or a lease wait could keep flushed memory charged and stall writes. Copy the tails and the flushed entry id instead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): count a direct Leader role set as a lease only when it promotes A direct role set to Leader counted as a Leader lease even when the region already was a leader. On a region a catchup promoted, it cleared the mark that keeps another node's unread listed tail until a lease arrives, so the next commit could drop that tail. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): move the tail replay leader check onto MitoRegion The engine and the replay task both use it, next to the other role checks. Its doc now names the Writable, Altering and Editing states and why staging and downgrading regions replay nothing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay waits for a Leader lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin the pressure skip, staging rejection and remote read error The WAL pressure test gets a case whose own tail starts above the flushed entry id and still flushes; the graceful staging case checks the rejection; and a remote tail on a log store that reads no other node's WAL fails as unsupported and keeps its tails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): list every tail replay trigger and the missing admission limit Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): rename tail replay helpers and notifications The names now say what each one checks, records or announces: the replay predicate is a capability rather than a running replay, the lease hook records a lease, the writable notice is an event like its siblings, and the commit notice pairs with the end notice. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): ask whether a Leader lease came after a catchup `is_writable_by_catchup` was negated at both of its call sites. Its inverse, `leader_lease_received`, is what the replay reads. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): plan the read of a pending tail in the tail replay `Wal` no longer interprets `PendingTail` or `UptoKind`. The replay task decides whether a tail comes from this node's own WAL or up to its bound through `Wal::read_node`, which now takes plain arguments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay result goes stale Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): send the writable notice through notify_worker The tail replay already has one helper that sends a background notice to a region's worker; the writable notice now uses it too, and so logs a worker that has stopped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): note that tail replays skip staging regions and index like flushes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): say why each tail gets its own SSTs and what chunks are for Ordering between tails needs a tail per SST: an SST can give all its rows its largest sequence, so an earlier tail sharing an SST with a later one would take the later tail's sequence and outrank rows that a leader between the two tails flushed. Within a tail, the 64 MiB chunks only bound memory: the sequence floor already puts writes accepted during the replay above every tail. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the tail replay end guard like other drop guards Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): skip a tail replay edit that a sync already superseded A commit notice that the worker handles after a sync rebuilt the region's version from a later manifest no longer brings back tails or files that manifest removed. The notice carries the tails the replay read, and the worker applies the edit only while the version still records them. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log when a tail replay stops before publishing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover lock waits, busy edits and repeated triggers in tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log every exit that drops a tail replay Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): tell the event listener when a tail replay checks for a lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover uncertain commits, the SST hook and lease waits of tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore(mito2): log each tail replay exit once, where it happens Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): reuse the mock region hook and wait for role change tasks in tail replay tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): cover failed-round cleanup, instance replacement and leased first rounds of tail replays Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): make the replacement helper a TestRegion method and fix a lease comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
6a202adbe7 |
fix(log-store): skip objects collected below the boundary in WAL reads (#10343)
* fix(log-store): skip objects collected below the boundary in WAL reads The read-only WAL reader lists the source prefix and then reads the header and footer of every listed object. While the store writing the prefix is still collecting objects, a listed object can be deleted before its footer is read. The NotFound failed the whole read, even when the object lay below the reclaim boundary and was off the chain the read replays, so reads of a large prefix under steady writes failed often. Apply the rule startup recovery already uses. An object gone below the reclaim boundary at which the chain of the read stops is skipped. Any other vanished object, on the chain or off it, still fails the read: it may be the object that completes a later chain or carries the largest epoch. Startup recovery and the reader now share one NotFound split and one boundary check. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover unreadable objects in listed WAL reads A NotFound below the boundary is skipped, but any other error reading a header or footer must still fail the read. Cover it with a corrupted header below the boundary in both modes. Describe the boundary in `ensure_vanished_below` and `read` in terms that also fit the `LastEntry` mode, where the chain can stop at the object of `after` instead of a reclaim boundary. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe where WAL reads stop following links In the `LastEntry` mode a later object can carry a reclaim boundary above the object of the reported entry. The read then keeps that object as its tip below the boundary and follows no link from it. Say that the boundary stops the links to predecessors, and that the tip is the one object below it a read can need. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): pin the outcomes of vanished-object reader tests The unreadable-object test now asserts the InvalidWalObject error for object 1 inside the index cache's wrapper, instead of any error other than NotFound. The vanished chain-object test states the outcome it guards against: without object 2 a listed read could select the chain ending at object 1 and succeed with no entries. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
6dbff8a081 |
fix(log-store): close raft-engine engines on shutdown (#9518)
* fix(log-store): close raft-engine engines on shutdown raft-engine preallocates the active log file and only trims that tail when the Engine is dropped. GreptimeDB never dropped its engines before exiting, so every start after a clean shutdown truncated the tail as a broken batch and logged "Truncating log file due to broken batch". RaftEngineBackend::close and RaftEngineLogStore::stop now release the engine; standalone closes the metadata backend last, and the datanode stops the raft-engine WAL after the region server. Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * refactor: address review on raft-engine shutdown - Add ClosableKvBackend to common-meta; the standalone metadata backend creator returns one and Instance closes it explicitly instead of downcasting to RaftEngineBackend, so wrappers forward the close. - Record a failed metadata backend close as a shutdown error. - Sync and drop the engine on a blocking thread. - Only check tail trimming when raft-engine preallocated, and check reopening after close/stop on every platform. Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: release raft-engine engines on every shutdown path - Stop the raft-engine WAL when the datanode build fails after opening it. - Purge and sync tasks hold the engine weakly, so dropping the last backend or log store reference (as the CLI tools do) closes the engine. - Fix needless borrows reported by clippy. Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: drop unused ClosableKvBackend import Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: cancel log store readers on stop; address review - RaftEngineLogStore tracks the tasks feeding read streams and cancels and joins them on stop, so a retained, unconsumed stream no longer keeps the engine open past shutdown. - Tests: stop with a retained read stream; datanode shutdown with a failing region engine and after a normal shutdown reopens the raft-engine WAL; standalone stop with failing steps reopens the metadata directory. - entries_to_batch no longer takes self; rename opened to get_open_engine, log_files_len to log_files_size and PREALLOCATED_LEN to PREALLOCATION_THRESHOLD; mod declarations before imports; rename the standalone shutdown closure to record_shutdown_error. Model: claude-opus-5-5 Tokens: 15.8k input, 148k output, 2.9M cache write, 103M cache read Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: update Cargo.lock for tokio-util's rt feature The rt feature makes tokio-util depend on futures-util, which the lock file has to record for `cargo check --locked`. Model: claude-opus-5-5 Tokens: 15.9k input, 162k output, 3.0M cache write, 124M cache read Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * refactor: close the raft-engine WAL by ownership instead of a lock Per review, RaftEngineLogStore holds the engine as a plain Arc<Engine> again: - MitoEngine drops its Wal raw entry reader once its workers are stopped, so it no longer holds the log store after stop. - RaftEngineLogStore::close(self) stops the tasks and readers and closes the engine. Datanode::shutdown takes the log store after stopping the region server and closes it if it holds the last reference; otherwise it logs a warning and only stops it. - A failed build stops the region server it created before closing the log store, so the engines release it there too. Model: claude-opus-5-5 Effort: medium Tokens: 78, 19161, 107293, 4916570 del commit Cost: 2.23 USD del commit Session: 82696960-1984-4432-bcf1-33bfece8b526 Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * test: drop the index argument from LogStore::read in the raft-engine tests #9563 removed it from the trait. Model: claude-opus-5-5 Effort: medium Tokens: 16, 3715, 6600, 1706794 del commit Cost: 0.47 USD del commit Session: 82696960-1984-4432-bcf1-33bfece8b526 Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix: end a cancelled WAL read with an error; test a written region - A raft-engine read cancelled by stop now ends its stream with an IllegalState error instead of an end of stream, so a replay cannot take a truncated WAL as fully read. The reader flags the cancel instead of sending the error, since it is usually blocked on a full channel at that point. - Datanode test: a raft-engine datanode with a created region and writes shuts down cleanly and its WAL reopens, which fails if anything still holds the log store. - Move the doc comment of is_stopped back onto it. Model: claude-opus-5-5 Effort: medium Tokens: 42, 10316, 407027, 4108529 del commit Cost: 4.28 USD del commit Session: 82696960-1984-4432-bcf1-33bfece8b526 Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> * fix(log-store): drop the no-op sync before closing the raft-engine engine Per review: on raft-engine 0.4.2 `Engine::sync` writes an empty batch, and `Engine::write` returns before the fsync, so the sync in `close_engine` read as a final flush that never ran. The active log file is trimmed and synced by `LogFileWriter::close` when the last reference is dropped, so `close_engine` now only drops it on a blocking thread. Model: claude-opus-5-5 Effort: xhigh Tokens: 120, 33613, 182918, 7511982 del commit Cost: 3.64 USD del commit Session: 57ca77a8-f6fe-4fa2-a55a-dda31e5c541f Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> --------- Signed-off-by: Silvio <62994417+AbstractBike@users.noreply.github.com> |
||
|
|
d061e08717 |
feat: flush regions under object store WAL retention pressure (#10309)
* feat(store-api): expose advisory WAL flush candidates Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(log-store): select regions blocking WAL object reclamation Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): schedule region flushes under WAL retention pressure Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): cover WAL pressure reclamation and recovery Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): prioritize oldest WAL reclamation blockers Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito): deterministically exhaust range cache admission budget Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
73698114de |
feat(log-store): add a read-only object store WAL reader (#10269)
* refactor(log-store): extract shared WAL reading helpers Move object indexing, chain traversal, history checks, segment prefetching and object I/O out of the writable store. Preserve local recovery and read policies. Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(log-store): separate read-only WAL object access Introduce WalObjectRead as the read capability shared by indexing and prefetching. Keep writes and startup listing on WalObjectIo, and adapt writable-store test doubles without changing their behavior. Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(log-store): add a read-only object store WAL reader Validate source identity with NodeWalPrefix and share readers by source. Read known-last-entry or listed ranges using fresh listings and bounded footer concurrency, without holding the cache lock across I/O. Evict indexes outside the current listing, keep release extraction in startup recovery, and cover bounds, history, cache lifetime, prefix validation and concurrent reads. Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): coalesce remote WAL footer loads with shared cache Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): report missing source WAL history separately Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): cover remote WAL chain and listing boundaries Signed-off-by: WenyXu <wenymedia@gmail.com> * test(log-store): wait for asynchronous WAL index release Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): align WAL recovery assertions with shared history messages Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
8e16710425 |
fix(log-store): accept a retried first create of the conditional create probe (#10034)
* fix(log-store): accept a retried first create of the conditional create probe The startup probe of the object store WAL wrote its first create by hand and failed on any error. When that create is stored but its response is lost, the retry layer repeats it, the repeat meets the object, and the probe reported a failed write, so the WAL refused to open. Move the create of put_if_absent, which reads the object back and accepts identical content, into a path-based core and use it for the probe's first create too. The second create, which the store must refuse, is unchanged. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): reuse LosingWriter for the retried probe create test Move LosingWriter from the fault matrix into the test module and use it, armed with the lose_next pattern, in place of RetriedWriter. Say why the probe's second create bypasses put_path_if_absent: a read-back would find the first create's object and let an overwriting store pass. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
5f0e22171e |
fix(log-store): skip records with unparsable keys instead of aborting the read (#9749)
* fix(log-store): skip records with unparsable keys instead of aborting the read Signed-off-by: lczllx <2181719471@qq.com> * fix(log-store): keep undecodable RecordMeta keys fatal Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
|
|
180e2716fc |
fix(log-store): make raft-engine periodic WAL sync actually fsync (#9715)
* fix(log-store): make raft-engine periodic WAL sync actually fsync On raft-engine 0.4.2, Engine::sync() writes an empty LogBatch and write() returns before fsync (tikv/raft-engine#395). The periodic sync task therefore never durable-flushed acknowledged WAL data. Work around by writing a minimal non-empty marker with sync=true so the fsync path is reached until a crates.io release includes tikv/raft-engine#400. Fixes #9712 Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> * perf(log-store): reuse LogBatch across periodic WAL syncs Keep the periodic sync task's LogBatch across ticks instead of allocating a fresh one (4 KiB encode buffer plus item vec) every period. A successful Engine::write drains the batch but retains its buffers; after a failed write or a panicked blocking task the batch is discarded and recreated on the next tick, since it may be left partially encoded. Keep the marker put rather than deleting an empty key: the put overwrites a single key, whereas every delete would become a recovery tombstone that raft-engine keeps in memory while replaying append files. Also document that the marker value must not decode as a NamespaceImpl (the marker key lies inside the range scanned by list_namespaces), add a test covering batch reuse and namespace listing, and apply rustfmt. Refs #9712 Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> --------- Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> |
||
|
|
1e97a0e7c3 |
fix(log-store): refuse an object store that ignores conditional create (#9909)
* fix(log-store): refuse an object store that ignores conditional create The object store WAL checks only the write_with_if_not_exists capability before it opens, and OpenDAL declares that capability per service. A store behind the S3 service can accept If-None-Match and overwrite the object anyway, so the WAL could overwrite its own objects and lose acknowledged writes without an error. Before recovery, the store now creates a probe key, unique to the open, twice under <node prefix>/probes/, outside the objects it lists, and deletes it. The second create must be refused with a condition failure, otherwise the open fails with an error naming the object store. The probe costs two writes and one delete per open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop a comment that restates the keys helper Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): accept AlreadyExists from the second probe create A backend may report an existing conditional-create target as AlreadyExists. The first create stored the probe key, so that error can only refer to it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
1c5d15fd36 |
perf(log-store): read object store WAL footers while the recovery listing goes on (#9889)
* perf(log-store): read object store WAL footers while the recovery listing goes on Recovery listed the whole prefix before it read any footer, so its LIST pages, about 80-120 ms each on S3 and one per 1000 objects, added to the footer reads. The listing now runs in a task of its own and feeds the footer reads as its pages arrive; a listing that fails part-way still fails recovery, and a failed footer read stops the listing. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): fail object store WAL recovery as soon as its listing fails A listing that failed part-way was reported only after every object it had already listed was read, which on S3 can be tens of thousands of reads. The stream now settles a finished listing before it yields another object, so its failure goes first. Tests cover a large listed backlog and a listing task that panics; the counting and parking test object accesses forward list_ahead. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): hold the listing until the first footer reads wait in the backlog test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): check a failed listing settles before buffered objects with a finished task Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): bound the objects the recovery listing holds ahead of the reads The listing fed the footer reads through an unbounded channel, so a listing much faster than the reads could hold most of the prefix in memory. It now holds at most 4096 objects, four LIST pages of S3, and waits for the reads while that many are queued. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): poll the recovery listing directly to show it waits on a full backlog Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): check the bound with the channel the recovery listing uses Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
baa5630119 |
perf(log-store): keep the object store WAL reclaim boundary independent of the region count (#9644)
* perf(log-store): keep the object store WAL reclaim boundary independent of the region count The WAL actor recomputed the reclaim boundary after every watermark update, release and indexed object by scanning the footers of the objects from the previous boundary, checking each segment against the obsolete watermarks. Each scan started at the first segment of the boundary object, so when one object holds segments of many regions whose watermarks advance one region at a time, every update inspected the segments already known to be obsolete again, about R^2/2 footer entries in total for R regions. The catalog now keeps the obsolete watermarks itself and counts the live segments of every object. A watermark move visits only the segments of its region that it newly reaches, through the per-region index, and the boundary is the first object at or above the floor that still holds a live segment. The boundary is unchanged: segments of release records never hold it, a region without a watermark does, a watermark of EntryId::MAX releases every segment of its region, and it never moves below the floor. The store records each watermark in the catalog together with the one reads use. The per-region index is now keyed by the largest entry id of each segment, which is also object sequence order because the entry ranges of a region increase strictly with the sequence. Indexing an object already enforced that; it now checks the neighbours by entry id instead of by sequence, which accepts and rejects the same objects. A property test checks the boundary against a scan of the footers over random appends, watermark moves, releases, drops, collections and recoveries, and ignored measurement tests time watermark updates and tail reads through the store. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): seek the object store WAL objects of a region by entry id A read found the objects of its region by filtering every object of the region that the catalog retains, so a read of the last few objects cost time in proportion to the whole retained history of the region, under the catalog read lock. The per-region index is keyed by the largest entry id of each segment, and the entry ranges of a region increase strictly with the object sequence, which the catalog enforces when it indexes an object, whatever entry ids the objects carry. So the lookup seeks to the first segment that ends at or after the start of the range and stops at the first that starts after its end, and the result is still in sequence order. The property test of the catalog now also checks range lookups against a scan of the footers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
e68e8c46d8 |
perf(log-store): reuse the release records the object store WAL recovery already read (#9645)
* perf(log-store): reuse the release records the object store WAL recovery already read Recovery reads every object it lists to verify its header and footer: an object of at most 64 KiB whole, a larger one as its header and the last 64 KiB. It then read the segment of release records of every indexed object that holds one again with a range read, although the segment, the last one before the footer, is almost always among the bytes already read. A prefix of many dropped regions, each released in an object of its own, paid two reads per such object. The footer read now keeps a copy of that segment when the bytes it read hold it, and recovery decodes the copy, or reads the segment as before when the copy is missing, only for the objects on the selected chain. The segment is copied so that the read window is not held until the releases are decoded. Chain selection, the merge of the largest released entry id per region, and the failure of recovery on a release segment that cannot be fetched or decoded, whatever on_corrupted_segment says, are unchanged. Tests cover the reads of a recovery whose release segments are all in the read bytes, a release segment that starts before the tail window, and that a damaged release segment fails recovery on the chain and is never decoded off it. An ignored measurement test counts the reads and times recovery over a prefix of dropped regions. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): bound the release segments object store WAL recovery keeps Recovery keeps the segment of release records of every object it fetches before it selects the chain, so an object off the chain could hold up to the tail window until recovery ends. Keep a segment only up to 4 KiB, far above the records of the few regions a release seals at once, and read a larger one again only when its object is on the chain, as before. Tests cover a release segment inside the tail window of an object read as its header and its tail, which exercises the offset into the window, and a segment above the kept size in an object read whole. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
deae75ffee |
fix(log-store): refuse to open object store WAL regions whose history is missing (#9561)
* fix(log-store): refuse to open object store WAL regions whose history is missing A region persists only its WAL prefix, not the object store that holds it. After `[wal].storage_provider`, or the bucket or root of the provider it names, changes, or after the objects under the prefix are deleted, every region reopened against a prefix without its history, replayed nothing and silently lost its acknowledged writes that were not flushed. Add `LogStore::check_region_history` and `LogStore::start_region_history`, whose defaults accept and do nothing, so Raft Engine and Kafka are unchanged. The object store WAL reports the history of a region as missing when recovery found no object under the prefix, or when the region's watermark names an object above the newest object holding entries that recovery indexed, unless the catalog holds the region's entries through the watermark or the region's history starts in the store. Both rest on invariants of the prefix: an opened prefix is never empty, and a persisted watermark names an object at or below the newest object holding entries, which collection never deletes. The segment of release records counts as entries, since an object holding only release records can be the one collection keeps; start objects hold no segment and do not move the bound. When a store that found its prefix empty refuses a region, the actor gives up its start object, so the next start refuses the region again. It agrees only while nothing was handed an id, no batch, append, release or create waits, no failed release is kept, no other object was listed, the catalog holds only the start object and no region's history started in the store, deciding under the lock region registration takes. It then records a terminal error, deletes the object as one of its tracked deletes, which `stop` waits for, and answers every refusal once that delete settled. Mito runs the check before it replays an existing region on open, on catch-up and when a create finds the region present, with the larger of the flushed and truncated entry ids, and records the history of a region it creates; regions that skip the WAL and noop regions are not checked. A missing history fails the open with an error naming the region, the prefix, the configured storage and how to move the WAL by copying its objects. The new `[wal].accept_missing_history` option, off by default, opens such a region without its unflushed writes instead, with a warning and the counter `greptime_logstore_object_store_wal_missing_history_accepted_total`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the object store WAL history check Describe the open-time history check, the prefix invariants it relies on, how release records bound the history, the start object withdrawal of a refused open and the window that remains, the accept_missing_history option and how to move a WAL by copying its objects, and update the metrics, Mito integration, evidence, compatibility and cluster alignment sections. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): unindex the start object a refused open withdrew A withdrawal that deleted the start object left it in the catalog and in the indexed object and byte gauges for the rest of the process. Remove it from both once the delete succeeded, and keep it when the delete failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): separate the withdrawal windows of start objects and entries An append or a release admitted before a refusal leaves an object holding entries, which may have collected the start object, so the next start also passes watermarks at or below that object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep a successor's chain complete across a late withdrawal A start object withdrawn after another instance recovered it was linked by that instance's start object, which carried reclaim boundary 0, so once the delete landed the next open found no complete chain. A start object now reclaims every object below it when the recovered chain holds no segment, so such a delete stays below its boundary. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): record a created region's WAL history before its manifest A create recorded the region's WAL history after it persisted the manifest, so a withdrawal in between failed the create but left a manifest that the next start refused as a region without its history. Record the history first, so such a create fails before it writes anything. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the start boundary of a chain without segments and the create order Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
60655a933b |
fix(log-store): persist object store WAL region releases (#9579)
* fix(log-store): persist object store WAL region releases A region's release lived only in memory: a restart after DROP TABLE and before the next object never re-established the dropped region's watermark, so its segments counted as live and pinned garbage collection of the whole prefix. A release now writes a release record (region id and the entry id the region is released through) into an object sealed at once, and returns once that object is durable and indexed, in both acknowledgement modes. Recovery seeds the obsolete watermarks from the release records of the indexed objects before it computes any reclaim boundary. obsolete_all releases in full; the new LogStore::release_region, a no-op by default, releases a dropped region through the largest entry id the store handed out for it, and Mito calls it after obsolete when it drops a region. Objects with release records use format version 2, the version 1 layout with the records at the end of the footer; every other object is still written as version 1. A counter of release records made durable is added. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe persisted object store WAL region releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): carry a failed region release into the next object A release whose batch failed left its record only in an orphan object, so a drop that failed there could still leave the region unreleased after a restart. The store now keeps the release records of failed batches and seals them with its next batch, on a timer tick when no entry is admitted, so the release becomes durable once a create succeeds again. The version 1 and version 2 object fixtures are frozen under tests/compatibility/wal/object_store and loaded by the format tests, and the RFC no longer says that state is persisted only in headers. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): complete a region drop whose WAL release fails A drop request is not sent again once the region was removed, so a drop whose WAL release failed used to leave the region half dropped. The drop now completes, cleaning up the region's runtime state and scheduling its files for deletion, and reports the release failure. In the durable mode the object store WAL now creates one more object at stop with the release records of failed and dropped batches, so a graceful stop does not lose a release that failed. The release record counter is described as counting the records of indexed objects. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe the stop-time release create of the WAL actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): state the accepted window of unreleased dropped regions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe release waiters in the WAL actor's outcome table Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail region releases while the object store WAL is failing A failing store keeps its backlog and rejects appends until a create succeeds, so a release would otherwise wait for an upload that is not coming. A release that arrives in that state now fails at once with the retryable error, and one whose batch is still pending when the store enters it fails as well. Neither poisons the store: the record of the first waits for the next sealed batch, like that of a batch that failed, and the record of the second stays in the backlog, so both become durable once a create succeeds. Rejected releases count in rejected_appends_total. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe region releases while the object store WAL is failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the drop crash window in the drop procedure The crash window between the deletion of a table's metadata and a durable release record is a gap of the drop procedure, not of the WAL. Describe it as far as the code shows, name the remedy that belongs in the procedure, cleaning up offline a region a resumed region drop finds not open, and list it under future work. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep rejected release records coalesced while failing A release rejected while the store is failing kept its record for the next sealed batch, but every timer tick then sealed that record alone behind the refused backlog, so a release repeated across ticks grew the backlog without bound. A failing store now keeps such records coalesced and seals them on the first tick after a create succeeds. The fault-matrix case of a release while the backlog is refused now runs with a 10 ms flush interval, repeats the release, and checks that the backlog does not grow and that recovery reads the record back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): keep rejected release records unsealed while failing Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): cite the drop procedure issue for the crash window Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): let a stop seal rejected release records of a failing store Keeping the records of rejected releases out of sealed batches while the store is failing also kept them out of the seal a stop requests, so a stop whose last attempt of the backlog succeeded dropped them and the released region pinned collection after the restart. A stop now seals them with the backlog. A fault-matrix case rejects a release while the store is failing, stops, lets both creates succeed, and checks the record after a reopen. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe the stop of a failing store with rejected releases Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep release records readable by older object store WAL readers Release records were written as a version 2 footer, which a build without it refuses to read, so rolling a node back after a drop failed to open the whole WAL prefix. They are now the single entry of a segment of the reserved region u64::MAX at the first position of a version 1 object: an older reader indexes that segment as a region it never opens, and recovery reads the segment back with one range read per object that holds one. The segment never holds the reclaim boundary and names no namespace. A durable stop whose last create of release records fails now reports the error instead of returning Ok and losing the records silently. The tests are trimmed: the release unit tests and fault-matrix cases that repeated each other are merged, and the Mito drop tests share one parameterized case. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe release records as a segment of a reserved region Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): report release records a poisoning stop drops A durable stop whose last create of release records met a conflict poisoned the store and returned Ok, losing the records silently. Stop now reports the error whenever release records are dropped after it began, by a rollback or by poisoning. The test of a failed release covers that conflict, and seals its record on a tick after a one-shot create failure, so the next tick cannot fail too. The documentation now says that recovery reads the segments of release records, and that one that does not decode fails recovery. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how recovery reads segments of release records Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): say that stop reports release records a conflict drops Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): restore the transiently failing release create in enqueued mode The fault matrix again covers a release whose create fails transiently in the enqueued mode: the release waits for the repeat under the same sequence and returns once it succeeds. The encode_batch doc now says the release segment is the last segment of the object. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): place the release segment and count the restored case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL release Returning the release error from the drop left a metric physical region half dropped: the metric engine stops after the data region and never drops the metadata region or forgets the physical region, and a retried drop finds the data region gone and counts it as done. The release error gave the caller nothing to retry, since the region is already removed and the log store keeps the record for its next object. The drop now logs the error and succeeds. A metric engine test drops a physical region while the WAL releases of both mito regions fail, and checks that both regions and the physical region state are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed release no longer fails the drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): do not fail a region drop on a failed WAL obsolete A store that the release of another region poisoned fails obsolete as well, which left the metric engine with its metadata region undropped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(metric-engine): drop a physical region whose WAL obsolete or release fails Cover a failed data release, two failed releases and a conflicting release that poisons the store, after the regions wrote to the WAL, and check that a store opened later keeps no segment of either region. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a failed obsolete does not fail the drop either Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): fail the drop on an obsolete error of other WAL providers Only the object store WAL is shared by the regions of a store that the release of one can poison; Raft Engine and Kafka keep failing the drop when obsolete fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): a stopped or poisoned store keeps no failed release Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
a08e5bcceb |
fix(log-store): keep the enqueued backlog on permanent object store errors (#9558)
* fix(log-store): keep the enqueued backlog on permanent object store errors On object storage the WAL's object store sits behind OpenDAL's retry layer, which marks every error it returns as persistent. The enqueued mode classified create failures by that status, so a 403 or a missing bucket was repeated every 100 ms on every create in flight, logged nothing, and made appends hang once the sealed batches were full. Object store errors are now classified by kind: denied access, an invalid configuration (how OpenDAL reports a missing S3 bucket) and a write to a missing location are non-retriable. In the enqueued mode such an error makes the store failing instead of poisoning it: the acknowledged backlog is kept under its sequences and bytes, its lowest pending batch is attempted alone with a delay that doubles from 100 ms to 30 s, appends are rejected at once with a retryable error that carries the last object store error, and the first create that succeeds ends the state. Stop uploads the backlog at once and reports a failure as before. The state is logged on entry, at most once a minute while it lasts and on exit, and exported as the `failing` gauge and the `rejected_appends_total` counter. Conflicts, encoding and catalog errors still poison the store; the durable mode is unchanged. An append held back because the sealed batches are full is now counted and timed by the stall metrics in both modes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(rfc): describe how the enqueued object store WAL handles refused creates Describe the enqueued failure contract as the store now implements it: non-retriable object store errors are told apart by kind because OpenDAL's retry layer under the store marks every error persistent, a failing store keeps its backlog and attempts it with backoff while it rejects appends, and stop uploads the backlog at once. Add the failure matrix row, the failing gauge and the rejected appends counter, the sealed-batch admission limit and the stall metrics that now count it, and the fault matrix case counts. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): time the failing attempts where they start and fix two statements The test of the attempts of a failing store now records when each refused write starts and checks the gaps between them, instead of sampling a counter after a sleep of the test task. The delay comment and the RFC describe 30 seconds as the longest delay between two attempts, not as a bound on when appends resume, and the RFC states that the next append is held once seven sealed batches wait. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop the wait_until helper that main now defines Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): name errors that need an operator and trim the RFC contract Rename is_non_retriable to requires_intervention: such a create is still repeated, only slowly, until someone fixes the credentials, configuration or bucket. Name the backoff test after what it asserts, cut the test of the lowest pending batch to two failed rounds since the cadence is covered by the backoff test, and shorten the RFC failure contract to the contract and its rationale, leaving error mappings, request counts and logging cadence to the code comments. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): tie the slow repeat to errors requiring intervention Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
f690bbb204 |
refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): update new WAL replay tests after index removal Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
55d753ca58 |
perf(log-store): fetch object store WAL replay segments concurrently (#9555)
* perf(log-store): fetch object store WAL replay segments concurrently A region read fetched each segment only after the previous object's entries were yielded, so a replay paid the full request latency once per object. Fetch up to READ_PREFETCH_DEPTH (8) segments ahead, each on its own task, and handle their outcomes in object sequence order: entries, collected-object skips, corrupted-segment holes and errors surface exactly as before, and dropping the stream aborts the fetches in flight. A read holds at most eight decoded segments besides the one it yields. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): raise object store WAL recovery concurrency to 64 On S3, recovery reads the header and tail of every object, about 57 to 91 ms per object per lane on 4, 8 and 16 vCPUs alike, and its time scales with objects divided by concurrency up to 128 lanes without throttling: about 9,500 objects recovered in 99 s at 8 and 10.5 s at 64. At 64 recovery issues about 1,800 reads per second, a third of S3's request rate per prefix, and holds one 64 KiB tail window per lane, 4 MiB in all. The recovery concurrency tests now scale with the constant. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): share one prefetch byte budget among object store WAL reads Each read fetched up to eight segments ahead on its own, so regions opened in parallel (16 by default) could together hold 128 segments of up to max_batch_bytes each. A read now always fetches the segment it yields next, as before, and fetches a later one only while its length fits in a 64 MiB budget shared by every read of the store. Regions replayed in parallel thus hold one segment each plus at most 64 MiB, however many open at once. Nothing waits on the budget: a read that cannot reserve stops fetching ahead until its next poll. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an object store WAL read whose fetch task panics A fetch task that panicked was re-raised in the reader with resume_unwind, which unwinds through the caller, such as Mito's region opener task, past its cleanup. Return it as a WalSegmentFetch error instead, so the read ends at that object like any other failed fetch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): bound replay prefetch by decoded size held by fetch tasks The prefetch reservation sat in the read's queue, so dropping a read released it at once while an aborted fetch could still hold its buffers until it next yielded. The fetch task now holds the reservation and returns it with its outcome, so it is released with the buffers. A reservation also counted only the encoded segment length, while a fetched segment is held decoded. It now counts the length plus one Record per entry, and the comment states the whole bound. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): round the measured figures in the recovery concurrency comment Keep the magnitudes in the comment; the exact S3 figures belong to the measurement run rather than the code. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): abort replay fetch tasks with tokio-util's AbortOnDropHandle Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): rename the replay prefetch reservation to try_reserve_prefetch_bytes Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): share the bounded polling helper across the object store WAL tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
29d5854e6a |
test(log-store): drive recovery, read, delete and stall failures through the fault matrix (#9488)
* test(log-store): drive recovery, read, delete and stall failures through the fault matrix Add the second half of the object store WAL fault matrix: the failures of the object store access and of the store state that the conditional-create rows did not drive. Each case asserts the failed waiters and their error, whether the store is poisoned, the sequence, both indexed gauges at the failure, the exact movement of every counter and what a read sees after. Recovery: a listing, a trailer fetch and a footer fetch that fail, the catalog rejecting an object after every footer was read, and an object gone by the time it is fetched, below the boundary of the tip (ignored) and at it (construction fails); each leaves the gauges and the recovery metrics untouched. Reads: a segment fetch that fails, without a hole or a read timer sample, and a later read of the region that succeeds. Collection: a delete that fails and is attempted again at the next collection, and a stop that waits for a delete in flight, with the catalog agreeing with the listing in both outcomes. Store state: per-region entry id exhaustion refusing the append without touching the batch it sealed, and the durable stop paths with several creates in flight that succeed, fail transiently or conflict after stop began. Stalled appends released by admission, by a stop, by a terminal error and by the teardown of the actor, the first two timed as a pair. The recovery and read timers observed around parked fetches, twice each, with holds that cannot overlap. The cases use the existing doubles, extended with a failing listing and a failing range read at an offset on RecordingIo and parked range reads on ParkedIo. Case::row is now optional, and the assertion that every row of the failure matrix has a case is unchanged. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): lose a wide object at its header and footer reads in the fault matrix The vanished-object case now runs three ways: a short object read whole, and a wide object deleted before its header and tail window reads or before its separate footer read, each below and at the boundary of the tip. The entry id exhaustion case reads the other region of the sealed batch back, the teardown case asserts the indexed gauges before and after the actor is gone, and two comments that restated their names are gone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): prove the stop-path ordering and share the wide-footer fixture in the fault matrix The case of creates that succeed after stop began keeps object 1 parked while objects 4 to 2 complete, round-trips a command through the actor and asserts that no append is acknowledged and nothing is indexed before object 1 is released. The stop-with-delete case replaces its yields with the same command barrier, queued behind the stop command. The wide-footer fixture moves to the tests module and is shared with the recovery test that fetches a footer longer than the tail window. The gauge helper is named for its side effect and the vanished-object stage enum for what it names. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): keep the four-create scenario doc on its fault matrix case Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
c8ea00b13f |
feat(log-store): add object store WAL metrics and the conditional-create fault matrix (#9475)
* feat(log-store): add object store WAL metrics and the conditional-create fault matrix Export the remaining fifteen metrics of the object store WAL under greptime_logstore_object_store_wal_: the write path timers and object shape histograms, created objects, the indexed object and byte gauges, recovery and read timers, recovered objects, create conflicts and failures, poisoned stores, and stalled appends with their wait. Add a table-driven fault matrix that drives every outcome of a conditional create and of the start object through the store and asserts the failed waiters, the poison, the sequence, both indexed gauges at the failure, the movement of every metric, and what a read sees afterwards. The test fails if a row of the failure matrix has no case. Every test of the store module now holds one lock, since the metrics are process-wide. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): count every conditional create of the object store WAL Count the outcome of the start object's create, and count a create that met a different object as a conflict before the epoch of that object is read, so a failed epoch read no longer hides the conflict. The fault matrix waits until no store actor of an earlier test is alive, checks that a poisoned read returns the recorded terminal error, and covers a conflicting object whose header cannot be read. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
e4872ec1a2 |
feat(log-store): collect object store WAL objects below a persisted reclaim boundary (#9471)
* feat(log-store): collect object store WAL objects below a persisted reclaim boundary Every object header carries a reclaim boundary: every object below it was fully flushed when the object was sealed. The writer computes the boundary from the catalog and the obsolete watermarks after every obsolete and every indexed object, and stamps it on every sealed object and on the start object. The boundary of the last indexed object permits deletion: every present object below it, indexed, listed by recovery off the chain, or left by a failed batch, is deleted in the background, at most four at a time. Recovery judges every candidate with its own boundary: a link to a predecessor below it holds whether or not the predecessor is present, and the chain stops there. A read that listed an object before it was collected skips it instead of failing. The header grows from 46 to 54 bytes; the format version stays 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): scan the reclaim boundary from the previous boundary The boundary was recomputed by scanning every region's indexed objects from the first one, so the flushed objects of busy regions were scanned again on every indexed object while another region held the boundary. Scan the objects in sequence order from the previous boundary instead and stop at the first one holding a live segment. Watermarks never move down and a region first appears above the boundary, so the result is the same and an object is not scanned again once the boundary passed it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): name the parked operation helpers after both their uses Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
7f0f592b6b | chore: update rust toolchain to 1.98.1 (#9429) | ||
|
|
f9ab3c296b |
feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region A segment that does not decode, because its checksum does not match or its content disagrees with its footer entry, is fetched once more. If it still does not decode, the read follows `on_corrupted_segment`: `skip` records a WAL hole for the region with the object key, sequence and the footer's entry id range, increments `greptime_logstore_object_store_wal_skipped_segments_total`, logs a warning and continues with the next object; `fail` fails the read. A failed fetch fails the read in both modes. Holes are kept in memory per region and can be listed from the store. The standalone test builder allocates WAL options through `build_standalone_wal_provider`, and integration tests run a standalone instance on the object store WAL across restarts, on S3 when the `GT_S3_*` variables are set. The standalone shutdown runs every step and returns the first error. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cmd): cover the best-effort standalone stop Build a standalone instance whose leader services fail to stop and whose datanode servers fail afterwards, and check that `Instance::stop` still stops the object store WAL and returns the leader services error. Rename `skip_segment` to `record_skipped_segment` and the file store integration test to `test_standalone_object_store_wal_survives_restarts_on_file`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
8553db3f27 |
feat(mito): wait for WAL durability before publishing a manifest watermark (#9410)
* feat(mito): wait for WAL durability before publishing a manifest watermark A log store that acknowledges appends before their entries are durable can hand out entry ids that a crash loses. Before a flush, a full truncate or a discard of unflushed data records an entry id in the manifest, Mito now waits on `LogStore::wait_durable` for that id. The flush waits after its SSTs are written and observes its cancellation, so a drop or truncate queued behind it is not held up by the upload. Add engine tests of the object store WAL across restarts and of the barrier, and a log store testing hook that stores an object and then reports its create as failed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito): synchronize the WAL barrier tests on store signals The barrier tests waited with fixed sleeps, which do not establish that a flush or truncate has reached its durability wait, and the lost backlog test could stop the store before the seal was handled. The object store WAL gets testing hooks that observe parked creates and durability waits, and one that ends the actor as a crash would; the tests wait on them, crash the store and the engine before a reopen until nothing holds the old store, and cover a truncate or discard of unflushed data that completes once its entry is durable. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito): rename the recovery state and the crash test in the WAL recovery tests Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito): drop a comment that restates the empty recovery state Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
a99cb56d71 |
feat(datanode): wire the object store WAL into the datanode and standalone (#9386)
* feat(datanode): wire the object store WAL into the datanode and standalone A datanode configured with `provider = "experimental_object_store"` now builds an `ObjectStoreLogStore` instead of failing with the placeholder "not supported yet" error. The builder resolves `storage_provider` against the object store manager (empty selects the default store, an unknown name is rejected as an invalid `storage_provider`), opens the store under the node id and the standalone generation, stops it if a later build step fails, and hands it to `Datanode`, whose shutdown stops it after the region server. Datanode shutdown is now best-effort: every step runs and the first error is returned. A datanode with a metasrv client still rejects the provider. The standalone bootstrap builds `WalProvider::ObjectStore` with the same derived node prefix before the metasrv WAL conversion, so the WAL options it allocates match the prefix the store runs under. `log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's test utilities can build an engine on the object store WAL over an in-memory object store, and new engine tests cover create, reopen, replay isolation between regions, batch open, drop and offline cleanup, the latest entry id after replay, and the rejection of object store WAL options on a Raft Engine log store. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): stop every region engine before the object store WAL The region server returned at the first engine whose stop failed, so the engines after it in map order, Mito included, were never stopped before the datanode stopped the object store WAL. It now attempts every engine, returns the first error and logs the later ones. `ObjectStoreLogStore` is also exported from the `log-store` crate root, and its callers import it from there. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
03823a9a01 |
feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the object store WAL config, validated by the datanode and the store. In the `enqueued` mode `append_batch` returns on admission with the entry ids assigned and the object is created in the background. At a backlog threshold the next append is held back until an upload completes. A transient create failure is repeated under the same sequence with the same bytes; any conflicting object poisons the store. `stop` uploads the backlog, or returns the error that dropped it once stop began. `obsolete` clamps the watermark to the durable entry id, and an id the store handed out needs no sequence floor. Add `LogStore::wait_durable` with a default that returns at once. The object store WAL answers it once the region is durable and indexed through the entry id, and fails it after a backlog was lost. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison an enqueued store on permanent create failures Repeat a failed create in the enqueued mode only when the storage error is retryable; any other storage error poisons the store. A conflicting object poisons an enqueued store without reading its epoch, so a failed header read cannot turn the conflict into a retry. A durability wait for an id above the highest id the store handed out now waits for the handed-out ids of the region below it instead of returning at once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison on permanent create failures after stop begins A create that fails with a storage error that is not retryable poisons an enqueued store even after stop began; only a transient failure drops the backlog without poisoning. Durability waiters whose callers stopped waiting are pruned before a new waiter is queued. The backlog age test no longer depends on a follow-up append finishing within the threshold. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): answer a durability wait once no earlier entry is pending A durability wait now returns once the region holds no entry at or below the target that is handed out but not durable, instead of waiting for the largest id handed out to the region. A later object of the region that is still being created no longer holds back a wait whose target it does not cover. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): order the pending durability wait check after the actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(store-api): state that the default wait_durable keeps each store's guarantee The default `LogStore::wait_durable` returns at once, which keeps each log store's own acknowledgement guarantee; Raft Engine with `sync_write = false` acknowledges before its periodic sync, so the documentation no longer claims that every entry id a caller holds is durable. The object store WAL configuration test now also serializes the new options and reads them back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): limit the acknowledgement guarantees to the durable mode Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): repeat enqueued creates that a retry layer marks persistent An object store wrapped in the OpenDAL retry layer reports a temporary error that outlasted its retries as persistent rather than temporary. The enqueued mode now repeats a create after any storage error that is not permanent, so a transient outage behind the retry layer no longer poisons the store and drops the acknowledged backlog; after stop began such a failure still drops the backlog without poisoning. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a persistent create failure after stop begins Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
194bc2fb3c |
feat(log-store): add the object store WAL durable write path (#9320)
* feat(log-store): add the object store WAL durable write path ObjectStoreLogStore can now write. append_batch admits entries into one open batch and assigns object-sequence-major entry ids at admission. The batch is sealed by size, by the flush interval, or before a region would run past the position range, and sealed batches are uploaded with at most four conditional creates in flight, started in sequence order. Created objects are indexed in sequence order, and an append is acknowledged only once its object is durable and indexed. A transient create failure rolls back the failed batch and every later batch unless a later object is already durable, in which case the store poisons itself with a history-gap error. A conflicting object, an encoding or catalog error, or taking the last representable sequence poisons the store. obsolete now goes through the actor and raises the sequence floor together with the watermark, refusing with a retryable error while the next sequence is not settled. stop drops the open batch and the batches whose create has not started, and lets creates in flight finish. Only the durable acknowledgement mode exists. A testing feature exposes hooks to wait for admissions, seal the open batch, and hold or fail creates. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): synchronize object store WAL write tests with the actor Tests that assert nothing happened round-trip a command through the actor instead of yielding the test task, the conflict test waits for the object it expects, and the obsolete-behind-stop test holds the command channel itself so the unanswered command is deterministic. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): bound admitted WAL appends and never reuse a failed sequence A create that reports an error may still have written its object, so the sequences of failed batches are no longer handed out again: the next batch keeps the sequence after the last sealed one and retries are assigned new ids. A retry batched differently can no longer conflict with that object. As the next sequence never moves back, the sequence floor of obsolete only waits for an open batch that has handed out ids. Appends now arrive on their own bounded channel, which the actor stops reading while MAX_SEALED_BATCHES batches wait to become durable, so a stalled object store holds callers back instead of growing the backlog, while stop and obsolete are still handled. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): reserve room for two seals before admitting a WAL append One admission can seal the open batch before an append that would exhaust its positions and then the append's own batch, so the actor takes an append only while two more sealed batches fit under MAX_SEALED_BATCHES. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(log-store): chain object store WAL objects and recover the latest chain A conditional create can fail with an unknown outcome while its object is stored, or still lands later, and Mito reuses the row sequences of a failed append. Replaying such an object next to a later acknowledged one can let the unacknowledged rows win after a restart. Format version 2 gives every object header its writer's epoch, a link to the object it extends (sequence and writer instance) and a header CRC32, and allows objects without segments. Recovery replays only the chain ending at the complete object with the largest epoch and sequence: a link holds when its predecessor is present with the recorded writer instance, or is missing below every present object. Objects off the chain are orphans that are never replayed but keep their sequences. Each open writes an empty object that starts an epoch above every present object, linked to the recovered tip, before it accepts writes, so a late object of an earlier instance never ends the chain. A start object that meets an object of an earlier epoch moves to the next sequence; one of an equal or later epoch fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): break the WAL chain at every missing predecessor Accepting a missing predecessor below every present object lets a late object that lands below the chain change which links hold. Nothing collects objects yet, so a missing predecessor now always breaks the link, and recovery fails when objects are present but none completes a chain. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep the object store WAL format at version 1 The object store WAL has not been enabled anywhere, so no object in the previous layout exists and the chained header can stay version 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): link object store WAL objects by epoch Only one store instance writes under an epoch, since every open starts an epoch above every present object and a start object that meets the same or a later epoch fails the open. The epoch therefore identifies the instance, and the random writer instance id is dropped from the header. A link now records the sequence and the epoch of the object it extends, and holds when the predecessor carries that epoch. The header shrinks to 46 bytes, and the store logs its epoch when it opens. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an open whose start object is already present Without a random writer instance, two opens that recover the same objects encode byte-identical start objects, and a conditional create treats the same bytes as its own retry. A start object that is already present therefore fails the open with a retryable error instead of letting both opens claim the epoch; the next open counts the object and starts a later epoch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): derive the WAL epoch from the claimed start sequence Two opens can recover different listings when a late object lands across a sequence gap between them, pick the same largest epoch plus one, and both create their start objects under different sequences. The epoch of an instance is now one above the sequence its start object claims, so a successful create decides the epoch and no two instances share one. It stays above every epoch recovery listed, and an object that carries an epoch above the next sequence fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): never run a held WAL create after the store is dropped The create test hook ignored the closed hold channel, so a create parked when the store was dropped could still run. It now returns without creating. Drop the per-admission bookkeeping of issued entry ids, which nothing reads, and move the parked I/O documentation to its helper. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a WAL start object stored with an unknown outcome Add a fault-injection test in which the create of the start object stores the object but reports an error: the open fails without moving to another sequence, and the next open counts the stored object and claims a later epoch. Rename the test helper that writes a whole object from a given header to put_object_with_header, and drop a needless clone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): make the held-create drop test deterministic Keep the actor running while the store drops the hold sender, so the parked create always completes on the closed channel instead of racing the actor's exit. Drop a comment that restates epoch_of. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
822042c7bb |
feat(log-store): chain object store WAL objects and recover the latest chain (#9334)
* feat(log-store): chain object store WAL objects and recover the latest chain A conditional create can fail with an unknown outcome while its object is stored, or still lands later, and Mito reuses the row sequences of a failed append. Replaying such an object next to a later acknowledged one can let the unacknowledged rows win after a restart. Format version 2 gives every object header its writer's epoch, a link to the object it extends (sequence and writer instance) and a header CRC32, and allows objects without segments. Recovery replays only the chain ending at the complete object with the largest epoch and sequence: a link holds when its predecessor is present with the recorded writer instance, or is missing below every present object. Objects off the chain are orphans that are never replayed but keep their sequences. Each open writes an empty object that starts an epoch above every present object, linked to the recovered tip, before it accepts writes, so a late object of an earlier instance never ends the chain. A start object that meets an object of an earlier epoch moves to the next sequence; one of an equal or later epoch fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): break the WAL chain at every missing predecessor Accepting a missing predecessor below every present object lets a late object that lands below the chain change which links hold. Nothing collects objects yet, so a missing predecessor now always breaks the link, and recovery fails when objects are present but none completes a chain. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep the object store WAL format at version 1 The object store WAL has not been enabled anywhere, so no object in the previous layout exists and the chained header can stay version 1. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): link object store WAL objects by epoch Only one store instance writes under an epoch, since every open starts an epoch above every present object and a start object that meets the same or a later epoch fails the open. The epoch therefore identifies the instance, and the random writer instance id is dropped from the header. A link now records the sequence and the epoch of the object it extends, and holds when the predecessor carries that epoch. The header shrinks to 46 bytes, and the store logs its epoch when it opens. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): fail an open whose start object is already present Without a random writer instance, two opens that recover the same objects encode byte-identical start objects, and a conditional create treats the same bytes as its own retry. A start object that is already present therefore fails the open with a retryable error instead of letting both opens claim the epoch; the next open counts the object and starts a later epoch. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): derive the WAL epoch from the claimed start sequence Two opens can recover different listings when a late object lands across a sequence gap between them, pick the same largest epoch plus one, and both create their start objects under different sequences. The epoch of an instance is now one above the sequence its start object claims, so a successful create decides the epoch and no two instances share one. It stays above every epoch recovery listed, and an object that carries an epoch above the next sequence fails the open. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a WAL start object stored with an unknown outcome Add a fault-injection test in which the create of the start object stores the object but reports an error: the open fails without moving to another sequence, and the next open counts the stored object and claims a later epoch. Rename the test helper that writes a whole object from a given header to put_object_with_header, and drop a needless clone. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
|
2a63ea2660 |
feat(log-store): add object store WAL reads and obsolete watermarks (#9248)
* feat(log-store): add object store WAL reads and obsolete watermarks Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf(log-store): reuse decoded WAL payload allocations Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): clarify WAL region mismatch error Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
5ccc29ce77 |
feat(log-store): add object store WAL store construction and recovery (#9238)
* feat(log-store): add object store WAL store construction and recovery Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): derive WAL store prefix from node and generation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): describe unsupported operation boundary Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
9c2933de0b |
feat(log-store): add the object store WAL batch, catalog and I/O (#9216)
* feat(log-store): add the object store WAL batch, catalog and I/O Add the three modules between the object format and the store of the object store WAL: - batch: the open batch that accumulates admitted entries into the next object and assigns entry ids at admission. Ids are object-sequence-major, `(object_seq << 20) | position`, with positions starting at one per region per object, so a batch that is rolled back and admitted again under the same sequence hands out the same ids. The time the first entry was admitted is kept for the store's age-based sealing; an empty admission does not start it. - catalog: the in-memory index over object footers, by sequence and per region. Insertion is atomic and rejects an empty footer, duplicate region segments, invalid entry ranges, an already indexed sequence and entry ranges that are not strictly increasing across objects. The next object sequence continues after the largest indexed one and is raised above the largest entry id of any region, and is rejected once it no longer fits an entry id. - io: object store access under `<prefix>/objects/`: a conditional create whose retry with identical content is a no-op and whose conflicting content is rejected, whole and range reads, and a listing of well-formed keys in sequence order. A prefix with a `.` or `..` component is rejected. When the store reports that the object exists but the read that compares its content fails, the read failure is returned with its retry hint instead of the create failure. The modules have no callers until the store lands, so they are declared with `#[allow(dead_code)]`. Only `entry_id` is exported. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): drop comments that restate the batch test assertions Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): use pub(crate) in the object store WAL batch, catalog and I/O Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): keep entry_id crate-private and narrow the create collision check Re-export `entry_id` as `pub(crate)`: nothing outside `log-store` uses it yet, and exporting it would freeze the raw `(object_seq, position)` encoding before the store owns id allocation. Treat only `ConditionNotMatch` as the sign that a conditional create collided with an existing object. A store may report `AlreadyExists` for an unrelated path, for example when a parent of the object is a file, and that failure must come back as the write failure rather than as the error of the read that would compare content. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): rename the catalog's out_of_order helper to out_of_order_reason Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
d45f5d6eaf |
feat(log-store): add the object store WAL object format (#9200)
* feat(log-store): add the object store WAL object format Add the byte format of a single object store WAL object: a header with the GTWALOBJ magic, format version 1, the object sequence and the writer instance id; one segment per region ordered by region id with entries ordered by entry id; a footer that records each segment's region id, entry id range, entry count, byte range and CRC32; and a fixed trailer with the GTWALTRL magic, the footer location, the footer CRC32 and the whole-object CRC32. The module encodes objects deterministically and decodes the header, trailer, footer and segments separately, with structural checks on footer ranges and segment tiling. It has no callers yet; the store that writes and reads objects follows in later changes. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): pin the object store WAL format with a byte fixture Add a fixed version 1 object with two regions and five entries as a hex literal. The test decodes it and checks the exact header, trailer, footer entries and records, and checks that encoding the same records, in either input order, reproduces the fixture byte for byte. Round-trip tests alone pass when a refactor changes field order, endianness or checksum coverage in both the encoder and the decoder. The fixture bytes were derived from the documented layout rather than from the encoder, so such a change now fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): reject an empty footer when decoding a WAL object The encoder never writes an object without records, but decode_footer accepted a footer that declares zero segments, and verify_segment_ranges accepts an empty footer too. Only the test-only decode_object rejected it, so a checksum-valid empty object would pass the header, trailer and footer checks that recovery runs. Reject a zero entry count in decode_footer and drop the now unreachable check in decode_object. Add a test that builds a checksum-valid object with an empty footer and checks that decode_footer and decode_object reject it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(log-store): use pub(crate) for the WAL object format API Other log-store modules use pub(crate) for items shared across module boundaries. Switch the format module from pub(super) to pub(crate) to follow that convention. No behavior change. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
4c12ea1aba |
chore(deps): bump opendal to 0.58.1 (#8742)
* chore(deps): bump opendal to 0.58.1 Upgrade direct opendal dependency and workspace object_store_opendal pin from 0.57 to 0.58 (lockfile resolves opendal 0.58.1 / object_store_opendal 0.58.0). Adapt to OpenDAL 0.58 composition API: - Operator::new returns a finished operator; drop .finish() call sites - Replace HttpClientLayer / raw::HttpClient with OperationContext + HttpTransporter (ReqwestTransport) - Migrate SecureFsBackend and MockLayer from Access/LayeredAccess to Service + Layer::apply_service - Rewrite SecureFs reader/writer/lister for sync factories and StreamRead - Use OperatorInfo::capability() instead of removed native_capability() Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: retrigger CI after udeps runner segfault Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(object-store): restore suffix read simulation for secure fs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: adapt remaining callers to OpenDAL 0.58 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: Xuanwo <github@xuanwo.io> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: WenyXu <wenymedia@gmail.com> |
||
|
|
a932433d21 |
fix(wal): bound Kafka requests and extend latency buckets (#9026)
* fix(wal): bound Kafka requests and extend latency buckets Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(wal): update rskafka request timeout revision Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(config): document Kafka WAL timeouts in MetaSrv Signed-off-by: WenyXu <wenymedia@gmail.com> * style: sort common-wal dev dependencies Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
d4af650ec0 |
perf: reduce cold workspace compile time (#8801)
* refactor: remove datanode and meta-srv dep from frontend Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: use on-device protoc if possible Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor: remove unused dep Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: version and docs Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: update logs Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> |
||
|
|
912db22417 |
fix(log-store): deduplicate Kafka WAL multipart records (#8514)
* fix(log-store): deduplicate Kafka WAL multipart records Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): preserve delayed Kafka WAL entry offsets Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): reject conflicting Kafka WAL last records Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): emit Kafka WAL entries on last record Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): discard duplicate Kafka WAL first records Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
56e9158819 |
feat: prepare soft-drop WAL retirement (#8475)
* fix: flush soft-dropped regions on close Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): handle flush-on-close race with concurrent in-flight flush When a region close with `flush_on_close: true` races with an already-running flush, pass the actual close request (including the flush_on_close flag) to the DDL handler instead of a default request so the pending flush is correctly awaited. Files: `src/mito2/src/worker/handle_close.rs` Also adds a test verifying that closing with flush-on-close while a flush is in progress still persists all written data correctly. Files: `src/mito2/src/engine/close_test.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: finish close request callsites Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: replace imperative close-region loop with functional combinators Transform the region close dispatch in `DropTableExecutor` from mutable `Vec` and `push` loops to iterator chains with `join_all`, improving idiomatic Rust style and readability. - `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode region-close logic to use `peers.map()` and nested `join_all`, moving `node_manager.datanode()` inside the closure to align with the new structure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: decouple Kafka client from WAL checkpoint Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: merge Kafka WAL index checkpoints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: delegate Kafka WAL retirement to metasrv Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: rebase main and resolve conflicts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: license header Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump proto to commits on main Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove Kafka obsolete-all index changes Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
fbdc35a109 |
ci: add remote WAL logical pruning fuzz (#8307)
* feat: configure remote wal checkpoint intervals Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add logical pruning fuzz target Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: add remote WAL logical pruning fuzz Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
e520ff8300 |
feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate retry hints in core errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: propagate retry hint through RPC metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fallback retry hints to status codes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: add explicit retry hints for retryable errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove status code retry fallback Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: classify io retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints across error wrappers Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minior Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve datasource retry hints in query Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve metric engine retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve flow and frontend retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: share rskafka retry hint mapping Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
e14404c677 |
chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21 * chore: new format * fix: lint * chore: resolve lint issues * chore: remove as_millis_f64 * chore: deps up |
||
|
|
4d66bd96b8 |
feat: make distributed time constants and client timeouts configurable (#7433)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
cb0f1afb01 |
fix: improve network failure detection (#7382)
* fix(meta): add default etcd client options with keep-alive settings (#7363) * fix: improve network failure detection (#7367) * Update src/meta-srv/src/handler.rs Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> --------- Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> |
||
|
|
3119464ff9 |
feat: introduce the Noop WAL provider for datanode (#7105)
* feat: introduce noop log store Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update config example Signed-off-by: WenyXu <wenymedia@gmail.com> * test: add noop wal tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
8fe17d43d5 |
chore: update rust to nightly 2025-10-01 (#7069)
* chore: update rust to nightly 2025-10-01 Signed-off-by: luofucong <luofc@foxmail.com> * chore: nix update --------- Signed-off-by: luofucong <luofc@foxmail.com> Co-authored-by: Ning Sun <sunning@greptime.com> |
||
|
|
c9377e7c5a |
build: bump rust edition to 2024 (#6920)
* bump edition Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * gen keyword Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime and env var Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * one more gen fix Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * lifetime of temporaries in tail expressions Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * format again Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy nested if Signed-off-by: Ruihang Xia <waynestxia@gmail.com> * clippy let and return Signed-off-by: Ruihang Xia <waynestxia@gmail.com> --------- Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
|
7cf47ccf54 |
fix: initialize remote WAL regions with correct flushed entry IDs (#6856)
* fix: initialize remote WAL regions with correct flushed entry IDs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add logs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: correct latest offset Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: update sqlness Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add replay checkpoint to catchup request Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: logs Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
d57c0db9e6 |
fix: no need to early lookup DNS for kafka broker (#6845)
Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
566a647ec7 |
feat: add replay checkpoint to reduce overhead for remote WAL (#6816)
* feat: introduce `TopicRegionValue` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: persist region replay checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: introduce checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: udpate config.md Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minor refactor Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: send open region instructions with reply checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: use usize Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add topic name pattern Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: enable wal prune by default Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
8894cb5406 |
feat: resolve unused dependencies with cargo-udeps (#6578) (#6619)
* feat:resolve unused dependencies with cargo-udeps (#6578) Signed-off-by: Arshdeep54 <balarsh535@gmail.com> * Apply suggestion from @zyy17 Co-authored-by: zyy17 <zyylsxm@gmail.com> * Apply suggestion from @zyy17 Co-authored-by: zyy17 <zyylsxm@gmail.com> --------- Signed-off-by: Arshdeep54 <balarsh535@gmail.com> Co-authored-by: Ning Sun <classicning@gmail.com> Co-authored-by: zyy17 <zyylsxm@gmail.com> |