mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 23:06:59 +00:00
3dce3be163ad4fa20bddd3533bd6ebd2c525bbef
795
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 shuiyisong
|
a57ab18eaa |
feat(ingest): report confirmed progress for batched writes (#9921)
* feat(ingest): track confirmed writes through batching and retries Batched writes can partially reach storage before another partition fails. Callers need the successful row count and the failure together to report accurate ingestion results. Add WriteProgress and a confirmed-write context, carry row ownership through partitioning and retry rebuilds, and retain completions through ordinary and logical-table batchers. Existing callers retain the completion/result wrappers. Tests cover partial progress, request ownership, batching and retries. This supplies the accounting required by Remote Write exemplar written-count headers. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(servers): coalesce flow notifications Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(ingest): reduce write progress overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: preserve retry hints across error wrappers Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): drain confirmed writes on cancellation Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(batcher): reduce row owner allocations Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(batcher): restore per-flush metrics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * revert: scope retry-hint fix to storage writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): restore flush metrics and benchmark Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(ingest): propagate sync batch write errors Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: preserve peer batching for confirmed writes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf: avoid unnecessary row owner copies Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf: reuse IPC buffers in tracked retries Signed-off-by: shuiyisong <xixing.sys@gmail.com> * style: order write dispatcher imports Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: rebase main Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
 jeremyhi
|
c40e86c653 |
fix(datanode): pick a node epoch above the recorded one (#9984)
* fix(datanode): pick a node epoch above the recorded one The metasrv fences a datanode whose node epoch is lower than the highest one it has recorded for the node id. A datanode used its start time as its epoch, so after a restart on a host whose clock is behind the previous process's, or after the clock stepped back, the new process picked a lower epoch and was fenced for good: no region lease, no keep-alive. Before it creates the heartbeat task, the datanode now reads the recorded epoch through its metadata backend and uses one more than it if its start time is not higher. The node info still reports the local start time. A failed read fails the startup, like the other metadata reads. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(datanode): tighten the node epoch tests Use a lower record that leaves room below the start time, so the test fails if the start time is ignored, and a nonzero node id, so it fails if the node id is ignored. Clarify the doc comment of the epoch choice. This assumes the previous process of a node id has stopped before a new one starts. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 zhaiAohan
|
8a38b90135 |
feat: add relational region engine skeleton and registration (#9953)
* feat: add relational region engine skeleton and registration Add a read-only relational RegionEngine (create/open/drop lifecycle) and register it as the 4th engine in build_store_engines alongside mito/metric/file. Add a datanode-level test asserting all 4 engines are registered. Signed-off-by: tian1220A <1573612521@qq.com> * feat: add relational connection and idempotent region lifecycle Move ConnectionInfo into relational-engine (DSN parsing + redacted rendering), place the RelationalOptions/RELATIONAL_META_KEY serialization contract in table::requests, make CREATE idempotent by comparing the persisted manifest definition against the incoming request (RegionExists on mismatch), and drop the empty EngineConfig placeholder. Signed-off-by: tian1220A <1573612521@qq.com> * refactor: address review comments on relational region engine - remove connection.rs (only used by its own tests); it will be reintroduced when connection handling is wired up - drop RelationalOptions::version (not validated or used to select a deserialization format) - validate an idempotent create against the existing region definition (region id / schema / primary key / options) and return Unexpected on mismatch, with a reason field on RegionExists - use serde.workspace and remove the empty [features] section - cargo fmt Signed-off-by: tian1220A <1573612521@qq.com> * fix: adapt relational engine to latest main - implement RegionEngine::set_region_role_with_lease (no-op lease, the engine takes no writes so no lease is needed) - pass node_id to build_object_store_manager in the registration test - drop stale url dependency from the relational-engine lock entry Signed-off-by: tian1220A <1573612521@qq.com> * ci: register relational-engine in scope rules Signed-off-by: tian1220A <1573612521@qq.com> * fix: remove table self-dependency from Cargo.lock Signed-off-by: tian1220A <1573612521@qq.com> --------- Signed-off-by: tian1220A <1573612521@qq.com> |
||
 Dragon Roar
|
b21cd16f1d |
feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup Signed-off-by: lczllx <2181719471@qq.com> * fix(datanode): harden the startup object storage self-test Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): cover the File store skip in the object store self-test Signed-off-by: lczllx <2181719471@qq.com> * refactor(datanode): rename the object storage self-test to probe Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): disable object store self-test in sqlness and integration tests Signed-off-by: lczllx <2181719471@qq.com> * docs(datanode): correct what the object storage probe claims to cover Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
|
jeremyhi
|
89a5df3141 |
fix: ignore stale leader grants after a region is downgraded (#9987)
* fix: ignore stale leader grants after a region is downgraded Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: compare stale leader grants with the longest lease seen Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: treat a lease too long to store as unknown A lease longer than a u64 of nanoseconds saturated to a shorter one, so the derived send time could be later than the real one. Now such a lease makes the send time unknown, which counts as earlier than any gate. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address review comments on stale leader grants Scope the early send time claim to after a lease has been seen, drop the unused role parameter of demote, clarify a test comment and check that a fresh grant still makes a reopened region writable. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: compare stale leader grants with the heartbeat send time Pass the send time of the heartbeat behind a lease to set_region_role_with_lease, and return whether the role was applied. Mito compares the send time with the grant gate directly, so it no longer derives it from the longest lease seen. The datanode finalizes leadership only when the grant was applied. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover stale grants through the datanode and repeated downgrades Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: keep leader grant rules in one region method Apply a role grant in MitoRegion::apply_role_grant, which renews the lease and decides whether the role changes; the engine only looks up the region. Rename the gate to downgrade_started_at and demote to begin_downgrade, keep the field private, and copy it on reopen through inherit_downgrade_state_from. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: drop a needless borrow Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: keep lease concepts in the engine and role ordering in the region ManifestContext, which owns the role, records when the role was last demoted and promotes a region only for a promotion issued after that. The time starts when the context is created, so a reopened region ignores grants for heartbeats sent before the reopen, without copying state. The engine translates leases: it renews the deadline and promotes with the heartbeat send time. MitoRegion keeps no grant state. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep the data region of a metric region off a grant its metadata region ignores The metric engine applies a grant to the metadata region first and stops if it is ignored, so the data region, whose role the engine reports, never becomes a leader that skips finalization. Also describe what promoting returns. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs: describe when a lease grant is ignored and flatten the finalize check Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: move lease types out of the region and the Leader deadline into ManifestContext Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: name the promotion fence after the downgrade it records Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9
|
bd77b599cc |
refactor(flow): remove tracked-write protocol (#9992)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
91a59b82fb |
fix: acknowledge writes only within the region's leader lease (#9904)
* fix: acknowledge writes only within the region's leader lease Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: drop cached logical columns after a failed logical drop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: keep the prepared region for the lease check after the WAL append Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: serialize a logical drop with fills of the logical column cache Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: check the drop lock and grant long leases deterministically Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: cover a bulk insert that loses its lease during the WAL append Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Weny Xu
|
c96cdd0026 |
perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): stream and prune manifest SST metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(datanode): stream inspection queries through reusable providers Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(catalog): push manifest filters through inspection queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: align manifest column constants with module conventions Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: reduce manifest inspection allocations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: narrow manifest inspection API visibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: retain residual filters for manifest inspection Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 Dhruv VaishnavandWeny Xu
|
2e37e19af4 |
feat(datanode): expose ingest task panic and cancellation metrics (#9830)
* feat(datanode): record ingest task join failures Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(datanode): remove extra ingest task metric tests Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
  NightG1owZandNightG1owZ
|
b886ae525d |
feat(config): support overriding region engine options from env (#9700)
* feat(config): support overriding region engine options from env `region_engine` is a `Vec` of externally tagged enums, so plain dotted environment keys cannot address it: `config` walks the path, replaces the default sequence with a table and loading fails with `invalid type: map, expected a sequence`. Address the entries by tag name instead (`<PREFIX>__REGION_ENGINE__<TAG>__<FIELD>`) and merge the overrides onto the default entries in a dedicated source, so the engines and fields that are not mentioned keep their configured value. An entry the default sequence does not have is appended; a numeric index, an override of the whole sequence, a field below a non-table default and two variables that set a path and a field below it are reported with a dedicated error instead of a serde type error. A configuration file that configures the sequence as a whole still replaces the merged value, which the example configs document. Closes #8620 Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * docs(config): clarify that the metric engine is always enabled `region_engine` defaults to [mito, file], but the datanode builds and registers the metric engine unconditionally. Reword the example configs so `mito` and `file` read as the default configuration entries and naming `metric` only supplies its options, then regenerate config.md. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * docs(config): drop the env-var explanation from the config reference Environment variables are documented in the GreptimeTeam/docs repository, so the example configs and the generated config reference only describe the config file. Keep the note that the metric engine is always registered. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> * test(config): cover tagged sequence environment collection Add unit tests for TaggedSeqEnvSource::collect: merging by tag, appending unknown tags, empty and null defaults, flat key removal, the error paths and the take_entries and is_index_like helpers. The variables are injected through Environment::source, so the tests do not touch the process environment. Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> --------- Signed-off-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> Co-authored-by: NightG1owZ <316247938+NightG1owZ@users.noreply.github.com> |
||
|
Weny Xu
|
0dc649e028 |
fix(datanode): release region map guard before awaiting role changes (#9824)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
f866ef2bd9 |
feat: return per-entry region write failures (#9728)
* feat: return per-entry region write failures Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints for single-entry region writes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve region write error compatibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: simplify region partial result handling Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align partial failure summaries with input order Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate region write response handling Signed-off-by: WenyXu <wenymedia@gmail.com> * test: fix partial result expectations and region map guard Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: limit lockfile changes to greptime proto Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: make partial region results an explicit client opt-in Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin greptime-proto to merged partial-result changes Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
7397ab6b23 |
perf(datanode): send the mito puts of one request to each worker at once (#9692)
* perf(datanode): send the mito puts of one request to each worker at once The region server handled the puts of a request to several regions as separate requests. A Mito worker woke on the first of them and wrote it alone; in the durable object store WAL mode its append then blocked the worker until the next seal, so the rest of the request went into the next object, and clients writing a partitioned table locked into one phase that wastes a tick. When every put of a request goes to a ready mito region, the region server now hands them to `MitoEngine::put_regions_batch`, which submits the puts of each worker in one message, so the worker writes its regions of the request in one batch and one WAL append. Requests to one region and to other engines are handled as before. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): await each region of a batched mito put on its own task The batched path waited for the regions of a request in order and sent the worker messages one after another, so an error or a full worker queue of one region held back the others, and its metrics counted the request as a whole. `MitoEngine::put_regions_batch` now sends the worker messages concurrently on their own task and returns each region's result. The region server awaits and accounts every region on its own ingest task, as the per-region path does: the first error returns at once, and each region records its latency, changed rows or failure and is named in its error. A request with several puts to one region keeps the per-region path. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): spawn the accounting of batched mito puts from the batch task The request spawned the per-region accounting tasks only after the batch task returned, so a request dropped in between left accepted writes unaccounted. The batch task now spawns them and the request only joins their handles. Also state in the architecture invariants that the region server's batched puts to the metric and mito engines are the one request dispatch that does not go through the RegionEngine trait. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch mito puts through the RegionEngine trait Add `RegionEngine::handle_batch_put_requests`, which returns each region's response of the puts of one request. The default handles each put with `handle_request`, so other engines keep their behaviour; Mito overrides it with the one message per worker submission. The region server calls the trait method instead of downcasting to `MitoEngine`, and the architecture invariants no longer need an exception. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): trace each region of a batched mito put Attach the per-region `RegionServer::handle_region_request` span of the per-region path to each region's task of a batched mito put, so the region ids stay in distributed traces. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): let a batch put response answer for several regions `RegionEngine::handle_batch_put_requests` now returns each response with the regions it answers for. The default and Mito answer each region on its own; an engine that writes several puts together, such as the metric engine's batched puts, can answer them with one response. The region server accounts a response for each of its regions. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): batch any request with several mito puts Take the batch path for a request with at least two puts to ready mito regions, also when they all target one region: a request of many puts to one region otherwise still sends one worker message per put. Each put keeps its own response, affected rows and error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * perf(datanode): account batched mito puts in one task Await the responses of a batched mito put in the batch task with a FuturesUnordered instead of one spawned task per put. The task accounts each response as it completes, hands the first error back to the caller at once and keeps accounting the rest, so a request of many puts no longer spawns a task per put. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): name the batched worker message BatchWrite Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(mito2): submit batched writes on the ingest runtime Grouping and submitting the writes of a request is part of ingestion, so the task runs on the ingest runtime and takes the write task class when the workload scheduler is enabled. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): share ingest result accounting across put paths The per-region path, the metric batch path and the mito batch path each recorded the changed rows or the failures of an ingest response. They now call one helper, so a metric or label change happens in one place. A batch put response lists the region of each put it answers for, so the failure count stays per put when two puts go to one region; the trait documentation now says so. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * fix(mito2): give batched puts of a stopped worker the worker's error A batched write message that could not reach a stopped worker was dropped, so its puts failed with "Failed to recv result" while a put submitted on its own fails with "Worker <id> is stopped". The worker now returns the message it could not take, and each of its puts receives the WorkerStopped error. Signed-off-by: jeremyhi <jiachun_feng@proton.me> * refactor(datanode): look up the batch latency histogram once The label of the request latency histogram is the same for every put of a batch, so the batch task looks it up once instead of per observation. Signed-off-by: jeremyhi <jiachun_feng@proton.me> --------- Signed-off-by: jeremyhi <jiachun_feng@proton.me> |
||
|
jeremyhi
|
cca95b2637 |
fix(datanode): stop the object store WAL before the region server (#9639)
* fix(mito2): cancel flushes waiting for the WAL when the engine stops A flush waits for the WAL to be durable through the entry id it records before it writes its manifest edit. With the object store WAL in the `enqueued` mode that wait lasts until an upload succeeds, while the log store is stopped only after the engines. A graceful shutdown during an outage of the WAL's object store therefore never finished: closing a region does not cancel its running flush, stopping the worker group waits for running flushes and runs the queued ones, and a region holding writes that skipped the WAL is flushed on close. The worker group now owns a cancellation token, raised before its schedulers stop and kept raised. `RegionEngine::begin_stop`, which the region server calls before it closes regions, raises it earlier, so that a close-time flush cannot wait either. A flush polls its durability barrier first and gives up at the barrier once the token is raised: it writes no manifest edit, deletes its SSTs and fails as cancelled. A flush whose entries are already durable commits as before. The region server reports a failed close, which loses the region's writes that skipped the WAL, in its stop result. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): wait for the outage on the store under test Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): give a healthy WAL a grace period before a stopping engine cancels flushes A flush of a stopping engine no longer gives up its wait for the WAL as soon as the stop begins. It gives up at once while the log store reports that it fails to make entries durable, through the new `LogStore::is_failing`, and otherwise 5 s after the engine began to stop, so a healthy object store WAL uploads its last entries first and a close-time flush keeps the writes that skipped the WAL. The object store WAL reports that it fails while the last create of its backlog failed, which covers a refused and an unreachable object store, and once it is poisoned or stopped. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drain the WAL for a stopping flush and cancel its failed waits A flush of a stopping engine now asks the log store to drain before it waits out the grace period, through a new `LogStore::begin_drain` that does nothing by default. The object store WAL then seals its open batch at once and, until it stops, as soon as a durability wait needs a region it holds, so a healthy store makes the entries durable after about one upload whatever its `flush_interval`, which has no upper bound. A barrier wait that fails while the engine stops, for instance on a poisoned or stopped log store, now fails the flush as cancelled like a wait that gives up. The healthy-store test no longer depends on the seal timer: its store never seals on its own, and the close must finish well within the grace period. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drain the WAL when the engine begins to stop The drain started only when a stopping flush reached its durability barrier, so a close behind slow flushes that reached it after the grace period had no time left to upload its entries. The worker group now asks the log store to drain as soon as it begins to stop, and the object store WAL seals its open batch then and every later admission right away, so its entries become durable while the flushes still write their SSTs. `LogStore::begin_drain` becomes synchronous for that. The healthy-store test now writes after the stop began and closes the region only after the grace period. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): refuse appends once the object store WAL drains A WAL write admitted late in a stop could still be uploading when the close-time flush of its region reached an expired grace period, so the writes that skipped the WAL in that region were lost although the store was healthy. A draining object store WAL now refuses every later append, also one held back, with the stopped error, so the entries a stopping engine waits for are fixed when the stop begins and become durable while the regions still close; writes that skip the WAL are still taken. Tests cover a close-time flush whose wait completes within the grace period once a parked create is released, a close that reaches its wait after the grace period, and a drain refusing a held-back append. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): word a failed region close neutrally in the shutdown log A close can fail for reasons other than its flush, such as a stopped worker, and for regions without writes that skipped the WAL, so the error line no longer states that such writes were lost: it says they may be. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): stop the object store WAL before the region server The datanode stopped the object store WAL only after the region server, whose stop waits for flushes that wait for the WAL to become durable, so a shutdown never finished while the WAL's uploads kept failing. The WAL now stops first: its stop uploads the backlog or drops it and answers every wait for its entries, so a running, queued or close-time flush either commits or fails with the WAL-stopped error without writing a manifest edit. This replaces the stop signal and grace period the engine used to give up those waits, and the log store and engine hooks they needed. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(datanode): check rows after a healthy shutdown and wait for the WAL stop Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: keep only the shutdown tests that prove the new stop order The engine-level tests pinned the log store's stop contract, which the log store's own tests already cover, and the stalled-create test asserted a limitation rather than a guarantee. Drop them with the testing hooks only they used; the datanode and region server tests cover the change end to end. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
778f4556a8 |
feat: unify query options across SQL, HTTP and gRPC (#9494)
Isolated rebase verification of PR #9494. Preserve the reviewed net change; original PR branch remains untouched. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
c3469fad51 |
fix(flow): retain tracked write completion across transient failures (#9525)
* fix(flow): retain tracked write completion across transient failures Use receiver-issued single-write capabilities at the frontend and datanode. Rejoin exact admitted writes across unknown transport outcomes, then close and drain before allowing subsequent rounds. Keep ordinary clients and reads on their existing paths. Separate schema map equality from encoded row identity so fresh protobuf decodes replay consistently even with NaN values. Cover receiver admission, owner cancellation, all-child drain, and retired capabilities. Local feature-worktree checkpoint only. Main verified core enterprise check and 484 tests, plus DN enterprise check and all four tracked owner tests. Corrected seven-case timer acceptance remains to be rerun; this checkpoint does not authorize publishing or integration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): preserve upstream dependencies after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): fail fast on definitive tracked-write open rejection A tracked write treats every application-level error from the frontend as an ambiguous transport failure and retries the same pinned frontend forever. During a rolling upgrade an older frontend does not recognize the tracked-write command extension and processes Open as an ordinary empty SQL query, rejecting it with InvalidSql (InvalidArguments); a non-internal endpoint rejects the extension outright. Both are definitive rejections that no same-frontend retry can ever resolve, so the flow stalls. Classify the Open-phase errors by status code: a definitive rejection (Unsupported / InvalidArguments / PermissionDenied) fails fast with a terminal pre-write error, while genuinely ambiguous transport failures (Unavailable, timeout, connection reset, ...) keep the same-capability retry. Write/Close rejections are deliberately NOT failed fast: once a write may have been admitted, a same-code rejection (e.g. WriterLost, or UnknownCapability after a VIP moved Open->A/Write->B) leaves the outcome ambiguous, and failing fast would let the caller replay and duplicate rows. Add a regression test that a legacy empty-SQL Open rejection fails fast with a single attempt and surfaces as Error::External carrying a TerminalError. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(flow): preflight tracked writes without submitting data Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(flow): retain tracked command extensions while decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
f690bbb204 |
refactor!: remove datanode remote WAL index (#9563)
* refactor: remove datanode remote WAL index Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(log-store): update new WAL replay tests after index removal Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
684c8ff291 |
feat(client): support region request reuse and route refresh hints (#9540)
* feat(client): support shared region requests without deep cloning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate route refresh retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * style: preserve local error variant imports in retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): distinguish staging rejection from write backpressure Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): preserve legacy retry hints in instruction replies Signed-off-by: WenyXu <wenymedia@gmail.com> * test(client): inline retry hint compatibility cases Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): align batch-open retry hint assertion Signed-off-by: WenyXu <wenymedia@gmail.com> * style: use imported types in region request tests Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
28697c9e55 |
feat(datanode): execute nested MergeScan plans on the datanode (#9206)
* feat(datanode): execute nested MergeScan plans on the datanode Give a datanode the ability to decode and execute a plan whose MergeScan carries a nested plan (e.g. an inner MergeScan over a build table), so a distributed sub-plan pushed to a datanode can itself fan out to other datanodes. This is the execution/codec capability only; it is not wired to any user-facing switch and produces no plan by itself. Codec (src/query/src/query_engine/default_serializer.rs): - MergeScanAwareSerializer keeps the request's SessionState and the engine catalog manager, and decodes a MergeScan payload's tables by (catalog, schema, table) name through the engine catalog rather than the region-aware request catalog list (which binds every table to the request's region and breaks on mismatched columns). Datanode (src/datanode): - New DatanodeRegionQueryHandler (region_query.rs) serves region queries for the MergeScan nodes of a received plan, resolving region leaders via PartitionRuleManager and dispatching to peer datanodes. - Wire KvBackendCatalogManager + PartitionRuleManager + NodeClients into the datanode's catalog/cache registry (with_dist_planner stays false: a datanode never rewrites its own plan). Bug fixes uncovered while wiring this up: - cache registry: register the table/table_info/table_name/table_route/ partition_info caches the datanode's catalog manager and partition rule manager require (previously missing -> panic on lookup). - default_serializer: decode_sub_plan bridge no longer drives a future on a current-thread runtime from within another executor (spawn a dedicated thread + runtime when not on a multi-thread runtime). Tested by query_engine::default_serializer unit tests: nested MergeScan round-trip and engine-catalog resolution with mismatched column names. No user-facing behavior change. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep GreptimeDB function bindings after rebuilding decode state SessionStateBuilder::new_from_existing collects the function registry (HashMap) into a Vec and build() re-registers each entry; register_udf registers aliases before the function's own name and the Vec order is unstable. Rebuilding the session state to install MergeScanAwareSerializer after the GreptimeDB functions were registered could therefore rebind a name/alias (e.g. date_format, an alias of DataFusion's to_char) back to the built-in function. This path runs for every decoded plan, not only nested MergeScan ones. Extract the GreptimeDB function registration into register_greptime_functions and re-apply it after the final build of both the top-level decode state and the MergeScan payload state. Add regression tests asserting date_format stays bound to the GreptimeDB implementation on both paths. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): address review on nested MergeScan capability Three follow-ups from review of the nested MergeScan execution capability: 1. Layer the datanode caches (P2b). table_cache derives from table_info_cache + table_name_cache, and partition_info_cache derives from table_route_cache, but they were all registered in a single flat CacheRegistry whose invalidate() runs concurrently with no ordering. A derived cache could be invalidated and refilled from a base cache that had not been invalidated yet, leaving stale metadata. Build the base caches and the derived caches as separate layers of a LayeredCacheRegistry (reusing the existing layering mechanism) so base caches finish invalidating before derived ones. Both the production path (cmd) and the test scaffolding now build the same layered registry; the flat build_datanode_cache_registry is removed. 2. Don't let a nested MergeScan's internal region query compete for the concurrency permit (P2a). An outer query holds its permit until its stream is fully consumed, and a nested MergeScan makes the datanode issue region queries to peer datanodes as execution stages of that same query. If such an inner stage also took a permit, the outer query waiting on it would deadlock against its own held permit (or time out when max_concurrent_queries > 0). Mark inner DN->DN region queries as internal stages: QueryRequest gains an flag carried over the wire in the request header's query_context extensions (reserved key, not injectable via hints), and the region server skips acquiring a concurrency permit for internal-stage requests on both acquire sites. Outer queries still acquire normally. 3. Add an end-to-end capability test (tests-integration/.../nested_merge_scan_capability_test.rs): hand-build Join(<local probe region>, MergeScan(build regions on both datanodes)) with mismatched column names and duplicate build keys, ship it to a datanode over a real TCP Flight endpoint, consume the stream fully and compare the row multiset against the frontend's reference join result, assert a real cross-datanode DoGet via an RPC counter, and assert an inner region failure fails the whole query. Also migrate the with_real_datanode_grpc_addr + DatanodeRpcStats test scaffolding. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt date_format regression test to the updated DataFusion fork ScalarUDFImpl's trait bounds changed so its impl can no longer be downcast via as_any(); assert the function equals the GreptimeDB implementation instead, which to_char can never satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover the internal stage under the concurrency limit Add an end-to-end regression test for the nested MergeScan capability under max_concurrent_queries = 1: both datanodes' only permit is held by the outer queries' streams while their inner MergeScans fetch the build regions from the same datanodes over real Flight connections. The result consumption is bounded by a timeout, so a broken link anywhere in the chain (marking the inner request internal, carrying the marker in the request header, honoring it at the receiving acquire sites) fails the test instead of deadlocking. Also add a minimal with_datanode_options_override hook to the cluster builder, and fix the test module doc: the outer request is handed to RegionServer::handle_remote_read in process; only the inner region queries go over the wire. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): cover cold route loads under unrelated invalidations The table route cache uses InitStrategy::VersionChecked, whose version counter belongs to the whole CacheContainer rather than to a single table: any TableId invalidation bumps the shared version, so a cold load of one table's route retries when an unrelated table is invalidated concurrently. Document this as a known limitation on the region query handler, and pin the behavior with an integration test: run the nested plans while a background task invalidates an unrelated table id on every datanode, assert the results still match the expected multiset (retries only add work, they never return a stale route), and report the observed load amplification via the cache miss counter (2 loads without invalidations, 8 over three runs with 24 unrelated invalidations in this run). Also expose the datanode cache registries on the test cluster so a test can drive the local invalidation path that the heartbeat handler drives after a metasrv broadcast. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(query): trim nested plan decoder commentary Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: remove redundant nested scan support plumbing Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): decode nested MergeScan payloads asynchronously Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): avoid unnecessary extension name allocation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain extension decoding for manual session states Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datanode): keep nested scan query admission unchanged Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): honor query handler factories and trim nested scan fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datanode): retain required cache error import Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): reuse registered state for nested scan decoding Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): verify payload information schema table functions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): retain nested scan runtime and cold-route guards Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(query): address nested scan review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(session): preserve read preference across region query hops Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(session): verify strict context rejection at Flow ingress Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): remove redundant serializer error case Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
jeremyhi
|
f9ab3c296b |
feat(log-store): skip corrupted object store WAL segments per region (#9425)
* feat(log-store): skip corrupted object store WAL segments per region A segment that does not decode, because its checksum does not match or its content disagrees with its footer entry, is fetched once more. If it still does not decode, the read follows `on_corrupted_segment`: `skip` records a WAL hole for the region with the object key, sequence and the footer's entry id range, increments `greptime_logstore_object_store_wal_skipped_segments_total`, logs a warning and continues with the next object; `fail` fails the read. A failed fetch fails the read in both modes. Holes are kept in memory per region and can be listed from the store. The standalone test builder allocates WAL options through `build_standalone_wal_provider`, and integration tests run a standalone instance on the object store WAL across restarts, on S3 when the `GT_S3_*` variables are set. The standalone shutdown runs every step and returns the first error. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(cmd): cover the best-effort standalone stop Build a standalone instance whose leader services fail to stop and whose datanode servers fail afterwards, and check that `Instance::stop` still stops the object store WAL and returns the leader services error. Rename `skip_segment` to `record_skipped_segment` and the file store integration test to `test_standalone_object_store_wal_survives_restarts_on_file`. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
a99cb56d71 |
feat(datanode): wire the object store WAL into the datanode and standalone (#9386)
* feat(datanode): wire the object store WAL into the datanode and standalone A datanode configured with `provider = "experimental_object_store"` now builds an `ObjectStoreLogStore` instead of failing with the placeholder "not supported yet" error. The builder resolves `storage_provider` against the object store manager (empty selects the default store, an unknown name is rejected as an invalid `storage_provider`), opens the store under the node id and the standalone generation, stops it if a later build step fails, and hands it to `Datanode`, whose shutdown stops it after the region server. Datanode shutdown is now best-effort: every step runs and the first error is returned. A datanode with a metasrv client still rejects the provider. The standalone bootstrap builds `WalProvider::ObjectStore` with the same derived node prefix before the metasrv WAL conversion, so the WAL options it allocates match the prefix the store runs under. `log-store` exports `ObjectStoreLogStore` and its testing hooks. Mito's test utilities can build an engine on the object store WAL over an in-memory object store, and new engine tests cover create, reopen, replay isolation between regions, batch open, drop and offline cleanup, the latest entry id after replay, and the rejection of object store WAL options on a Raft Engine log store. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(datanode): stop every region engine before the object store WAL The region server returned at the first engine whose stop failed, so the engines after it in map order, Mito included, were never stopped before the datanode stopped the object store WAL. It now attempts every engine, returns the first error and logs the later ones. `ObjectStoreLogStore` is also exported from the `log-store` crate root, and its callers import it from there. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
jeremyhi
|
03823a9a01 |
feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the object store WAL config, validated by the datanode and the store. In the `enqueued` mode `append_batch` returns on admission with the entry ids assigned and the object is created in the background. At a backlog threshold the next append is held back until an upload completes. A transient create failure is repeated under the same sequence with the same bytes; any conflicting object poisons the store. `stop` uploads the backlog, or returns the error that dropped it once stop began. `obsolete` clamps the watermark to the durable entry id, and an id the store handed out needs no sequence floor. Add `LogStore::wait_durable` with a default that returns at once. The object store WAL answers it once the region is durable and indexed through the entry id, and fails it after a backlog was lost. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison an enqueued store on permanent create failures Repeat a failed create in the enqueued mode only when the storage error is retryable; any other storage error poisons the store. A conflicting object poisons an enqueued store without reading its epoch, so a failed header read cannot turn the conflict into a retry. A durability wait for an id above the highest id the store handed out now waits for the handed-out ids of the region below it instead of returning at once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison on permanent create failures after stop begins A create that fails with a storage error that is not retryable poisons an enqueued store even after stop began; only a transient failure drops the backlog without poisoning. Durability waiters whose callers stopped waiting are pruned before a new waiter is queued. The backlog age test no longer depends on a follow-up append finishing within the threshold. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): answer a durability wait once no earlier entry is pending A durability wait now returns once the region holds no entry at or below the target that is handed out but not durable, instead of waiting for the largest id handed out to the region. A later object of the region that is still being created no longer holds back a wait whose target it does not cover. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): order the pending durability wait check after the actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(store-api): state that the default wait_durable keeps each store's guarantee The default `LogStore::wait_durable` returns at once, which keeps each log store's own acknowledgement guarantee; Raft Engine with `sync_write = false` acknowledges before its periodic sync, so the documentation no longer claims that every entry id a caller holds is durable. The object store WAL configuration test now also serializes the new options and reads them back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): limit the acknowledgement guarantees to the durable mode Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): repeat enqueued creates that a retry layer marks persistent An object store wrapped in the OpenDAL retry layer reports a temporary error that outlasted its retries as persistent rather than temporary. The enqueued mode now repeats a create after any storage error that is not permanent, so a transient outage behind the retry layer no longer poisons the store and drops the acknowledged backlog; after stop began such a failure still drops the backlog without poisoning. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a persistent create failure after stop begins Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Yingwen
|
bc80071409 |
feat: expose region open failure metrics (#9283)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
    
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
edbcb8224e |
fix: fail startup on duplicate region engine configs (#9281)
* fix: reject duplicate region engine configurations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: reuse canonical engine names in config validation Replace duplicated engine-name literals with the existing common-catalog constants so duplicate-config validation uses the shared engine names. Keep the TOML regression inputs independent to verify the public configuration tags. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: clarify zero-based duplicate engine config indices State explicitly that duplicate region engine configuration indices are zero-based so users can map them to the order of TOML entries. Preserve the existing index values and error classification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Dhruv VaishnavandWeny Xu
|
2ab8475b84 |
feat: add status label to datanode failed-insert metric (#9191)
* feat: label datanode failed inserts by status Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix: rename failed-insert metric label to code Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
jeremyhi
|
7fb0d9350c |
feat(wal): add the object store WAL provider identity and configuration (#9203)
* feat(wal): add the object store WAL provider identity and configuration Add the identity and configuration of the experimental object store WAL without the log store implementation: - store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a remote WAL. - common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`) with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and `on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as `object_store` with the key `wal.object_store.prefix`. The metasrv config conversion rejects the new provider. - common-meta, meta-srv, mito2: handle the new variants, map the region WAL options to the provider and reject them on a Raft Engine or Kafka log store. - datanode: validate the configuration and fail with a "not supported yet" error until the log store lands. - Example configs and the generated config docs. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(wal): accept object store WAL options when re-enabling WAL Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
discord9
|
13c69cdaee |
perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): address packed reference review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): stop without retry when maintenance is enabled Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(meta): avoid logging malformed mailbox payloads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
1409e66837 |
refactor(flight): add request builder and defer DoGet execution (#8953)
* refactor: add flight request builder Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use flight request builder in flow Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: defer frontend flight query execution Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grpc): avoid cloning requests during auth Signed-off-by: WenyXu <wenymedia@gmail.com> * test(grpc): cover flight request timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): share Flight message reader Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(flow): add Flight DoGet timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): gate Flight DDL helpers for testing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): restore Flight stream error semantics Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(grpc): document Flight stream input constructors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(flight): preserve deferred stream context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): use Flight stream SNAFU context Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
28398138ec |
fix(flight): bound DoGet response wait (#8943)
* fix(flight): defer datanode query initialization Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): retain Flight stream peer context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): improve Flight stream diagnostics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
a502dfdefd |
fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): fix transient downgrade setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): fix checkpoint lifecycle test setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover cancelled downgrade waiter retry Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence direct follower transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: trim checkpoint transition coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify checkpoint task lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b3faf22290 |
fix(tests): make two Windows CI failures deterministic (Nightly CI #8837) (#8840)
* fix(tests): reject overlay directories before opening on all platforms DatanodeOverlay::load() opened the target before checking is_file(). On Unix, File::open on a directory succeeds and the loader rejects it with "must be a regular file". On Windows, File::open on a directory fails up front with "Access is denied", so the type check was never reached and the rejects_directories_and_parse_errors test failed 4/4 in Nightly CI (issue #8837). Check std::fs::metadata before File::open: metadata succeeds on directories on both platforms, so the error message is now identical everywhere and the test assertion holds on Windows too. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): make test_region_error deterministic across platforms The second phase raced a 100ms mock handle delay against a 200ms replay_timeout; on busy Windows CI runners the error could land after the timeout fired, flaking reply.error.is_some() (Nightly CI, issue #8837). Use a mock handle that returns the error on its first poll with no delay: the catchup future completes before replay_timeout can ever fire, so the test no longer depends on wall-clock scheduling. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
8026064659 |
feat: add health-aware gRPC client routing (#8684)
* feat: add gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: defer gRPC client health checks until first use Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
255572020a |
fix: scope live analyze metrics to streaming requests (#8644)
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
45fc51ebd2 |
fix: demote expected remote dynamic filter misses (#8574)
Signed-off-by: discord9 <discord9@163.com> |
||
|
Ning Sun
|
d9b58edbb0 |
feat: add a region hook for gc cleanup (#8547)
* feat: add a region hook for gc cleanup * chore: more tests and review fix * fix: address review comments * fix: address review comments |
||
|
discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
b513dbaf4a |
fix: reject datanode startup on GC config mismatch (#8509)
* fix: reject datanode gc config mismatch Signed-off-by: discord9 <discord9@163.com> * refactor: minimize datanode gc startup check Signed-off-by: discord9 <discord9@163.com> * chore: update greptime-proto revision Signed-off-by: discord9 <discord9@163.com> * chore: use merged greptime-proto revision Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
81d8beee3a |
feat: clean up soft-dropped regions offline (#8458)
* fix(meta): skip reopening dropped tables during purge Purge soft-dropped tables by dropping stored routes directly instead of reopening tombstoned regions first. Treat legacy \`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only transition to \`DropRegions\`. Files: - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): guard undrop restore race Serialize `UNDROP TABLE` with same-name creates by seeding the original table name before procedure submission, and clean up reopened regions when metadata restore fails. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl_manager.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: clean up soft-dropped regions offline Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them. Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(datanode): reject cleanup for registered regions Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping. Add coverage for `OfflineCleanup` engine selection and registered-region rejection. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): require tombstone before undrop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): reject file-engine soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: harden soft-drop cleanup paths Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop. Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/mito2/src/engine/open_test.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump cleanup proto dependency Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/store-api/src/region_request.rs\` - \`src/common/meta/src/ddl/drop_table/executor.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: share region cleanup helpers Share common region cleanup helpers between normal drop and offline cleanup while keeping their preconditions separate. - Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`. - Share runtime artifact and manifest cache cleanup after full deletion paths. - Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC. Files: - `src/mito2/src/worker/handle_drop.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: preserve soft-drop cleanup split state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove obsolete CleanUp match arm from RegionRequest The `CleanUp` variant in the `region_request::Body` match is now handled exclusively by `RegionServer` via a separate path. This arm would have returned an unexpected error, so removing it eliminates dead code. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): clean every soft-dropped region replica Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): order soft-drop replica cleanup Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): order soft-drop replica cleanup" This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): clean every soft-dropped region replica" This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
56e9158819 |
feat: prepare soft-drop WAL retirement (#8475)
* fix: flush soft-dropped regions on close Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): handle flush-on-close race with concurrent in-flight flush When a region close with `flush_on_close: true` races with an already-running flush, pass the actual close request (including the flush_on_close flag) to the DDL handler instead of a default request so the pending flush is correctly awaited. Files: `src/mito2/src/worker/handle_close.rs` Also adds a test verifying that closing with flush-on-close while a flush is in progress still persists all written data correctly. Files: `src/mito2/src/engine/close_test.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: finish close request callsites Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: replace imperative close-region loop with functional combinators Transform the region close dispatch in `DropTableExecutor` from mutable `Vec` and `push` loops to iterator chains with `join_all`, improving idiomatic Rust style and readability. - `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode region-close logic to use `peers.map()` and nested `join_all`, moving `node_manager.datanode()` inside the closure to align with the new structure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: decouple Kafka client from WAL checkpoint Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: merge Kafka WAL index checkpoints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: delegate Kafka WAL retirement to metasrv Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: rebase main and resolve conflicts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: license header Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump proto to commits on main Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove Kafka obsolete-all index changes Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
1cc4038880 |
fix: spawn read operations on query runtime (#8433)
Ensure read requests from `RegionServer` are dispatched to the query runtime via `common_runtime::spawn_query` instead of executing on the calling thread. This prevents runtime contention and keeps read processing on the designated query runtime. Affected file: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
9936a83384 |
feat(query): add runtime provider interface (#8386)
* feat(query): add runtime provider interface Signed-off-by: discord9 <discord9@163.com> * docs(query): document runtime provider interface Signed-off-by: discord9 <discord9@163.com> * feat(query): pass runtime builder to provider Signed-off-by: discord9 <discord9@163.com> * fix(query): make runtime provider fallible Signed-off-by: discord9 <discord9@163.com> * fix(query): propagate runtime provider errors Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
2efe9884b1 |
feat: support structured instruction reply errors (#8335)
* feat: add structured instruction errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use structured instruction reply errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: respect instruction error retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused code Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b72465e679 |
feat: add remote dynamic filter metrics (#8309)
* feat: add remote dyn filter metrics Signed-off-by: discord9 <discord9@163.com> * fix: unify remote dyn filter payload budget Signed-off-by: discord9 <discord9@163.com> * style: import remote dyn filter metrics Signed-off-by: discord9 <discord9@163.com> * fix: separate remote dyn filter unregister metrics Signed-off-by: discord9 <discord9@163.com> * fix: rename remote dyn filter update outcome metric Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
|
Weny Xu
|
e520ff8300 |
feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate retry hints in core errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: propagate retry hint through RPC metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fallback retry hints to status codes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: add explicit retry hints for retryable errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove status code retry fallback Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: classify io retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints across error wrappers Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minior Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve datasource retry hints in query Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve metric engine retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve flow and frontend retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: share rskafka retry hint mapping Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
15597047c9 |
feat: pass Kafka pruned entry id when creating regions (#8282)
* refactor: use typed region wal options Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: pass kafka pruned entry id to regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused error Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify wal options serialization errors Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover legacy region wal options encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: recover legacy create table wal options Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: lock remote wal during table creation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests-s Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: refresh remote wal prune hints Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |