mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-10-03 18:45:35 +00:00
5ccfcd4644fd93d49b138a995f47f257864c292e
774
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jeremyhi
|
03823a9a01 |
feat(log-store): add the enqueued acknowledgement mode to the object store WAL (#9358)
* feat(log-store): add the enqueued acknowledgement mode to the object store WAL Add `ack_mode` (`durable` by default, or `enqueued`) and the backlog thresholds `max_unpersisted_bytes` and `max_unpersisted_age` to the object store WAL config, validated by the datanode and the store. In the `enqueued` mode `append_batch` returns on admission with the entry ids assigned and the object is created in the background. At a backlog threshold the next append is held back until an upload completes. A transient create failure is repeated under the same sequence with the same bytes; any conflicting object poisons the store. `stop` uploads the backlog, or returns the error that dropped it once stop began. `obsolete` clamps the watermark to the durable entry id, and an id the store handed out needs no sequence floor. Add `LogStore::wait_durable` with a default that returns at once. The object store WAL answers it once the region is durable and indexed through the entry id, and fails it after a backlog was lost. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison an enqueued store on permanent create failures Repeat a failed create in the enqueued mode only when the storage error is retryable; any other storage error poisons the store. A conflicting object poisons an enqueued store without reading its epoch, so a failed header read cannot turn the conflict into a retry. A durability wait for an id above the highest id the store handed out now waits for the handed-out ids of the region below it instead of returning at once. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): poison on permanent create failures after stop begins A create that fails with a storage error that is not retryable poisons an enqueued store even after stop began; only a transient failure drops the backlog without poisoning. Durability waiters whose callers stopped waiting are pruned before a new waiter is queued. The backlog age test no longer depends on a follow-up append finishing within the threshold. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): answer a durability wait once no earlier entry is pending A durability wait now returns once the region holds no entry at or below the target that is handed out but not durable, instead of waiting for the largest id handed out to the region. A later object of the region that is still being created no longer holds back a wait whose target it does not cover. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): order the pending durability wait check after the actor Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(store-api): state that the default wait_durable keeps each store's guarantee The default `LogStore::wait_durable` returns at once, which keeps each log store's own acknowledgement guarantee; Raft Engine with `sync_write = false` acknowledges before its periodic sync, so the documentation no longer claims that every entry id a caller holds is durable. The object store WAL configuration test now also serializes the new options and reads them back. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(log-store): limit the acknowledgement guarantees to the durable mode Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(log-store): repeat enqueued creates that a retry layer marks persistent An object store wrapped in the OpenDAL retry layer reports a temporary error that outlasted its retries as persistent rather than temporary. The enqueued mode now repeats a create after any storage error that is not permanent, so a transient outage behind the retry layer no longer poisons the store and drops the acknowledged backlog; after stop began such a failure still drops the backlog without poisoning. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(log-store): cover a persistent create failure after stop begins Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 Yingwen
|
bc80071409 |
feat: expose region open failure metrics (#9283)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
      
|
1d8d95c12a |
chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks (#9294)
* chore(deps): replace cargo-udeps with cargo-shear for unused dependency checks cargo-udeps requires a nightly toolchain and its pinned version (0.1.61) no longer detects unused dependencies against current cargo internals — unused deps have landed on main undetected (e.g. humantime in common-frontend since #6689). cargo-shear is a standalone static analyzer that runs on any toolchain. - Swap 'make check-udeps' / 'make fix-udeps' recipes to 'cargo shear' / 'cargo shear --fix' and retire scripts/fix-udeps.py - CI: install cargo-shear in the check-udeps job; drop the build cache and protoc steps (cargo-shear never compiles) - Remove ~150 unused dependency declarations found by cargo-shear, move misplaced deps to the correct sections, drop orphaned [workspace.dependencies] entries (arrow-cast, rustc-hash) - Add [package.metadata.cargo-shear] ignored entries with explanations for dependencies that are structurally required despite no textual reference: sqlparser (required by sqlparser_derive expansions in datatypes, common-query), common-error (required by common-macro's stack_trace_debug expansions in session, tests-fuzz), k8s-openapi (feature-pinning for the transitive kube dependency in tests-fuzz), tikv-jemalloc-sys (link-only, enables jemalloc profiling features in common-mem-prof), protobuf (required by build.rs-generated bindings in log-store) - Drop the obsolete [package.metadata.cargo-udeps.ignore] sections Part of #9289 Signed-off-by: Ning Sun <sunning@greptime.com> * fix(meta): populate physical metric table column ids (#9286) * fix(meta): populate physical metric table column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * test(meta): verify physical metric column ids Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix(postgres): return empty responses for comment-only SQL (#9295) fix(postgres): handle parsed empty queries in both protocols Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> * ci: create docs follow-up issue on PR merge instead of on label (#9237) * ci: create docs follow-up issue on PR merge instead of on label The docbot workflow previously created a docs-repo issue as soon as the 'docs-required' condition was detected (PR opened/edited with the docs checkbox ticked), even if the PR was never merged. Now the workflow also triggers on PR 'closed': - opened/edited: only manage the docs-required/docs-not-required labels - closed: create the docs issue only when the PR was actually merged and carries the docs-required label This also lets maintainers control issue creation by manually adding or removing the docs-required label before merging. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: address review comments on docs issue creation timing - Only touch docs labels when the docs checkbox state actually changed in an edit. Previously, editing any other part of the PR body while the checkbox stayed checked removed the docs-required label, silently dropping the docs follow-up now that issue creation happens at merge. Unchanged checkbox now leaves labels untouched, which also preserves manual label overrides. - Do not trust the closed event's stale label snapshot at merge time: re-read the live PR via the API and create the docs issue if the docs-required label is present OR the checkbox is ticked in the current body. - Make the workflow concurrency group action-aware so a merge run does not cancel an in-flight label update from an edit run. Signed-off-by: Ning Sun <sunning@greptime.com> * fix: make docs-required label the single source of truth at merge The label-OR-checkbox merge condition could not distinguish an intentional opt-out from an unfinished label update: removing docs-required while the checkbox stayed checked still produced an issue, and unchecking the box could still produce one if the merge read the stale label before the edit run removed it. At merge time, wait for any pending docbot runs on the PR head SHA to finish their label updates (bounded to 5 minutes), then decide solely by the live docs-required label. Adds actions: read permission for listing workflow runs. Signed-off-by: Ning Sun <sunning@greptime.com> --------- Signed-off-by: Ning Sun <sunning@greptime.com> * perf(promql): push label filters into grouped join inputs (#9280) * perf(promql): propagate matching filters through grouped joins Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * perf(promql): check matcher safety on the receiving operand Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): spell out the shapes a filter may cross `preserves_filter` ended in `_ => true`, which was only sound because `selector_matchers` independently rejects label rewriting, `count_values`, subqueries and non-rollup calls on the same operand. Loosening the latter alone would have silently pushed a matcher below a label rewrite. List the shapes that carry a scan filter instead and default to `false`. Cite #9207 for the result labels the grouped cases record: the join projects the right operand's tag set, so `zone` is missing wherever the right side aggregates it away. No behavior change. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): assert the new pushdowns reach the scan The grouped-join unit tests feed tag columns by hand and the SQLness case only checks results, which are identical whether or not the rewrite fires. Nothing would have failed if scalar arithmetic, ranking or grouped matching stopped propagating. Assert through the planner that the matcher reaches both scans, with a global topk one-side as the counter-example. Also state that the duplicate-one-side cases record a cross product Prometheus rejects (#9209), so the baseline is not read as intended semantics. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(ci): build tests-integration lib with meta-srv/mock (#9299) * fix(ci): build tests-integration lib with meta-srv/mock tests-integration's lib code (src/cluster.rs) uses meta_srv::mocks, but the dependency carrying the mock feature sits in [dev-dependencies]. Builds that only touch the lib, such as the apidoc job's cargo doc --workspace, resolve meta-srv without mock and fail with E0432. --all-targets builds unify dev-dependency features, which is why check, clippy and nextest stayed green. Move the mock-enabled meta-srv entry back to [dependencies]. The other testing features moved out in #9072 are not needed by the lib and stay in [dev-dependencies]. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(repartition): split per-case repartition tests test_repartition_metric ran four format/primary-key-encoding cases in a single test function, and test_repartition_mito ran two format cases. Each case builds its own 3-datanode cluster and runs a full repartition plus GC cycle, so on S3 the metric test took 165-178s against the 180s nextest terminate-after. Merge queue runs failed on it at random. Split each case into its own test. Cases were already independent, so they now run in parallel and each stays far inside the timeout, and a failure points at one encoding instead of four. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(json2): support altering JSON2 settings (#9029) * feat(sql): support alter syntax for JSON2 columns Signed-off-by: fys <fengys1996@gmail.com> * fix(json2): preserve rows on type hint mismatch during compaction * refactor(json2): simplify alter settings handling * fix(json2): preserve coerced values during compaction * chore: remove unnecessary clone * chor: reduce memory allocations * fix: cargo clippy * chore: update greptime-proto to main branch * refactor(datatypes): unify string handling with other JSON type hints * fix: cr --------- Signed-off-by: fys <fengys1996@gmail.com> * fix: keep compaction pruning, metadata, and index work on compact runtime (#9304) * fix: run compaction pruner tasks on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: keep compaction metadata and index work on compact runtime Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add AI matching, classification, and scoring functions (#9300) * feat: return matching scores from jev Replace the experimental three-argument Boolean function with jev(text, prompt) returning a Float64 probability in [0, 1]. Move threshold comparisons into SQL and update tests and migration examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: add Jev choice and score functions Share asynchronous execution across Noul, Choice, and Score. Validate JSON criteria before requests and return typed scalar answers. Add SQL and HTTP mock coverage with usage examples. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: use generic AI SQL function names Expose ai_match, ai_choose, and ai_score and move their implementation, tests, and usage guide under generic AI names. Document the current unreleased interface without migration history. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: share constant AI criteria within each batch Borrow scalar string arguments and lazily parse constant criteria once per batch. Share the parsed allocation across requests while preserving NULL propagation and batch validation before HTTP calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: preserve AI score uncertainty in JSONB results Return score, confidence, and probabilities in criteria-level order from one evaluation. Validate the distribution and preserve provider precision. Add JSON extraction, uncertainty, and single-request regressions, and document confidence-aware ranking. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs: explain reuse of volatile AI evaluations Document repeated SELECT and WHERE evaluation costs as N + M requests, and show subquery aliases for reusing scalar or structured AI results without additional model calls. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: share logical table batching with OTLP metrics (#9288) * feat: share logical table batching with OTLP metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: unify pending rows batch acknowledgement policy Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align logical batcher example configuration expectations Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: align batcher worker channel defaults to 65536 Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): lazily decode dense primary key columns (#9226) * perf(mito2): lazily decode dense primary key columns Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): bypass lazy decoding for full primary keys Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito-codec): preserve prefix decoding errors Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): align encoded length helper naming Rename encoded_length to encoded_len and update all callers to match the other length helpers in the module. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): clarify conditional dense key decoding Rename decode_dense_pk to ensure_dense_pk_decoded so callers can see that existing decoded values are preserved and only missing caches are populated. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito-codec): share string framing in row converter Move encoded_string_len to the parent module so Dense and Sparse use the same framing helper without depending on each other. Preserve its implementation and visibility. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: sync lock * fix: shear and check issues --------- Signed-off-by: Ning Sun <sunning@greptime.com> Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Signed-off-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Signed-off-by: fys <fengys1996@gmail.com> Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> Signed-off-by: WenyXu <wenymedia@gmail.com> Co-authored-by: Dhruv Vaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: houyuwushang <180804215+houyuwushang@users.noreply.github.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> Co-authored-by: fys <40801205+fengys1996@users.noreply.github.com> Co-authored-by: Lei, HUANG <6406592+v0y4g3r@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
edbcb8224e |
fix: fail startup on duplicate region engine configs (#9281)
* fix: reject duplicate region engine configurations Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: reuse canonical engine names in config validation Replace duplicated engine-name literals with the existing common-catalog constants so duplicate-config validation uses the shared engine names. Keep the TOML regression inputs independent to verify the public configuration tags. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: clarify zero-based duplicate engine config indices State explicitly that duplicate region engine configuration indices are zero-based so users can map them to the order of TOML entries. Preserve the existing index values and error classification. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Dhruv VaishnavandWeny Xu
|
2ab8475b84 |
feat: add status label to datanode failed-insert metric (#9191)
* feat: label datanode failed inserts by status Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> * fix: rename failed-insert metric label to code Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> --------- Signed-off-by: dhruvxvaishnav <dhruvvaishnav687@gmail.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
jeremyhi
|
7fb0d9350c |
feat(wal): add the object store WAL provider identity and configuration (#9203)
* feat(wal): add the object store WAL provider identity and configuration Add the identity and configuration of the experimental object store WAL without the log store implementation: - store-api: `Provider::ObjectStore` scoped by region id and prefix; it is a remote WAL. - common-wal: `DatanodeWalConfig::ObjectStore` (`experimental_object_store`) with `storage_provider`, `prefix`, `flush_interval`, `max_batch_bytes` and `on_corrupted_segment`, and `WalOptions::ObjectStore` persisted as `object_store` with the key `wal.object_store.prefix`. The metasrv config conversion rejects the new provider. - common-meta, meta-srv, mito2: handle the new variants, map the region WAL options to the provider and reject them on a Raft Engine or Kafka log store. - datanode: validate the configuration and fail with a "not supported yet" error until the log store lands. - Example configs and the generated config docs. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(wal): accept object store WAL options when re-enabling WAL Count object store WAL options as an existing WAL provider when setting skip_wal to false, drop an inaccurate replay note on Provider::is_remote_wal and fix the new rustdoc link. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9andNing Sun
|
94d7e2c7fc |
feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor: migrate DataFusion 55 APIs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve table function planning behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve PostgreSQL query compatibility Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: preserve distributed execution plan behavior Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover DataFusion 55 behavior regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update DataFusion 55 SQLness expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DataFusion 55 test API migration Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address DataFusion 55 CI regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: address remaining DataFusion 55 regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: adapt latest base code to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: normalize environment-specific DataFusion 55 plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align final DataFusion 55 expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: isolate DataFusion 55 regression cases Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve empty result schema in timestamp widening Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: preserve JSON source column order Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore: use released DataFusion 55 integrations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: adapt latest execution plan mock to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: pin DataFusion recursive schema and date repairs Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): align dictionary temporality match keys Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: retain Greptime DataFusion fork behaviors on version 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore ordinary function error expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh distributed count compatibility plan Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): adapt last-row cast hint to DataFusion 55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh instant last-row empty results for Arrow 59 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style: simplify DataFusion expression visitor imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore sorting and PostgreSQL column-order assertions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): restore primitive numeric coercion signatures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(function): share geo integer signature types Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: cover timestamp widening overflow boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: fix decimal coercion regression imports Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(function): preserve scalar count_hash NULL state semantics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: simplify decimal clamp case type inference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: retain historical count_hash wrapper result Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: restore timestamp widening equality and IN pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream aggregate dynamic filter correctness fix Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: carry upstream null and predicate simplification fixes Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore baseline JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: restore histogram JSON ordering expectations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh empty PromQL range result schemas Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: align native timestamp plan with DF55 decimal display Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: refresh native timestamp SQLness results for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regenerate NULL sample empty result headers for DF55 Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: use DF55 child replacement API in timestamp regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: expose pushed scan dynamic filters to DF55 producers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: encode string-backed PostgreSQL OID aliases in binary results Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: verify REGPROC binary and text over PostgreSQL protocol Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: register real PostgreSQL catalogs in server fixtures Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: complete DF55 expression inventories for custom query plans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: correct RangeSelect expression fixture and column identities Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * ci: wait for Kafka WAL helper deployment rollout Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: update custom storage empty result headers Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: require exact row counts in scan statistics Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: suppress deprecated partition_statistics warning in test Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Ning Sun <sunng@protonmail.com> |
||
|
discord9
|
13c69cdaee |
perf(gc): pack file reference exchange (#9009)
* perf(gc): pack file reference exchange Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): address packed reference review feedback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(gc): stop without retry when maintenance is enabled Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(meta): avoid logging malformed mailbox payloads Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
fa128adb8e |
feat: support request-level insert WAL skipping (#9088)
* refactor: add skip_wal fields to internal write requests Signed-off-by: WenyXu <wenymedia@gmail.com> * chore(deps): update greptime-proto for insert skip_wal Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): support request-level WAL skipping Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): handle request-level WAL policies Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate insert WAL policy through query context Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: support session-level insert WAL policy via SET Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(metric-engine): require uniform WAL policy in batch puts Signed-off-by: WenyXu <wenymedia@gmail.com> * test(metric-engine): simplify WAL policy coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(servers): simplify gRPC hint extraction Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten Mito and Metric WAL scenario orchestration Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: separate WAL and memtable-only mutations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: carry skip-WAL policy in table insert requests Signed-off-by: WenyXu <wenymedia@gmail.com> * test: flatten skip-WAL policy cases Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify WAL notifier naming Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: pin merged skip-WAL proto revision Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
1409e66837 |
refactor(flight): add request builder and defer DoGet execution (#8953)
* refactor: add flight request builder Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use flight request builder in flow Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: defer frontend flight query execution Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(grpc): avoid cloning requests during auth Signed-off-by: WenyXu <wenymedia@gmail.com> * test(grpc): cover flight request timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): share Flight message reader Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(flow): add Flight DoGet timeout Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(client): gate Flight DDL helpers for testing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): restore Flight stream error semantics Signed-off-by: WenyXu <wenymedia@gmail.com> * docs(grpc): document Flight stream input constructors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(flight): preserve deferred stream context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): use Flight stream SNAFU context Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
28398138ec |
fix(flight): bound DoGet response wait (#8943)
* fix(flight): defer datanode query initialization Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): retain Flight stream peer context Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(client): improve Flight stream diagnostics Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
a502dfdefd |
fix(mito2): fence checkpoints during region transitions (#8847)
* fix: fence checkpoints during region transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): fix transient downgrade setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): fix checkpoint lifecycle test setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover cancelled downgrade waiter retry Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence direct follower transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: trim checkpoint transition coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify checkpoint task lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b3faf22290 |
fix(tests): make two Windows CI failures deterministic (Nightly CI #8837) (#8840)
* fix(tests): reject overlay directories before opening on all platforms DatanodeOverlay::load() opened the target before checking is_file(). On Unix, File::open on a directory succeeds and the loader rejects it with "must be a regular file". On Windows, File::open on a directory fails up front with "Access is denied", so the type check was never reached and the rejects_directories_and_parse_errors test failed 4/4 in Nightly CI (issue #8837). Check std::fs::metadata before File::open: metadata succeeds on directories on both platforms, so the error message is now identical everywhere and the test assertion holds on Windows too. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(datanode): make test_region_error deterministic across platforms The second phase raced a 100ms mock handle delay against a 200ms replay_timeout; on busy Windows CI runners the error could land after the timeout fired, flaking reply.error.is_some() (Nightly CI, issue #8837). Use a mock handle that returns the error on its first poll with no delay: the catchup future completes before replay_timeout can ever fire, so the test no longer depends on wall-clock scheduling. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Weny Xu
|
8026064659 |
feat: add health-aware gRPC client routing (#8684)
* feat: add gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: harden gRPC client health routing Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: defer gRPC client health checks until first use Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
255572020a |
fix: scope live analyze metrics to streaming requests (#8644)
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
45fc51ebd2 |
fix: demote expected remote dynamic filter misses (#8574)
Signed-off-by: discord9 <discord9@163.com> |
||
|
Ning Sun
|
d9b58edbb0 |
feat: add a region hook for gc cleanup (#8547)
* feat: add a region hook for gc cleanup * chore: more tests and review fix * fix: address review comments * fix: address review comments |
||
|
discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
b513dbaf4a |
fix: reject datanode startup on GC config mismatch (#8509)
* fix: reject datanode gc config mismatch Signed-off-by: discord9 <discord9@163.com> * refactor: minimize datanode gc startup check Signed-off-by: discord9 <discord9@163.com> * chore: update greptime-proto revision Signed-off-by: discord9 <discord9@163.com> * chore: use merged greptime-proto revision Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
81d8beee3a |
feat: clean up soft-dropped regions offline (#8458)
* fix(meta): skip reopening dropped tables during purge Purge soft-dropped tables by dropping stored routes directly instead of reopening tombstoned regions first. Treat legacy \`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only transition to \`DropRegions\`. Files: - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): guard undrop restore race Serialize `UNDROP TABLE` with same-name creates by seeding the original table name before procedure submission, and clean up reopened regions when metadata restore fails. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl_manager.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: clean up soft-dropped regions offline Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them. Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(datanode): reject cleanup for registered regions Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping. Add coverage for `OfflineCleanup` engine selection and registered-region rejection. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): require tombstone before undrop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): reject file-engine soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: harden soft-drop cleanup paths Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop. Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/mito2/src/engine/open_test.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump cleanup proto dependency Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/store-api/src/region_request.rs\` - \`src/common/meta/src/ddl/drop_table/executor.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: share region cleanup helpers Share common region cleanup helpers between normal drop and offline cleanup while keeping their preconditions separate. - Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`. - Share runtime artifact and manifest cache cleanup after full deletion paths. - Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC. Files: - `src/mito2/src/worker/handle_drop.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: preserve soft-drop cleanup split state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove obsolete CleanUp match arm from RegionRequest The `CleanUp` variant in the `region_request::Body` match is now handled exclusively by `RegionServer` via a separate path. This arm would have returned an unexpected error, so removing it eliminates dead code. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): clean every soft-dropped region replica Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): order soft-drop replica cleanup Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): order soft-drop replica cleanup" This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): clean every soft-dropped region replica" This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
56e9158819 |
feat: prepare soft-drop WAL retirement (#8475)
* fix: flush soft-dropped regions on close Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): handle flush-on-close race with concurrent in-flight flush When a region close with `flush_on_close: true` races with an already-running flush, pass the actual close request (including the flush_on_close flag) to the DDL handler instead of a default request so the pending flush is correctly awaited. Files: `src/mito2/src/worker/handle_close.rs` Also adds a test verifying that closing with flush-on-close while a flush is in progress still persists all written data correctly. Files: `src/mito2/src/engine/close_test.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: finish close request callsites Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: replace imperative close-region loop with functional combinators Transform the region close dispatch in `DropTableExecutor` from mutable `Vec` and `push` loops to iterator chains with `join_all`, improving idiomatic Rust style and readability. - `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode region-close logic to use `peers.map()` and nested `join_all`, moving `node_manager.datanode()` inside the closure to align with the new structure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: decouple Kafka client from WAL checkpoint Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: merge Kafka WAL index checkpoints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: delegate Kafka WAL retirement to metasrv Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: rebase main and resolve conflicts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: license header Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump proto to commits on main Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove Kafka obsolete-all index changes Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
1cc4038880 |
fix: spawn read operations on query runtime (#8433)
Ensure read requests from `RegionServer` are dispatched to the query runtime via `common_runtime::spawn_query` instead of executing on the calling thread. This prevents runtime contention and keeps read processing on the designated query runtime. Affected file: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
9936a83384 |
feat(query): add runtime provider interface (#8386)
* feat(query): add runtime provider interface Signed-off-by: discord9 <discord9@163.com> * docs(query): document runtime provider interface Signed-off-by: discord9 <discord9@163.com> * feat(query): pass runtime builder to provider Signed-off-by: discord9 <discord9@163.com> * fix(query): make runtime provider fallible Signed-off-by: discord9 <discord9@163.com> * fix(query): propagate runtime provider errors Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
2efe9884b1 |
feat: support structured instruction reply errors (#8335)
* feat: add structured instruction errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use structured instruction reply errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: respect instruction error retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused code Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b72465e679 |
feat: add remote dynamic filter metrics (#8309)
* feat: add remote dyn filter metrics Signed-off-by: discord9 <discord9@163.com> * fix: unify remote dyn filter payload budget Signed-off-by: discord9 <discord9@163.com> * style: import remote dyn filter metrics Signed-off-by: discord9 <discord9@163.com> * fix: separate remote dyn filter unregister metrics Signed-off-by: discord9 <discord9@163.com> * fix: rename remote dyn filter update outcome metric Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
|
Weny Xu
|
e520ff8300 |
feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate retry hints in core errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: propagate retry hint through RPC metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fallback retry hints to status codes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: add explicit retry hints for retryable errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove status code retry fallback Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: classify io retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints across error wrappers Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minior Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve datasource retry hints in query Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve metric engine retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve flow and frontend retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: share rskafka retry hint mapping Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
15597047c9 |
feat: pass Kafka pruned entry id when creating regions (#8282)
* refactor: use typed region wal options Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: pass kafka pruned entry id to regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused error Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify wal options serialization errors Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover legacy region wal options encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: recover legacy create table wal options Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: lock remote wal during table creation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests-s Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: refresh remote wal prune hints Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
8fc5a3b1c7 |
feat: apply remote dynamic filters on datanode scans (#8262)
* feat: apply rdf Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * fix: drop remote dyn filter fallback exec Signed-off-by: discord9 <discord9@163.com> * Revert "fix: drop remote dyn filter fallback exec" This reverts commit bb757a596c3c2c078a836aa623226b7934103634. Signed-off-by: discord9 <discord9@163.com> * refactor: use rdf receiver logical plan instead Signed-off-by: discord9 <discord9@163.com> * test: update sqlness Signed-off-by: discord9 <discord9@163.com> * feat: rdf disable option Signed-off-by: discord9 <discord9@163.com> * tests: large int tests Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * test: update prec fix Signed-off-by: discord9 <discord9@163.com> * fix: make receiver node works Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * fix: tql disable rdf Signed-off-by: discord9 <discord9@163.com> * chore: rm useless joins Signed-off-by: discord9 <discord9@163.com> * fix: also disable in flow tql Signed-off-by: discord9 <discord9@163.com> * refactor: per review rm to promql Signed-off-by: discord9 <discord9@163.com> * chore: promql ut Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * test: rm misleading&add some nested/cleanup Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
e74a73638d |
feat: separate datanode query and ingestion runtimes (#8246)
* feat: add datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode runtime handles Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: wire datanode runtimes into region server Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode ingestion to ingestion runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode query runtime stream bridge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode reads to query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode global runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: use common datanode runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: run mito scan tasks on query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: split datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: share global fallback for datanode runtimes Use the global runtime as the fallback for datanode query and ingestion runtimes when datanode-specific pools are not initialized. This avoids creating unused datanode worker pools in non-datanode services. Files: - `src/common/runtime/src/global.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: docs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: forward query runtime stream metrics Forward inner stream metrics through the datanode query runtime bridge so `EXPLAIN ANALYZE` can report plan metrics after stream polling moves to the query runtime. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: route metric batch puts to ingest runtime Run the optimized metric batch put path on the datanode ingest runtime so metric ingestion does not bypass runtime isolation. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: abort query producer on stream drop Abort the datanode query runtime producer when the returned read stream is dropped so cancelled clients do not leave query work running in the background. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify query stream bridge setup Create the inner read stream before spawning the datanode query runtime producer so setup does not use an extra task and initialization channel. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Update Datanode Runtime Options and Region Server Logic - **`global.rs`**: Adjusted `datanode_ingest_rt_size` to utilize all available CPUs for improved performance. - **`region_server.rs`**: Simplified the collection of `put_requests` and optimized the `put_regions_batch` call for better efficiency. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Remove Redundant Checks and Simplify Code - **`global.rs`**: Removed the assertion check for already initialized global runtimes to streamline the initialization process. - **`region_server.rs`**: Simplified the extraction of `Put` requests by removing unnecessary cloning and restructuring the iterator logic. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: remove redundant spawn_datanode_query in RegionServer::handle_read The outer `spawn_datanode_query` wrapped `handle_read_inner` on the same runtime, creating a nested spawn that consumed query runtime threads unnecessarily under concurrent read load. The gRPC handler already provides runtime isolation, so the inner call is sufficient. - `src/datanode/src/region_server.rs` — inline `handle_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resolve test mismatch and redundant spawn in handle_remote_read - `src/common/runtime/src/global.rs` — update test assertion to match default `datanode_ingest_rt_size` of `cpus` instead of `1` - `src/datanode/src/region_server.rs` — inline `handle_remote_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: rename datanode runtimes Summary: - Rename datanode runtime APIs from `datanode_query` and `datanode_ingest` to `query` and `ingest`. - Rename runtime config keys from `datanode_query_rt_size` and `datanode_ingest_rt_size` to `query_rt_size` and `ingest_rt_size`. - Update config docs, example config, and config-loading coverage. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/tests/load_config_test.rs` - `src/datanode/src/region_server.rs` - `src/mito2/src/read/pruner.rs` - `src/mito2/src/read/range_cache.rs` - `src/mito2/src/read/scan_region.rs` - `src/mito2/src/read/series_scan.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: consolidate runtime options Summary: - Embed datanode runtime sizes in shared `RuntimeOptions` and remove the extra `GreptimeOptions` runtime type parameter. - Use the unified `RuntimeOptions` for datanode global and datanode-specific runtime initialization. - Update datanode runtime config coverage and ingest runtime default documentation. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/src/options.rs` - `src/cmd/src/datanode.rs` - `src/cmd/src/datanode/builder.rs` - `src/cmd/tests/load_config_test.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: guard against double initialization of datanode runtimes Add an assertion in `init_datanode_runtimes` to panic when global runtimes are already initialized, preventing silent overwrites. - `src/common/runtime/src/global.rs` — assert guard in `init_datanode_runtimes` and test `test_set_datanode_runtimes_panics_after_global_runtimes_initialized` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Ning Sun
|
fd64ced4da |
feat: introduce plugin setup functions with richer context (#8256)
feat: enrich plugin setup context |
||
|
discord9
|
6b7772e457 |
feat: add remote dynamic filter frontend registration (#8148)
* feat: filter id Signed-off-by: discord9 <discord9@163.com> * feat: dyn filter registry Signed-off-by: discord9 <discord9@163.com> * feat: filter id&refactor to type Signed-off-by: discord9 <discord9@163.com> * feat: merge scan register dyn filter(not send yet) Signed-off-by: discord9 <discord9@163.com> * feat: init reg dyn filter Signed-off-by: discord9 <discord9@163.com> * wip: remote dyn filter task 03 Signed-off-by: discord9 <discord9@163.com> * fix: resolve remote dyn filter rebase fallout Signed-off-by: discord9 <discord9@163.com> * chore: keep remote dyn filter docs local Signed-off-by: discord9 <discord9@163.com> * chore: remove stale filter id allow Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * chore: fix remote dyn filter import style Signed-off-by: discord9 <discord9@163.com> * chore: fix query metrics test fallout Signed-off-by: discord9 <discord9@163.com> * fix: exclude region from remote dyn filter id Signed-off-by: discord9 <discord9@163.com> * chore: import Signed-off-by: discord9 <discord9@163.com> * refactor: rm some to latter Signed-off-by: discord9 <discord9@163.com> * feat: add initial dyn filter snapshot Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * docs: better comment, rm some slop Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
3bd915865a |
fix: improve remote WAL replay checkpoint handling (#8225)
* fix: correct topic_latest_entry_id Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: use pruned wal id for replay checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: persist remote wal checkpoints periodically Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover remote wal topic latest entry id Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: merge pruned wal id for migration checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
50b1a07232 |
feat(datanode): hold query permit for stream and expose limiter timeout (#8215)
* feat(datanode): hold query permit for stream and expose limiter timeout Signed-off-by: evenyag <realevenyag@gmail.com> * style(datanode): use as_mut() to poll inner stream in PermitGuardedStream Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
935ef9a361 |
feat: check open region requirements (#8194)
* feat: check open region capabilities Signed-off-by: WenyXu <wenymedia@gmail.com> * test: allow file region migration tests Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refine open region requirements Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use fs scheme constant Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover open region requirement predicate Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: check file engine open requirements Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
9a4e5e8457 |
chore: expose region info inspection table (#8178)
* chore/region-sync-diff: add region info inspection core - `store-api`: add `RegionInfoEntry` schema and plan builder in `src/store-api/src/region_info.rs` and export it from `src/store-api/src/lib.rs` - `mito2`: collect region runtime metadata with `MitoEngine::all_region_infos` and `RegionRoleState::as_str` in `src/mito2/src/engine.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, `src/mito2/Cargo.toml`, and `Cargo.lock` - `datanode`: expose the reserved `InspectRegionInfo` provider in `src/datanode/src/region_server/catalog.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: expose region info schema table - `information_schema.region_info`: add frontend table wiring in `src/catalog/src/system_schema/information_schema.rs`, `src/catalog/src/system_schema/information_schema/region_info.rs`, `src/catalog/src/system_schema/information_schema/table_names.rs`, and `src/common/catalog/src/consts.rs` - `region_group` removal: drop `region_group` from `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, and `src/mito2/src/engine/basic_test.rs` - `SQLness coverage`: add standalone coverage in `tests/cases/standalone/common/information_schema/region_info.sql` and `tests/cases/standalone/common/information_schema/region_info.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: restore region group info - `region_info` schema: restore `region_group` alongside `region_sequence` in `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, and `tests/cases/standalone/common/information_schema/region_info.result` - `MitoEngine::all_region_infos`: remove redundant iterator conversion in `src/mito2/src/engine.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: clarify region sequence columns - `region_info` schema: rename `sequence` to `committed_sequence` and add nullable `flushed_sequence` in `src/store-api/src/region_info.rs` and `src/mito2/src/region.rs` - `region_info` coverage: update sequence assertions and expected metadata in `src/mito2/src/engine/basic_test.rs`, `tests/cases/standalone/common/information_schema/region_info.sql`, `tests/cases/standalone/common/information_schema/region_info.result`, and `tests/cases/standalone/common/system/information_schema.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: report region options errors - `region_info` output: preserve `region_options` serialization failures as JSON error objects in `src/mito2/src/region.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
 shuiyisong
|
17815830ed |
chore: add LeaderServicesContext control to standalone (#8164)
* chore: add refresh hook Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: merge start_with_context and start Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: place reset in recover Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: revert stop changes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
44f1804b5e |
feat: add flow query-context plumbing for terminal watermarks (#8154)
* feat: add flow checkpoint plumbing Signed-off-by: discord9 <discord9@163.com> * fix: restore when fail Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * chore: clean up some test Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * refactor: move more to pr3b Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
923c4227a3 |
feat: remote dyn filter basics (#7979)
* feat: dyn filter update abi Signed-off-by: discord9 <discord9@163.com> * feat: add remote dyn filter region rpc scaffolding Signed-off-by: discord9 <discord9@163.com> * feat: query id Signed-off-by: discord9 <discord9@163.com> * fix: prevent spoof Signed-off-by: discord9 <discord9@163.com> * fix: more proofing Signed-off-by: discord9 <discord9@163.com> * feat: make payload base64 Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * update Signed-off-by: discord9 <discord9@163.com> * chore: resolve conflict Signed-off-by: discord9 <discord9@163.com> * chore: update to main Signed-off-by: discord9 <discord9@163.com> * fix: add back defensive check Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9anddennis zhuang
|
a282b207a6 |
feat: flow inc query terminal metrics transport (#8045)
* feat: OutputMetrics for inc query Signed-off-by: discord9 <discord9@163.com> * chore: rm unused Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * docs: explain should_collect_region_watermark_from_extensions Signed-off-by: discord9 <discord9@163.com> * fix: transport flow extensions via metadata Avoid routing Flow-specific query extensions through comma-separated hints so checkpoint JSON values remain intact over Flight. Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * more per review Signed-off-by: discord9 <discord9@163.com> * chore: more review Signed-off-by: discord9 <discord9@163.com> * test: compat test Signed-off-by: discord9 <discord9@163.com> * fix: standalone also monkey patch Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
7279e48e22 |
chore: wrap standalone runtime with trait (#8083)
* chore: introduce standalone start service trait Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add region server to the trait Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
42aa58aa27 |
feat: support env vars in heartbeat (#8064)
* feat: support reporting env vars in heartbeat messages to metasrv Add `heartbeat_env_vars` config option for datanode and frontend. When configured, the specified environment variable values are read at startup and sent to metasrv in every heartbeat via the `extensions` map. Metasrv extracts and stores them in `NodeInfo` for use in routing decisions (e.g. AZ-aware region placement). - Add `EnvVars` helper in `common/meta/src/datanode.rs` following the existing `GcStat` extension pattern with `into_extensions`/`from_extensions` - Add `env_vars: HashMap<String, String>` field to `NodeInfo` in `common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat - Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`, `FrontendOptions`, and `StandaloneOptions` - Inject env vars into heartbeat `extensions` in both datanode and frontend heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`) - Extract env vars from `req.extensions` in all three metasrv `CollectXxxClusterInfoHandler`s - Update `NodeInfo` construction sites in `meta-client`, `discovery/lease.rs`, and `standalone/information_extension.rs` - Update expected TOML output in `tests-integration/tests/http.rs` - Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: address heartbeat env review feedback Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: log error on deserialization failure Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: send heartbeat env vars once Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resend heartbeat env vars after reconnect Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * revert: keep env vars in every heartbeat Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
 QuakeWang
|
45e990b7f3 |
refactor: propagate flush reasons through FlushRegions path (#8051)
* feat: propagate flush reasons through FlushRegions path Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * refactor: address flush reason review feedback Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * refactor: keep flush instruction helper name Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
QuakeWang
|
8825ea3fdf |
fix!: align gRPC CLI option names with config naming (#8021)
* fix: align gRPC CLI option names with config naming Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: warn on deprecated metasrv grpc config Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
Weny Xu
|
57f1921253 |
feat: propagate staging leader through lease and heartbeat (#7950)
* feat(mito): expose staging leader role state * fix(region): clear staging metadata on leader exit * feat: propagate staging leader role through heartbeat and metasrv * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(region): unify staging exit role transitions * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
e14404c677 |
chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21 * chore: new format * fix: lint * chore: resolve lint issues * chore: remove as_millis_f64 * chore: deps up |
||
|
jeremyhi
|
8058ce7cf2 |
refactor: simplify scan memory tracking (#7827)
* refactor: simplify scan memory tracking Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make confg-docs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by codex review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: track_with_policy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: mem granularity mb to kb Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by scan_memory_on_exhausted comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: typo Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9
|
56ee8baa3f |
feat: admin gc table/regions (#7619)
* feat: gc table Signed-off-by: discord9 <discord9@163.com> * test: admin gc Signed-off-by: discord9 <discord9@163.com> * chore: after rebase fix Signed-off-by: discord9 <discord9@163.com> * refactor: GcStats Signed-off-by: discord9 <discord9@163.com> * refactor: use gc ticker for admin gc Signed-off-by: discord9 <discord9@163.com> * fix: region routes override Signed-off-by: discord9 <discord9@163.com> * test: non happy path Signed-off-by: discord9 <discord9@163.com> * refactor: gc job report enum Signed-off-by: discord9 <discord9@163.com> * test: process 0 regions Signed-off-by: discord9 <discord9@163.com> * after rebase Signed-off-by: discord9 <discord9@163.com> * feat: allow manual gc to return error Signed-off-by: discord9 <discord9@163.com> * chore: update proto Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: timeout and update proto Signed-off-by: discord9 <discord9@163.com> * chore: udpate proto Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Ruihang Xia
|
b4f0886f80 |
chore: grouping batch open region logs (#7758)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |