rust/greptimedb
1
0
Fork 0
mirror of https://github.com/GreptimeTeam/greptimedb.git synced 2026-07-04 21:10:37 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,534 Commits 144 Branches 260 Tags
651cf045255b97d2f00fbaf806d037fde6bd973e
Commit Graph

743 Commits

Author SHA1 Message Date
Ning Sun
fd64ced4da feat: introduce plugin setup functions with richer context (#8256) 
feat: enrich plugin setup context
 2026-06-08 06:53:08 +00:00
discord9
6b7772e457 feat: add remote dynamic filter frontend registration (#8148) 
* feat: filter id

Signed-off-by: discord9 <discord9@163.com>

* feat: dyn filter registry

Signed-off-by: discord9 <discord9@163.com>

* feat: filter id&refactor to type

Signed-off-by: discord9 <discord9@163.com>

* feat: merge scan register dyn filter(not send yet)

Signed-off-by: discord9 <discord9@163.com>

* feat: init reg dyn filter

Signed-off-by: discord9 <discord9@163.com>

* wip: remote dyn filter task 03

Signed-off-by: discord9 <discord9@163.com>

* fix: resolve remote dyn filter rebase fallout

Signed-off-by: discord9 <discord9@163.com>

* chore: keep remote dyn filter docs local

Signed-off-by: discord9 <discord9@163.com>

* chore: remove stale filter id allow

Signed-off-by: discord9 <discord9@163.com>

* chore: clippy

Signed-off-by: discord9 <discord9@163.com>

* chore: fix remote dyn filter import style

Signed-off-by: discord9 <discord9@163.com>

* chore: fix query metrics test fallout

Signed-off-by: discord9 <discord9@163.com>

* fix: exclude region from remote dyn filter id

Signed-off-by: discord9 <discord9@163.com>

* chore: import

Signed-off-by: discord9 <discord9@163.com>

* refactor: rm some to latter

Signed-off-by: discord9 <discord9@163.com>

* feat: add initial dyn filter snapshot

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: better comment, rm some slop

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-06-04 12:31:42 +00:00
Weny Xu
3bd915865a fix: improve remote WAL replay checkpoint handling (#8225) 
* fix: correct topic_latest_entry_id

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: use pruned wal id for replay checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: persist remote wal checkpoints periodically

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover remote wal topic latest entry id

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: merge pruned wal id for migration checkpoint

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-06-04 06:10:38 +00:00
Yingwen
50b1a07232 feat(datanode): hold query permit for stream and expose limiter timeout (#8215) 
* feat(datanode): hold query permit for stream and expose limiter timeout

Signed-off-by: evenyag <realevenyag@gmail.com>

* style(datanode): use as_mut() to poll inner stream in PermitGuardedStream

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2026-06-02 08:00:01 +00:00
Weny Xu
935ef9a361 feat: check open region requirements (#8194) 
* feat: check open region capabilities

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: allow file region migration tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: refine open region requirements

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: use fs scheme constant

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: cover open region requirement predicate

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: check file engine open requirements

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-06-02 06:43:23 +00:00
Lei, HUANG
9a4e5e8457 chore: expose region info inspection table (#8178) 
* chore/region-sync-diff: add region info inspection core

- `store-api`: add `RegionInfoEntry` schema and plan builder in `src/store-api/src/region_info.rs` and export it from `src/store-api/src/lib.rs`
- `mito2`: collect region runtime metadata with `MitoEngine::all_region_infos` and `RegionRoleState::as_str` in `src/mito2/src/engine.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, `src/mito2/Cargo.toml`, and `Cargo.lock`
- `datanode`: expose the reserved `InspectRegionInfo` provider in `src/datanode/src/region_server/catalog.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/region-sync-diff: expose region info schema table

- `information_schema.region_info`: add frontend table wiring in `src/catalog/src/system_schema/information_schema.rs`, `src/catalog/src/system_schema/information_schema/region_info.rs`, `src/catalog/src/system_schema/information_schema/table_names.rs`, and `src/common/catalog/src/consts.rs`
- `region_group` removal: drop `region_group` from `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, and `src/mito2/src/engine/basic_test.rs`
- `SQLness coverage`: add standalone coverage in `tests/cases/standalone/common/information_schema/region_info.sql` and `tests/cases/standalone/common/information_schema/region_info.result`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/region-sync-diff: restore region group info

- `region_info` schema: restore `region_group` alongside `region_sequence` in `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, and `tests/cases/standalone/common/information_schema/region_info.result`
- `MitoEngine::all_region_infos`: remove redundant iterator conversion in `src/mito2/src/engine.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: sqlness

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: sqlness

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/region-sync-diff: clarify region sequence columns

- `region_info` schema: rename `sequence` to `committed_sequence` and add nullable `flushed_sequence` in `src/store-api/src/region_info.rs` and `src/mito2/src/region.rs`
- `region_info` coverage: update sequence assertions and expected metadata in `src/mito2/src/engine/basic_test.rs`, `tests/cases/standalone/common/information_schema/region_info.sql`, `tests/cases/standalone/common/information_schema/region_info.result`, and `tests/cases/standalone/common/system/information_schema.result`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/region-sync-diff: report region options errors

- `region_info` output: preserve `region_options` serialization failures as JSON error objects in `src/mito2/src/region.rs`

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
 2026-05-28 09:52:37 +00:00
shuiyisong
17815830ed chore: add LeaderServicesContext control to standalone (#8164) 
* chore: add refresh hook

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: merge start_with_context and start

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: place reset in recover

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: revert stop changes

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-05-28 09:23:30 +00:00
discord9
44f1804b5e feat: add flow query-context plumbing for terminal watermarks (#8154) 
* feat: add flow checkpoint plumbing

Signed-off-by: discord9 <discord9@163.com>

* fix: restore when fail

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: clean up some test

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

* refactor: move more to pr3b

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-05-26 07:24:18 +00:00
discord9
923c4227a3 feat: remote dyn filter basics (#7979) 
* feat: dyn filter update abi

Signed-off-by: discord9 <discord9@163.com>

* feat: add remote dyn filter region rpc scaffolding

Signed-off-by: discord9 <discord9@163.com>

* feat: query id

Signed-off-by: discord9 <discord9@163.com>

* fix: prevent spoof

Signed-off-by: discord9 <discord9@163.com>

* fix: more proofing

Signed-off-by: discord9 <discord9@163.com>

* feat: make payload base64

Signed-off-by: discord9 <discord9@163.com>

* chore

Signed-off-by: discord9 <discord9@163.com>

* update

Signed-off-by: discord9 <discord9@163.com>

* chore: resolve conflict

Signed-off-by: discord9 <discord9@163.com>

* chore: update to main

Signed-off-by: discord9 <discord9@163.com>

* fix: add back defensive check

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-05-14 04:05:56 +00:00
discord9
a282b207a6 feat: flow inc query terminal metrics transport (#8045) 
* feat: OutputMetrics for inc query

Signed-off-by: discord9 <discord9@163.com>

* chore: rm unused

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* docs: explain should_collect_region_watermark_from_extensions

Signed-off-by: discord9 <discord9@163.com>

* fix: transport flow extensions via metadata

Avoid routing Flow-specific query extensions through comma-separated hints so checkpoint JSON values remain intact over Flight.

Signed-off-by: discord9 <discord9@163.com>

* refactor: per review

Signed-off-by: discord9 <discord9@163.com>

* more per review

Signed-off-by: discord9 <discord9@163.com>

* chore: more review

Signed-off-by: discord9 <discord9@163.com>

* test: compat test

Signed-off-by: discord9 <discord9@163.com>

* fix: standalone also monkey patch

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
Co-authored-by: dennis zhuang <killme2008@gmail.com>
 2026-05-11 11:58:51 +00:00
shuiyisong
7279e48e22 chore: wrap standalone runtime with trait (#8083) 
* chore: introduce standalone start service trait

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add region server to the trait

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
 2026-05-11 10:06:29 +00:00
Lei, HUANG
42aa58aa27 feat: support env vars in heartbeat (#8064) 
* feat: support reporting env vars in heartbeat messages to metasrv

Add `heartbeat_env_vars` config option for datanode and frontend. When
configured, the specified environment variable values are read at startup
and sent to metasrv in every heartbeat via the `extensions` map. Metasrv
extracts and stores them in `NodeInfo` for use in routing decisions
(e.g. AZ-aware region placement).

- Add `EnvVars` helper in `common/meta/src/datanode.rs` following the
  existing `GcStat` extension pattern with `into_extensions`/`from_extensions`
- Add `env_vars: HashMap<String, String>` field to `NodeInfo` in
  `common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat
- Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`,
  `FrontendOptions`, and `StandaloneOptions`
- Inject env vars into heartbeat `extensions` in both datanode and frontend
  heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`)
- Extract env vars from `req.extensions` in all three metasrv
  `CollectXxxClusterInfoHandler`s
- Update `NodeInfo` construction sites in `meta-client`,
  `discovery/lease.rs`, and `standalone/information_extension.rs`
- Update expected TOML output in `tests-integration/tests/http.rs`
- Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat

Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: address heartbeat env review feedback

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore: log error on deserialization failure

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* refactor: send heartbeat env vars once

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* fix: resend heartbeat env vars after reconnect

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* revert: keep env vars in every heartbeat

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
 2026-05-08 07:37:53 +00:00
QuakeWang
45e990b7f3 refactor: propagate flush reasons through FlushRegions path (#8051) 
* feat: propagate flush reasons through FlushRegions path

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* refactor: address flush reason review feedback

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* refactor: keep flush instruction helper name

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
 2026-05-01 02:28:55 +00:00
QuakeWang
8825ea3fdf fix!: align gRPC CLI option names with config naming (#8021) 
* fix: align gRPC CLI option names with config naming

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

* fix: warn on deprecated metasrv grpc config

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

---------

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
 2026-04-24 09:51:01 +00:00
Weny Xu
57f1921253 feat: propagate staging leader through lease and heartbeat (#7950) 
* feat(mito): expose staging leader role state

* fix(region): clear staging metadata on leader exit

* feat: propagate staging leader role through heartbeat and metasrv

* chore: update comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(region): unify staging exit role transitions

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-04-13 09:04:02 +00:00
Ning Sun
e14404c677 chore: update rust toolchain to 2026-03-21 (#7849) 
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
 2026-03-30 12:13:14 +00:00
jeremyhi
8058ce7cf2 refactor: simplify scan memory tracking (#7827) 
* refactor: simplify scan memory tracking

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: make confg-docs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by codex review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: track_with_policy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: mem granularity mb to kb

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by scan_memory_on_exhausted comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: by review comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: typo

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-03-26 03:25:50 +00:00
discord9
56ee8baa3f feat: admin gc table/regions (#7619) 
* feat: gc table

Signed-off-by: discord9 <discord9@163.com>

* test: admin gc

Signed-off-by: discord9 <discord9@163.com>

* chore: after rebase fix

Signed-off-by: discord9 <discord9@163.com>

* refactor: GcStats

Signed-off-by: discord9 <discord9@163.com>

* refactor: use gc ticker for admin gc

Signed-off-by: discord9 <discord9@163.com>

* fix: region routes override

Signed-off-by: discord9 <discord9@163.com>

* test: non happy path

Signed-off-by: discord9 <discord9@163.com>

* refactor: gc job report enum

Signed-off-by: discord9 <discord9@163.com>

* test: process 0 regions

Signed-off-by: discord9 <discord9@163.com>

* after rebase

Signed-off-by: discord9 <discord9@163.com>

* feat: allow manual gc to return error

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* chore: timeout and update proto

Signed-off-by: discord9 <discord9@163.com>

* chore: udpate proto

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-03-06 08:25:44 +00:00
Ruihang Xia
b4f0886f80 chore: grouping batch open region logs (#7758) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2026-03-05 13:39:29 +00:00
Weny Xu
6b54fb6c21 feat(tracing): propagate mailbox trace context and refine procedure spans (#7726) 
* feat(tracing): propagate mailbox trace context and refine procedure spans

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: minor

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-02-26 11:35:04 +00:00
Weny Xu
df04267c54 fix(repartition): reject writes on deallocating regions during region merge (#7694) 
* feat(meta): add write route policy to region route with backward compatibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(meta): use partition_expr compatibility accessor in repartition matching

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): introduce staging partition rule enum for repartition instructions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(datanode): plumb staging partition rule enum through heartbeat handlers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): mark pending-deallocate regions as reject-all during merge staging

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(partition): exclude reject-all regions from write partitioning

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): store staging partition rule enum in region state

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(mito): reject writes in staging when partition rule is reject-all

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): send enter staging instruction with reject-all

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(repartition): preserve reject-all on exit, merge enter-staging instructions, and allow staged bulk writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: refactor to ignore all writes

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename StagingPartitionRule to StagingPartitionDirective across staging flow

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: nit

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-02-25 07:04:38 +00:00
dennis zhuang
6d998c043e feat: implements anomaly_score_iqr, anomaly_score_mad etc. (#7681) 
* feat: implements anomaly_score_iqr, anomaly_score_mad and anomaly_score_zscore functions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: forgot result file

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: return NaN and zero for scale=0

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: comment

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: signature, cast and degenerate case

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: tweak comment

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: format

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* feat: update min samples threshold

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: missing closing quote

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: mock engine

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix: oudated comment and test name

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: todo

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* chore: apply suggestions

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-02-25 07:01:22 +00:00
Weny Xu
d0c610f3c7 feat: add partial_drop to DropRequest (#7597) 
* feat: add `partial_drop` to `DropRequest`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: handle non-partial-drop drop task

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: remove files immediately

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-27 10:46:52 +00:00
discord9
67e51b4573 feat: gc worker on dropped region (#7537) 
* feat: allow clean up for dropped region

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

* pcr

Signed-off-by: discord9 <discord9@163.com>

* fix: get access layer correct

Signed-off-by: discord9 <discord9@163.com>

* chore: invalid gc args

Signed-off-by: discord9 <discord9@163.com>

* chore: fix test

Signed-off-by: discord9 <discord9@163.com>

* feat: more defend check

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* feat: messy impl of drop region

Signed-off-by: discord9 <discord9@163.com>

* feat: add dropped region GC handling module and integrate with GcScheduler

Signed-off-by: discord9 <discord9@163.com>

* refactor: simplify access layer creation

Signed-off-by: discord9 <discord9@163.com>

* c

Signed-off-by: discord9 <discord9@163.com>

* fix: path type

Signed-off-by: discord9 <discord9@163.com>

* feat: gc handle drop

Signed-off-by: discord9 <discord9@163.com>

* chore: use proper const

Signed-off-by: discord9 <discord9@163.com>

* fix: recursive list when check empty dir

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* refactor: with gc only delete if metadata region

Signed-off-by: discord9 <discord9@163.com>

* feat: add batch_get_table_route method to SchedulerCtx and MockSchedulerCtx

Signed-off-by: discord9 <discord9@163.com>

* chore: comment

Signed-off-by: discord9 <discord9@163.com>

* refactor: retry delete method

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2026-01-20 11:45:37 +00:00
Weny Xu
e7c19a87cd fix(datanode): fetch region engine via get_engine for sync/remap operations (#7594) 
fix(datanode): fetch region engine via get_engine for sync/remap

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-20 07:58:03 +00:00
Weny Xu
653740b42a feat(meta): add repartition timeouts and metrics (#7587) 
* feat(meta): add repartition timeouts and metrics

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add logs and remove default value

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-19 09:37:35 +00:00
jeremyhi
007e6cc860 chore: batch requests in metric engine (#7535) 
* chore: batch requests in metric engine

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix clippy

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: merge multiple schemas

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: add tracing span

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: add metrics for batch put

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: add sparse and dense encoding test case

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: avoid allocation of vec

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: table_id_for_row

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: algorithm to reduce hash lookups by using array indexing

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2026-01-16 02:57:57 +00:00
Weny Xu
2ae20daa62 feat: add sync region instruction for repartition procedure (#7562) 
* feat: add sync region instruction for repartition procedure

This commit introduces a new sync region instruction and integrates it
into the repartition procedure flow, specifically for metric engine tables.

Changes:
- Add SyncRegion instruction type and SyncRegionsReply in instruction.rs
- Implement SyncRegionHandler in datanode to handle sync region requests
- Add SyncRegion state in repartition procedure to sync newly allocated regions
- Integrate sync region step after enter_staging_region for metric engine tables
- Add sync_region flag and allocated_region_ids to PersistentContext
- Make SyncRegionFromRequest serializable for instruction transmission
- Add test utilities and mock support for sync region operations

The sync region step is conditionally executed based on the table engine type,
ensuring that newly allocated regions in metric engine tables are properly
synced from their source regions before proceeding with manifest remapping.

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add logs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(repartition): improve staging region handling and support metric engine repartition
- Reorder sync region flow: move SyncRegion from EnterStagingRegion to RepartitionStart to sync before applying staging
- Add ExitStaging metadata update state to properly clear staging leader info after repartition completes
- Update build_template_from_raw_table_info to optionally skip metric engine internal columns when creating region requests
- Fix region state transition: set_dropping now expects specific state (Staging or Writable) for proper validation
- Adjust region drop and copy handlers to handle staging regions correctly
- Add comprehensive test cases for metric engine SPLIT/MERGE partition operations on physical tables with logical tables
- Improve logging for table route updates, region drops, and repartition operations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: removes code duplication

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: update result

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: refine comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add error strategy support for flush region and flush pending deallocate regions

- **Add `ErrorStrategy` enum** in `procedure/utils.rs`:
  - Supports `Ignore` and `Retry` strategies for error handling
  - Refactor `flush_region` to accept `error_strategy` parameter
  - Extract `handle_flush_region_reply` helper function for better code organization

- **Add pending deallocate region support**:
  - Add `pending_deallocate_region_ids` field to `PersistentContext`
  - Implement `flush_pending_deallocate_regions` in `EnterStagingRegion` state
  - Flush pending deallocate regions before entering staging regions to ensure data consistency

- **Update error handling**:
  - `flush_leader_region`: Use `ErrorStrategy::Ignore` to skip unreachable datanodes
  - `sync_region`: Use `ErrorStrategy::Retry` for critical operations
  - `enter_staging_region`: Use `ErrorStrategy::Retry` when flushing pending deallocate regions

This change improves the robustness of the repartition procedure by:
1. Providing flexible error handling strategies for flush operations
2. Ensuring pending deallocate regions are properly flushed before repartitioning
3. Preventing data inconsistency during region migration

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: compile

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-15 04:52:57 +00:00
Weny Xu
f3e2d333e4 feat(repartition): implement region allocation for repartition procedure (#7534) 
* refactor: rename WalOptionsAllocator to WalProvider

The name "WalOptionsAllocator" was misleading because:
- For RaftEngine variant, it doesn't actually allocate anything
- The actual allocation logic lives in KafkaTopicPool

"WalProvider" better describes its role as providing WAL options
based on the configured WAL backend (RaftEngine or Kafka).

Changes:
- Rename `WalOptionsAllocator` to `WalProvider`
- Rename `WalOptionsAllocatorRef` to `WalProviderRef`
- Rename `build_wal_options_allocator` to `build_wal_provider`
- Rename module `wal_options_allocator` to `wal_provider`
- Rename error types: `BuildWalOptionsAllocator` -> `BuildWalProvider`,
  `StartWalOptionsAllocator` -> `StartWalProvider`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(meta): extract allocator traits from TableMetadataAllocator

Refactor TableMetadataAllocator to use trait-based dependency injection
for better testability and separation of concerns.

Changes:
- Add `ResourceIdAllocator` trait to abstract ID allocation
- Add `WalOptionsAllocator` trait to abstract WAL options allocation
- Implement traits for `Sequence` and `WalProvider`
- Remove duplicate `allocate_region_wal_options` function
- Rename `table_id_sequence` to `table_id_allocator` for consistency
- Rename `TableIdSequenceHandler` to `TableIdAllocatorHandler`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta): add max_region_number tracking to PhysicalTableRouteValue

Add `max_region_number` field to track the highest region number ever
allocated for a table. This value only increases when regions are added
and never decreases when regions are dropped, ensuring unique region
numbers across the table's lifetime.

Changes:
- Add `max_region_number` field to `PhysicalTableRouteValue`
- Implement custom `Deserialize` for backward compatibility
- Update `update_region_routes` to maintain max_region_number
- Calculate max_region_number from region_routes in `new()`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: extract TableRouteAllocator trait from TableMetadataAllocator

- Add TableRouteAllocator trait for abstracting region route allocation
- Implement blanket impl for all PeerAllocator types
- Add PeerAllocator impl for Arc<T> to support trait object delegation
- Update TableMetadataAllocator to use TableRouteAllocatorRef

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: rename TableRouteAllocator to RegionRoutesAllocator

- Rename table_route.rs to region_routes.rs
- Rename TableRouteAllocator trait to RegionRoutesAllocator
- Rename wal_option.rs to wal_options.rs for consistency
- Update TableMetadataAllocator to use new naming

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(meta-srv): implement region allocation for repartition procedure

This commit implements the region allocation phase of the repartition procedure,
which handles allocating new regions when a table needs to be split into more partitions.

Key changes:
- Refactor `RegionRoutesAllocator::allocate` to accept `(region_number, partition_expr)` tuples
  for more flexible region number assignment
- Simplify `AllocationPlanEntry` by removing `regions_to_allocate` and `regions_to_deallocate`
  fields (now derived from source/target counts)
- Add `convert_allocation_plan_to_repartition_plan` function to handle allocation, equal,
  and deallocation cases
- Fix `RepartitionPlanEntry::allocate_regions()` to return target regions (was incorrectly
  returning source regions)
- Implement complete `AllocateRegion` state with:
  - Region route allocation via `RegionRoutesAllocator`
  - WAL options allocation via `WalOptionsAllocator`
  - Operating region registration for concurrency control
  - Region creation on datanodes via `CreateTableExecutor`
  - Table route metadata update
- Add `TableRouteValue::max_region_number()` helper method
- Add comprehensive unit tests for plan conversion and allocation logic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-08 11:03:58 +00:00
Weny Xu
2f242927a8 feat(repartition): implement region deallocation for repartition procedure (#7522) 
* feat: implement deallocate regions for repartition procedure

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(metric-engine): add force flag to drop physical regions with associated logical regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: update table metadata after deallocating regions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-07 06:13:48 +00:00
jeremyhi
898e84898c feat!: make heartbeat config only in metasrv (#7510) 
* feat: make heartbeat config only in metasrv

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* feat: refine config doc

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: make the heartbeat setup simple

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by comment

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: revert config

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: proto update

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: fix sqlness wrong cfg

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-01-06 09:43:36 +00:00
Weny Xu
2d756b24c8 feat: implement RemapManifest and ApplyStagingManifest for repartition procedure (#7509) 
* feat: add RemapManifest and ApplyStagingManifest heartbeat handler

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: add `RemapManifest` and `ApplyStagingManifest` states for repartition

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2026-01-05 08:33:44 +00:00
Weny Xu
294f19fa1d feat(metric-engine): support sync logical regions from source region (#7438) 
* chore: move file

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(metric-engine): support sync logical regions from source region

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add comments

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-25 09:06:58 +00:00
Weny Xu
e1b18614ee feat(mito2): implement ApplyStagingManifest request handling (#7456) 
* feat(mito2): implement `ApplyStagingManifest` request handling

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: fmt

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix logic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: update proto

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-24 09:05:09 +00:00
Weny Xu
ee86987912 feat(repartition): implement enter staging region state (#7447) 
* feat(repartition): implement enter staging region state

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-24 02:50:27 +00:00
Yingwen
f6afb10e33 feat!: download file to fill the cache on write cache miss (#7294) 
* feat: download inverted index file

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: download for bloom and fulltext

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: implement maybe_download_background for FileCache

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: load file for parquet

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: reduce channel size

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: use ManifestCache

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: pass cache to ManifestObjectStore::new

Signed-off-by: evenyag <realevenyag@gmail.com>

* style: fix fmt and clippy

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: remove manifest cache ttl

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: remove read cache

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: clean old read cache path

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update config

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update config examples

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update test

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix CI

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: also clean the root directory

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update manifest test

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fix compiler errors

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: skip file if it exists

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: remove warn in replace

Signed-off-by: evenyag <realevenyag@gmail.com>

* feat: add a flag to enable/disable background download

set the concurrency to 1 for background download

Signed-off-by: evenyag <realevenyag@gmail.com>

* refactor: rename write_cache_enable_background_download to enable_refill_cache_on_read

Signed-off-by: evenyag <realevenyag@gmail.com>

* test: update config test

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: address comments

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs: update config.md

Signed-off-by: evenyag <realevenyag@gmail.com>

* chore: fmt code

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
 2025-12-16 08:31:26 +00:00
Weny Xu
f7d5c87ac0 feat: introduce copy_region_from for mito engine (#7389) 
* feat: introduce `copy_region_from`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: fix clippy

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-16 06:12:06 +00:00
LFC
f1abe5d215 feat: suspend frontend and datanode (#7370) 
Signed-off-by: luofucong <luofc@foxmail.com>
 2025-12-10 12:18:24 +00:00
discord9
1ebcef4794 chore: rm unnecessary warning (#7352) 
Signed-off-by: discord9 <discord9@163.com>
 2025-12-05 03:44:45 +00:00
Ruihang Xia
9d35b8cad4 refactor: remove datafusion data frame wrapper (#7347) 
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
 2025-12-04 09:38:41 +00:00
Weny Xu
18875eed4d feat: implement Display trait for FlushRegions (#7320) 
feat: implement Display trait for FlushRegions

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-12-01 06:33:23 +00:00
Weny Xu
8346acb900 feat: introduce EnterStagingRequest for RegionEngine (#7261) 
* feat: introduce `EnterStagingRequest` for region engine

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: improve error handling in staging mode entry

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-11-28 09:02:32 +00:00
Weny Xu
6b6d1ce7c4 feat: introduce remap_manifests for RegionEngine (#7265) 
* refactor: consolidate RegionManifestOptions creation logic

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: introduce`remap_manifests` for `RegionEngine`

Signed-off-by: WenyXu <wenymedia@gmail.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* chore: apply suggestions from CR

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2025-11-25 12:09:20 +00:00
discord9
0cee4fa115 feat: gc get ref from manifest (#7260) 
feat: get file ref from other manifest

Signed-off-by: discord9 <discord9@163.com>
 2025-11-19 12:13:28 +00:00
discord9
e59612043d feat: gc scheduler ctx&procedure (#7252) 
* feat: gc ctx&procedure

Signed-off-by: discord9 <discord9@163.com>

* fix: handle region not found case

Signed-off-by: discord9 <discord9@163.com>

* docs: more explain&todo

Signed-off-by: discord9 <discord9@163.com>

* per review

Signed-off-by: discord9 <discord9@163.com>

* chore: add time for region gc

Signed-off-by: discord9 <discord9@163.com>

* fix: explain why loader for gc region should fail

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-11-19 08:35:17 +00:00
discord9
29bbff3c90 feat: gc worker only local regions&test (#7203) 
* feat: gc worker only on local region

Signed-off-by: discord9 <discord9@163.com>

* more check

Signed-off-by: discord9 <discord9@163.com>

* chore: stuff

Signed-off-by: discord9 <discord9@163.com>

* fix: ignore async index file for now

Signed-off-by: discord9 <discord9@163.com>

* fix: file removal rate calc

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* chore: per review

Signed-off-by: discord9 <discord9@163.com>

* clippy

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
 2025-11-18 02:45:09 +00:00
WaterWhisperer
de9ae6066f refactor: remove export_metrics and related configuration (#7236) 
Signed-off-by: WaterWhisperer <waterwhisperer24@qq.com>
 2025-11-17 02:32:22 +00:00
jeremyhi
c7fded29ee feat: query mem limiter (#7078) 
* feat: query mem limiter

* feat: config docs

* feat: frontend query limit config

* fix: unused imports

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: add metrics for query memory tracker

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: right postion for tracker

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: avoid race condition

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: soft and hard limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: docs

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: when soft_limit == 0

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: upgrade limit algorithm

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: remove batch window

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: batch mem size

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: refine limit algorithm

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix: get sys mem

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor change

* feat: up tracker to the top stream

* feat: estimated_size for batch

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: minor refactor

* feat: scan_memory_limit connect to max_concurrent_queries

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: make callback clearly

* feat: add unlimted enum

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: by review comment

* chore: comment on recursion_limit

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat: refactor and put permit into RegionScanExec

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* chore: multiple lazy static blocks

* chore: minor change

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
 2025-11-11 07:47:55 +00:00
Lei, HUANG
abcfbd7f41 chore(metrics): add region server requests failures count metrics (#7173) 
* chore/add-region-insert-failure-metric: Add metric for failed insert requests to region server in datanode module

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

* chore/add-region-insert-failure-metric:
 Add metric for tracking failed region server requests

 - Introduce a new metric `REGION_SERVER_REQUEST_FAILURE_COUNT` to count failed region server requests.
 - Update `REGION_SERVER_INSERT_FAIL_COUNT` metric description for consistency.
 - Implement error handling in `RegionServerHandler` to increment the new failure metric on request errors.

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>

---------

Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
 2025-11-05 07:23:40 +00:00
Weny Xu
9de680f456 refactor: add support for batch region upgrade operations part2 (#7160) 
* add tests for metric engines

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat: catchup in background

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: replace sequential catchup with batch processing

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* remove single catchup

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: remove unused error

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: refine catchup tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: add unit tests

Signed-off-by: WenyXu <wenymedia@gmail.com>

* chore: apply suggestions

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
 2025-11-03 06:01:38 +00:00