mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-09-22 21:25:41 +00:00
2f7dbd21689dbf0e86b0d993f949d48be9df31da
761
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
+1         
|
c60311973d |
fix: backport v1.2.0 correctness and liveness fixes (#9023)
* fix(mito2): fence checkpoints during region transitions (#8847) * fix: fence checkpoints during region transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test(datanode): fix transient downgrade setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): fix checkpoint lifecycle test setup Signed-off-by: WenyXu <wenymedia@gmail.com> * test(mito2): cover cancelled downgrade waiter retry Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(mito2): fence direct follower transitions Signed-off-by: WenyXu <wenymedia@gmail.com> * test: trim checkpoint transition coverage Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(mito2): clarify checkpoint task lifecycle Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> (cherry picked from commit |
||
 jeremyhi
|
448f973593 |
fix: sandbox SQL local filesystem access (#8708)
* fix: sandbox SQL local filesystem access Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file sandbox review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: support Windows local copy paths Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve sandbox path errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor: simplify local path error context Signed-off-by: jeremyhi <fengjiachun@gmail.com> * perf: stream secure filesystem listings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * style: derive local file access default Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: improve local file access errors Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: address local file access review findings Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test: simplify local file access coverage Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: harden sandboxed local file backends Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: reject directory copy targets before creation Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: avoid implicit string clone in file table listing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
discord9
|
255572020a |
fix: scope live analyze metrics to streaming requests (#8644)
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
45fc51ebd2 |
fix: demote expected remote dynamic filter misses (#8574)
Signed-off-by: discord9 <discord9@163.com> |
||
|
Ning Sun
|
d9b58edbb0 |
feat: add a region hook for gc cleanup (#8547)
* feat: add a region hook for gc cleanup * chore: more tests and review fix * fix: address review comments * fix: address review comments |
||
|
discord9
|
56addd0623 |
fix: stream remote analyze metrics while pending (#8405)
* fix: stream remote analyze metrics while pending Signed-off-by: discord9 <discord9@163.com> * test: verify flight metrics preserve pending batch Signed-off-by: discord9 <discord9@163.com> * fix: preserve direct SST perf queries in plans Signed-off-by: discord9 <discord9@163.com> * fix: bind flight metrics capability to query Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
b513dbaf4a |
fix: reject datanode startup on GC config mismatch (#8509)
* fix: reject datanode gc config mismatch Signed-off-by: discord9 <discord9@163.com> * refactor: minimize datanode gc startup check Signed-off-by: discord9 <discord9@163.com> * chore: update greptime-proto revision Signed-off-by: discord9 <discord9@163.com> * chore: use merged greptime-proto revision Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Lei, HUANG
|
81d8beee3a |
feat: clean up soft-dropped regions offline (#8458)
* fix(meta): skip reopening dropped tables during purge Purge soft-dropped tables by dropping stored routes directly instead of reopening tombstoned regions first. Treat legacy \`PurgeDroppedTableState::OpenRegions\` snapshots as a compatibility-only transition to \`DropRegions\`. Files: - \`src/common/meta/src/ddl/purge_dropped_table.rs\` - \`src/common/meta/src/ddl/undrop_table.rs\` - \`src/common/meta/src/ddl/tests/drop_table.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): guard undrop restore race Serialize `UNDROP TABLE` with same-name creates by seeding the original table name before procedure submission, and clean up reopened regions when metadata restore fails. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/common/meta/src/ddl_manager.rs` - `src/common/meta/src/ddl/tests/drop_table.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: clean up soft-dropped regions offline Use an explicit RegionCleanUp request for purge-table cleanup so tombstoned regions can be removed without reopening them. Route cleanup through datanode, Mito, and metric-engine offline paths, including WAL obsoletion and region directory removal. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(datanode): reject cleanup for registered regions Return `RegionBusy` when `RegionCleanUp` targets a region already tracked by the datanode, so offline cleanup only runs for regions without a local mapping. Add coverage for `OfflineCleanup` engine selection and registered-region rejection. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): require tombstone before undrop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): reject file-engine soft drop Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: harden soft-drop cleanup paths Reject `RegionCleanUp` for already-open Mito regions instead of turning cleanup into a drop. Make `UndropTableProcedure` tolerate missing persisted table names and always deregister failure detectors after restore-failure cleanup. Files: - `src/common/meta/src/ddl/undrop_table.rs` - `src/mito2/src/engine/open_test.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump cleanup proto dependency Bump \`greptime-proto\` to the reviewed cleanup RPC revision and align cleanup request parsing and dispatch with the renamed \`CleanUpRequest\` payload. Files: - \`Cargo.toml\` - \`Cargo.lock\` - \`src/store-api/src/region_request.rs\` - \`src/common/meta/src/ddl/drop_table/executor.rs\` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: share region cleanup helpers Share common region cleanup helpers between normal drop and offline cleanup while keeping their preconditions separate. - Extract shared dropped-region runtime cleanup for `handle_drop_request` and `handle_offline_cleanup_request`. - Share runtime artifact and manifest cache cleanup after full deletion paths. - Make full-drop directory removal policy explicit: full drop and purge cleanup force physical deletion, while partial drop may defer to global GC. Files: - `src/mito2/src/worker/handle_drop.rs` - `src/mito2/src/worker/handle_open.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: preserve soft-drop cleanup split state Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove obsolete CleanUp match arm from RegionRequest The `CleanUp` variant in the `region_request::Body` match is now handled exclusively by `RegionServer` via a separate path. This arm would have returned an unexpected error, so removing it eliminates dead code. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): clean every soft-dropped region replica Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(meta): order soft-drop replica cleanup Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): order soft-drop replica cleanup" This reverts commit e77162d3e5ebcf2817e2845a6a5177c328fb2c60. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * Revert "fix(meta): clean every soft-dropped region replica" This reverts commit 2378e00cc258ca1b6a85a1aafbd68c79c666f43c. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
56e9158819 |
feat: prepare soft-drop WAL retirement (#8475)
* fix: flush soft-dropped regions on close Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): handle flush-on-close race with concurrent in-flight flush When a region close with `flush_on_close: true` races with an already-running flush, pass the actual close request (including the flush_on_close flag) to the DDL handler instead of a default request so the pending flush is correctly awaited. Files: `src/mito2/src/worker/handle_close.rs` Also adds a test verifying that closing with flush-on-close while a flush is in progress still persists all written data correctly. Files: `src/mito2/src/engine/close_test.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat: support full WAL retirement Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: complete close request migration Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: finish close request callsites Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: guard Kafka provider setup behind index collector check Move Kafka provider initialization and `get_or_insert` inside the existing `if let Some(collector)` block so these operations are skipped when no global index collector is configured. Affected file: - `src/log-store/src/kafka/log_store.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: avoid to_vec Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: replace imperative close-region loop with functional combinators Transform the region close dispatch in `DropTableExecutor` from mutable `Vec` and `push` loops to iterator chains with `join_all`, improving idiomatic Rust style and readability. - `src/common/meta/src/ddl/drop_table/executor.rs` — rewired datanode region-close logic to use `peers.map()` and nested `join_all`, moving `node_manager.datanode()` inside the closure to align with the new structure Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: decouple Kafka client from WAL checkpoint Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: merge Kafka WAL index checkpoints Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: delegate Kafka WAL retirement to metasrv Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: rebase main and resolve conflicts Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix: license header Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: bump proto to commits on main Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor: remove Kafka obsolete-all index changes Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
1cc4038880 |
fix: spawn read operations on query runtime (#8433)
Ensure read requests from `RegionServer` are dispatched to the query runtime via `common_runtime::spawn_query` instead of executing on the calling thread. This prevents runtime contention and keeps read processing on the designated query runtime. Affected file: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
9936a83384 |
feat(query): add runtime provider interface (#8386)
* feat(query): add runtime provider interface Signed-off-by: discord9 <discord9@163.com> * docs(query): document runtime provider interface Signed-off-by: discord9 <discord9@163.com> * feat(query): pass runtime builder to provider Signed-off-by: discord9 <discord9@163.com> * fix(query): make runtime provider fallible Signed-off-by: discord9 <discord9@163.com> * fix(query): propagate runtime provider errors Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
2efe9884b1 |
feat: support structured instruction reply errors (#8335)
* feat: add structured instruction errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use structured instruction reply errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: respect instruction error retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused code Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
b72465e679 |
feat: add remote dynamic filter metrics (#8309)
* feat: add remote dyn filter metrics Signed-off-by: discord9 <discord9@163.com> * fix: unify remote dyn filter payload budget Signed-off-by: discord9 <discord9@163.com> * style: import remote dyn filter metrics Signed-off-by: discord9 <discord9@163.com> * fix: separate remote dyn filter unregister metrics Signed-off-by: discord9 <discord9@163.com> * fix: rename remote dyn filter update outcome metric Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
a3461caf9d |
feat: expose region read load metrics (#8316)
* feat: expose region read load through Prometheus metrics and heartbeat Introduce region-level query load tracking (CPU time and scanned bytes) collected by `RegionScanExec`, exposed via Prometheus metrics and optionally reported through heartbeat region stats. - **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add `greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`, and `greptime_mito_region_written_bytes_since_open` gauge metrics. - **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`, `src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with `IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle management (init, reset, remove-on-drop). - **RegionStatistic** (`src/store-api/src/region_engine.rs`, `src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and `query_scanned_bytes` fields. - **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from metadata and data regions. - **Heartbeat** (`src/datanode/src/heartbeat.rs`, `src/common/meta/src/datanode.rs`): Relay region query load via heartbeat `RegionStat`; add test. - **Query engine** (`src/query/src/options.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dist_plan/merge_scan.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config; wire `RegionScanExec` to accumulate CPU time and scanned bytes. - **Table scan** (`src/table/src/table/scan.rs`, `src/table/src/table/metrics.rs`): Wire table scan metrics. - **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/config.md`): Add example config and documentation for `enable_region_query_load_report`. - **Tests** (`src/mito2/src/engine/basic_test.rs`, `src/mito2/src/engine/close_test.rs`, `src/cmd/tests/load_config_test.rs`, `src/flow/src/adapter.rs`): Add unit tests for region query load reporting and metric cleanup on region close; set default config values. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region read load report config from query layer to mito engine Move the `enable_region_query_load_report` setting from query-level config (`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config (`MitoConfig`), and expose it through the `RegionScanner` trait instead of `ScanRequest`/`PrepareRequest`. - Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs` - Scan region plumbing: `src/mito2/src/read/scan_region.rs` - RegionScanner trait: `src/store-api/src/region_engine.rs` - Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs` - RegionScanExec: `src/table/src/table/scan.rs` - Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs` - Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs` - Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move region query load report config from MitoConfig to LoggingOptions Relocate the `enable_region_query_load_report` setting from `MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`), and thread it into `MitoEngineBuilder` instead of reading from the engine config directly. This makes the region read-load reporting a per-node logging/observability concern rather than a per-engine storage setting. - `config/config.md` - `config/datanode.example.toml` - `config/standalone.example.toml` - `src/common/telemetry/src/logging.rs` - `src/datanode/src/datanode.rs` - `src/mito2/src/config.rs` - `src/mito2/src/engine.rs` - `src/mito2/src/region.rs` Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: report region query load on stream drop instead of stream end Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()` to `Drop::drop()` so that region query load is reported even when the stream is dropped prematurely (not just when fully consumed). Affected files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, Huang <huanglei@qiyi.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: make region query load reporting configurable Introduce `enable_region_query_load_report` flag to optionally report per-region `query_cpu_time` and `query_scanned_bytes` metrics instead of always creating them. When disabled, the Prometheus gauges are not created (`None`), avoiding metric churn for workloads that do not need query-level load tracking. - `src/common/meta/src/datanode.rs` — Placeholder fields for query load - `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset - `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener` - `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop` - `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open - `src/mito2/src/worker/handle_create.rs` — Pass flag on region create - `src/mito2/src/worker/handle_open.rs` — Pass flag on region open - `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder` - `src/mito2/src/test_util.rs` — Test helpers for both modes - `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases - `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove elapsed_compute metric from scan stream The elapsed_compute metric conflated poll-wait time with actual CPU computation, making it misleading. Removed the metric and its recording path from StreamMetrics and StreamWithMetricWrapper. Added a test asserting that poll duration is not reported as elapsed_compute. - `src/table/src/table/metrics.rs` — removed elapsed_compute field, builder, and record_elapsed_compute method - `src/table/src/table/scan.rs` — removed record_elapsed_compute call; added SlowRecordBatchStream test helper and wrapper_poll_time_is_not_elapsed_compute test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: disable region query load report for compaction scans Compaction scans are internal operations initiated by the engine, not user queries. Disable region query load reporting when the scan input is marked as compaction to avoid misleading load metrics. - `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report` to `false` when compaction is enabled; add unit test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * test: add `enable_per_region_metrics` config to HTTP integration test - Enable per-region metrics config in HTTP test setup \`tests-integration/tests/http.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove region query load reporting tests and helpers Remove the region query load reporting feature from the codebase, including tests, test utilities, and helper infrastructure that were part of this now-deprecated functionality. Specifically: - Remove region query load reporting tests from `src/mito2/src/engine/basic_test.rs` and `src/table/src/table/scan.rs`, and the region close metrics test from `src/mito2/src/engine/close_test.rs` - Remove region query load report test utilities and simplify engine construction helpers in `src/mito2/src/test_util.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * perf: avoid disabled region query load timing Summary: - Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled. - Keep region query-load CPU accounting active only when reporting is enabled. Files: - `src/table/src/table/scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: move per-region query load reporting from storage to query engine Move `enable_per_region_metrics` from datanode to frontend config and migrate query load tracking (CPU time, scanned bytes) from mito2 storage engine to the query engine's distributed scan planner. The storage-level metrics plumbing and `enable_region_query_load_report` flag are removed from mito2, `ScanInput`, `ScanRegion`, and `RegionScanner`. Query-level metrics are now collected in `merge_scan.rs` via `scan_region_load`. - `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes` metrics, `enable_region_query_load_report` plumbing from engine, region, opener, scanner types, workers - `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes` from `RegionStatistic` - `src/metric-engine/` -- Remove query load fields from `get_region_statistic` - `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`; wire through planner, optimizer, merge scan with `scan_region_load` metrics - `src/frontend/` -- Pass `enable_per_region_metrics` into `QueryOptions` - `src/common/meta/` -- Remove TODO for query load fields - `config/` -- Move `enable_per_region_metrics` from datanode to frontend and standalone example configs - `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode config test - `src/flow/` -- Add `enable_per_region_metrics` default to flownode options - `src/table/` -- Remove unused query load fields from scan - `src/datanode/` -- Remove `with_enable_region_query_load_report` calls Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: remove obsolete mito write load metric Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path. Related files: - \`src/mito2/src/metrics.rs\` - \`src/mito2/src/region.rs\` - \`src/mito2/src/region/opener.rs\` - \`src/mito2/src/region_write_ctx.rs\` - \`src/mito2/src/engine/basic_test.rs\` - \`src/mito2/src/worker.rs\` - \`src/mito2/src/config.rs\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: change region query load metrics from gauge to counter Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from `IntGaugeVec` to `IntCounterVec` since these values are monotonically increasing and do not need gauge semantics. Update corresponding `add` calls to `inc_by` in merge scan reporting. Files: - `src/store-api/src/metrics.rs` — metric type and label changes - `src/query/src/dist_plan/merge_scan.rs` — caller adaptation Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: pass ReadItem directly to report_region_query_load Move `region_scan_load` call to the caller, so `report_region_query_load` accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`. - `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call, remove stale test Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: ensure region query load is reported on MergeScanExec drop Remove the `enable_per_region_metrics` parameter from `report_region_query_load` so region load metrics are always emitted. Add a `Drop` impl for `MergeScanExec` that reports sub-stage metrics when the executor is dropped, covering edge cases where per-region metric emission was missed. Add a unit test verifying CPU time and scanned bytes are recorded on drop. Affected file: `src/query/src/dist_plan/merge_scan.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: gate region query load reporting Guard drop-time region query load reporting with the configured per-region metrics flag. Related files: - \`src/query/src/dist_plan/merge_scan.rs\` Symbols: - \`MergeScanExec::drop\` - \`enable_per_region_metrics\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clean region query load metrics on drop Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry. Related files: - \`src/mito2/src/region.rs\` Symbols: - \`MitoRegion::drop\` - \`REGION_QUERY_CPU_TIME\` - \`REGION_QUERY_SCANNED_BYTES\` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> Signed-off-by: Lei Huang <lei@huang.to> Signed-off-by: Lei, Huang <huanglei@qiyi.com> |
||
|
Weny Xu
|
e520ff8300 |
feat: decouple error retryability from status codes (#8301)
* feat: add retry hint to common error Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: propagate retry hints in core errors Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: propagate retry hint through RPC metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fallback retry hints to status codes Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: add explicit retry hints for retryable errors Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: remove status code retry fallback Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: classify io retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve retry hints across error wrappers Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minior Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve datasource retry hints in query Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve metric engine retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: preserve flow and frontend retry hints Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: share rskafka retry hint mapping Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
15597047c9 |
feat: pass Kafka pruned entry id when creating regions (#8282)
* refactor: use typed region wal options Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: pass kafka pruned entry id to regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: remove unused error Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: clarify wal options serialization errors Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover legacy region wal options encoding Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: recover legacy create table wal options Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: lock remote wal during table creation Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: fix unit tests-s Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: refresh remote wal prune hints Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
8fc5a3b1c7 |
feat: apply remote dynamic filters on datanode scans (#8262)
* feat: apply rdf Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * fix: drop remote dyn filter fallback exec Signed-off-by: discord9 <discord9@163.com> * Revert "fix: drop remote dyn filter fallback exec" This reverts commit bb757a596c3c2c078a836aa623226b7934103634. Signed-off-by: discord9 <discord9@163.com> * refactor: use rdf receiver logical plan instead Signed-off-by: discord9 <discord9@163.com> * test: update sqlness Signed-off-by: discord9 <discord9@163.com> * feat: rdf disable option Signed-off-by: discord9 <discord9@163.com> * tests: large int tests Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * test: update prec fix Signed-off-by: discord9 <discord9@163.com> * fix: make receiver node works Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * fix: tql disable rdf Signed-off-by: discord9 <discord9@163.com> * chore: rm useless joins Signed-off-by: discord9 <discord9@163.com> * fix: also disable in flow tql Signed-off-by: discord9 <discord9@163.com> * refactor: per review rm to promql Signed-off-by: discord9 <discord9@163.com> * chore: promql ut Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * test: rm misleading&add some nested/cleanup Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Lei, HUANG
|
e74a73638d |
feat: separate datanode query and ingestion runtimes (#8246)
* feat: add datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode runtime handles Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: wire datanode runtimes into region server Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode ingestion to ingestion runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode query runtime stream bridge Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: route datanode reads to query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: add datanode global runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: use common datanode runtimes Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: run mito scan tasks on query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: split datanode runtime options Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: clippy Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: share global fallback for datanode runtimes Use the global runtime as the fallback for datanode query and ingestion runtimes when datanode-specific pools are not initialized. This avoids creating unused datanode worker pools in non-datanode services. Files: - `src/common/runtime/src/global.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: docs Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: forward query runtime stream metrics Forward inner stream metrics through the datanode query runtime bridge so `EXPLAIN ANALYZE` can report plan metrics after stream polling moves to the query runtime. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: route metric batch puts to ingest runtime Run the optimized metric batch put path on the datanode ingest runtime so metric ingestion does not bypass runtime isolation. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: abort query producer on stream drop Abort the datanode query runtime producer when the returned read stream is dropped so cancelled clients do not leave query work running in the background. Files: - `src/datanode/src/query_stream.rs` - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: simplify query stream bridge setup Create the inner read stream before spawning the datanode query runtime producer so setup does not use an extra task and initialization channel. Files: - `src/datanode/src/region_server.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Update Datanode Runtime Options and Region Server Logic - **`global.rs`**: Adjusted `datanode_ingest_rt_size` to utilize all available CPUs for improved performance. - **`region_server.rs`**: Simplified the collection of `put_requests` and optimized the `put_regions_batch` call for better efficiency. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat/runtime-priority: ### Remove Redundant Checks and Simplify Code - **`global.rs`**: Removed the assertion check for already initialized global runtimes to streamline the initialization process. - **`region_server.rs`**: Simplified the extraction of `Put` requests by removing unnecessary cloning and restructuring the iterator logic. Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: remove redundant spawn_datanode_query in RegionServer::handle_read The outer `spawn_datanode_query` wrapped `handle_read_inner` on the same runtime, creating a nested spawn that consumed query runtime threads unnecessarily under concurrent read load. The gRPC handler already provides runtime isolation, so the inner call is sufficient. - `src/datanode/src/region_server.rs` — inline `handle_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resolve test mismatch and redundant spawn in handle_remote_read - `src/common/runtime/src/global.rs` — update test assertion to match default `datanode_ingest_rt_size` of `cpus` instead of `1` - `src/datanode/src/region_server.rs` — inline `handle_remote_read_inner` directly instead of spawning onto the datanode query runtime Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: rename datanode runtimes Summary: - Rename datanode runtime APIs from `datanode_query` and `datanode_ingest` to `query` and `ingest`. - Rename runtime config keys from `datanode_query_rt_size` and `datanode_ingest_rt_size` to `query_rt_size` and `ingest_rt_size`. - Update config docs, example config, and config-loading coverage. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/tests/load_config_test.rs` - `src/datanode/src/region_server.rs` - `src/mito2/src/read/pruner.rs` - `src/mito2/src/read/range_cache.rs` - `src/mito2/src/read/scan_region.rs` - `src/mito2/src/read/series_scan.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: consolidate runtime options Summary: - Embed datanode runtime sizes in shared `RuntimeOptions` and remove the extra `GreptimeOptions` runtime type parameter. - Use the unified `RuntimeOptions` for datanode global and datanode-specific runtime initialization. - Update datanode runtime config coverage and ingest runtime default documentation. Files: - `src/common/runtime/src/global.rs` - `src/common/runtime/src/lib.rs` - `src/cmd/src/options.rs` - `src/cmd/src/datanode.rs` - `src/cmd/src/datanode/builder.rs` - `src/cmd/tests/load_config_test.rs` - `config/datanode.example.toml` - `config/config.md` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * feat: guard against double initialization of datanode runtimes Add an assertion in `init_datanode_runtimes` to panic when global runtimes are already initialized, preventing silent overwrites. - `src/common/runtime/src/global.rs` — assert guard in `init_datanode_runtimes` and test `test_set_datanode_runtimes_panics_after_global_runtimes_initialized` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
|
Ning Sun
|
fd64ced4da |
feat: introduce plugin setup functions with richer context (#8256)
feat: enrich plugin setup context |
||
|
discord9
|
6b7772e457 |
feat: add remote dynamic filter frontend registration (#8148)
* feat: filter id Signed-off-by: discord9 <discord9@163.com> * feat: dyn filter registry Signed-off-by: discord9 <discord9@163.com> * feat: filter id&refactor to type Signed-off-by: discord9 <discord9@163.com> * feat: merge scan register dyn filter(not send yet) Signed-off-by: discord9 <discord9@163.com> * feat: init reg dyn filter Signed-off-by: discord9 <discord9@163.com> * wip: remote dyn filter task 03 Signed-off-by: discord9 <discord9@163.com> * fix: resolve remote dyn filter rebase fallout Signed-off-by: discord9 <discord9@163.com> * chore: keep remote dyn filter docs local Signed-off-by: discord9 <discord9@163.com> * chore: remove stale filter id allow Signed-off-by: discord9 <discord9@163.com> * chore: clippy Signed-off-by: discord9 <discord9@163.com> * chore: fix remote dyn filter import style Signed-off-by: discord9 <discord9@163.com> * chore: fix query metrics test fallout Signed-off-by: discord9 <discord9@163.com> * fix: exclude region from remote dyn filter id Signed-off-by: discord9 <discord9@163.com> * chore: import Signed-off-by: discord9 <discord9@163.com> * refactor: rm some to latter Signed-off-by: discord9 <discord9@163.com> * feat: add initial dyn filter snapshot Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * docs: better comment, rm some slop Signed-off-by: discord9 <discord9@163.com> * chore: per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
3bd915865a |
fix: improve remote WAL replay checkpoint handling (#8225)
* fix: correct topic_latest_entry_id Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: use pruned wal id for replay checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: persist remote wal checkpoints periodically Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover remote wal topic latest entry id Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: merge pruned wal id for migration checkpoint Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Yingwen
|
50b1a07232 |
feat(datanode): hold query permit for stream and expose limiter timeout (#8215)
* feat(datanode): hold query permit for stream and expose limiter timeout Signed-off-by: evenyag <realevenyag@gmail.com> * style(datanode): use as_mut() to poll inner stream in PermitGuardedStream Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Weny Xu
|
935ef9a361 |
feat: check open region requirements (#8194)
* feat: check open region capabilities Signed-off-by: WenyXu <wenymedia@gmail.com> * test: allow file region migration tests Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refine open region requirements Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: use fs scheme constant Signed-off-by: WenyXu <wenymedia@gmail.com> * test: cover open region requirement predicate Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: check file engine open requirements Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Lei, HUANG
|
9a4e5e8457 |
chore: expose region info inspection table (#8178)
* chore/region-sync-diff: add region info inspection core - `store-api`: add `RegionInfoEntry` schema and plan builder in `src/store-api/src/region_info.rs` and export it from `src/store-api/src/lib.rs` - `mito2`: collect region runtime metadata with `MitoEngine::all_region_infos` and `RegionRoleState::as_str` in `src/mito2/src/engine.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, `src/mito2/Cargo.toml`, and `Cargo.lock` - `datanode`: expose the reserved `InspectRegionInfo` provider in `src/datanode/src/region_server/catalog.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: expose region info schema table - `information_schema.region_info`: add frontend table wiring in `src/catalog/src/system_schema/information_schema.rs`, `src/catalog/src/system_schema/information_schema/region_info.rs`, `src/catalog/src/system_schema/information_schema/table_names.rs`, and `src/common/catalog/src/consts.rs` - `region_group` removal: drop `region_group` from `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, and `src/mito2/src/engine/basic_test.rs` - `SQLness coverage`: add standalone coverage in `tests/cases/standalone/common/information_schema/region_info.sql` and `tests/cases/standalone/common/information_schema/region_info.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: restore region group info - `region_info` schema: restore `region_group` alongside `region_sequence` in `src/store-api/src/region_info.rs`, `src/mito2/src/region.rs`, `src/mito2/src/engine/basic_test.rs`, and `tests/cases/standalone/common/information_schema/region_info.result` - `MitoEngine::all_region_infos`: remove redundant iterator conversion in `src/mito2/src/engine.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: sqlness Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: clarify region sequence columns - `region_info` schema: rename `sequence` to `committed_sequence` and add nullable `flushed_sequence` in `src/store-api/src/region_info.rs` and `src/mito2/src/region.rs` - `region_info` coverage: update sequence assertions and expected metadata in `src/mito2/src/engine/basic_test.rs`, `tests/cases/standalone/common/information_schema/region_info.sql`, `tests/cases/standalone/common/information_schema/region_info.result`, and `tests/cases/standalone/common/system/information_schema.result` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore/region-sync-diff: report region options errors - `region_info` output: preserve `region_options` serialization failures as JSON error objects in `src/mito2/src/region.rs` Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
 shuiyisong
|
17815830ed |
chore: add LeaderServicesContext control to standalone (#8164)
* chore: add refresh hook Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: merge start_with_context and start Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: place reset in recover Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: revert stop changes Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: CR issue Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
discord9
|
44f1804b5e |
feat: add flow query-context plumbing for terminal watermarks (#8154)
* feat: add flow checkpoint plumbing Signed-off-by: discord9 <discord9@163.com> * fix: restore when fail Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * chore: clean up some test Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * refactor: move more to pr3b Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
923c4227a3 |
feat: remote dyn filter basics (#7979)
* feat: dyn filter update abi Signed-off-by: discord9 <discord9@163.com> * feat: add remote dyn filter region rpc scaffolding Signed-off-by: discord9 <discord9@163.com> * feat: query id Signed-off-by: discord9 <discord9@163.com> * fix: prevent spoof Signed-off-by: discord9 <discord9@163.com> * fix: more proofing Signed-off-by: discord9 <discord9@163.com> * feat: make payload base64 Signed-off-by: discord9 <discord9@163.com> * chore Signed-off-by: discord9 <discord9@163.com> * update Signed-off-by: discord9 <discord9@163.com> * chore: resolve conflict Signed-off-by: discord9 <discord9@163.com> * chore: update to main Signed-off-by: discord9 <discord9@163.com> * fix: add back defensive check Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9anddennis zhuang
|
a282b207a6 |
feat: flow inc query terminal metrics transport (#8045)
* feat: OutputMetrics for inc query Signed-off-by: discord9 <discord9@163.com> * chore: rm unused Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * docs: explain should_collect_region_watermark_from_extensions Signed-off-by: discord9 <discord9@163.com> * fix: transport flow extensions via metadata Avoid routing Flow-specific query extensions through comma-separated hints so checkpoint JSON values remain intact over Flight. Signed-off-by: discord9 <discord9@163.com> * refactor: per review Signed-off-by: discord9 <discord9@163.com> * more per review Signed-off-by: discord9 <discord9@163.com> * chore: more review Signed-off-by: discord9 <discord9@163.com> * test: compat test Signed-off-by: discord9 <discord9@163.com> * fix: standalone also monkey patch Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Co-authored-by: dennis zhuang <killme2008@gmail.com> |
||
|
shuiyisong
|
7279e48e22 |
chore: wrap standalone runtime with trait (#8083)
* chore: introduce standalone start service trait Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add region server to the trait Signed-off-by: shuiyisong <xixing.sys@gmail.com> * chore: add comments Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
Lei, HUANG
|
42aa58aa27 |
feat: support env vars in heartbeat (#8064)
* feat: support reporting env vars in heartbeat messages to metasrv Add `heartbeat_env_vars` config option for datanode and frontend. When configured, the specified environment variable values are read at startup and sent to metasrv in every heartbeat via the `extensions` map. Metasrv extracts and stores them in `NodeInfo` for use in routing decisions (e.g. AZ-aware region placement). - Add `EnvVars` helper in `common/meta/src/datanode.rs` following the existing `GcStat` extension pattern with `into_extensions`/`from_extensions` - Add `env_vars: HashMap<String, String>` field to `NodeInfo` in `common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat - Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`, `FrontendOptions`, and `StandaloneOptions` - Inject env vars into heartbeat `extensions` in both datanode and frontend heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`) - Extract env vars from `req.extensions` in all three metasrv `CollectXxxClusterInfoHandler`s - Update `NodeInfo` construction sites in `meta-client`, `discovery/lease.rs`, and `standalone/information_extension.rs` - Update expected TOML output in `tests-integration/tests/http.rs` - Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: address heartbeat env review feedback Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * chore: log error on deserialization failure Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * refactor: send heartbeat env vars once Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * fix: resend heartbeat env vars after reconnect Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> * revert: keep env vars in every heartbeat Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> --------- Signed-off-by: Lei, HUANG <leih@nvidia.com> Signed-off-by: Lei, HUANG <mrsatangel@gmail.com> |
||
 QuakeWang
|
45e990b7f3 |
refactor: propagate flush reasons through FlushRegions path (#8051)
* feat: propagate flush reasons through FlushRegions path Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * refactor: address flush reason review feedback Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * refactor: keep flush instruction helper name Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
QuakeWang
|
8825ea3fdf |
fix!: align gRPC CLI option names with config naming (#8021)
* fix: align gRPC CLI option names with config naming Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> * fix: warn on deprecated metasrv grpc config Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> --------- Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com> |
||
|
Weny Xu
|
57f1921253 |
feat: propagate staging leader through lease and heartbeat (#7950)
* feat(mito): expose staging leader role state * fix(region): clear staging metadata on leader exit * feat: propagate staging leader role through heartbeat and metasrv * chore: update comments Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(region): unify staging exit role transitions * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Ning Sun
|
e14404c677 |
chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21 * chore: new format * fix: lint * chore: resolve lint issues * chore: remove as_millis_f64 * chore: deps up |
||
|
jeremyhi
|
8058ce7cf2 |
refactor: simplify scan memory tracking (#7827)
* refactor: simplify scan memory tracking Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: make confg-docs Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by codex review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: track_with_policy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: minor change Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: mem granularity mb to kb Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by scan_memory_on_exhausted comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: by review comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: typo Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
 discord9
|
56ee8baa3f |
feat: admin gc table/regions (#7619)
* feat: gc table Signed-off-by: discord9 <discord9@163.com> * test: admin gc Signed-off-by: discord9 <discord9@163.com> * chore: after rebase fix Signed-off-by: discord9 <discord9@163.com> * refactor: GcStats Signed-off-by: discord9 <discord9@163.com> * refactor: use gc ticker for admin gc Signed-off-by: discord9 <discord9@163.com> * fix: region routes override Signed-off-by: discord9 <discord9@163.com> * test: non happy path Signed-off-by: discord9 <discord9@163.com> * refactor: gc job report enum Signed-off-by: discord9 <discord9@163.com> * test: process 0 regions Signed-off-by: discord9 <discord9@163.com> * after rebase Signed-off-by: discord9 <discord9@163.com> * feat: allow manual gc to return error Signed-off-by: discord9 <discord9@163.com> * chore: update proto Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * chore: timeout and update proto Signed-off-by: discord9 <discord9@163.com> * chore: udpate proto Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
 Ruihang Xia
|
b4f0886f80 |
chore: grouping batch open region logs (#7758)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com> |
||
|
Weny Xu
|
6b54fb6c21 |
feat(tracing): propagate mailbox trace context and refine procedure spans (#7726)
* feat(tracing): propagate mailbox trace context and refine procedure spans Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: minor Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
df04267c54 |
fix(repartition): reject writes on deallocating regions during region merge (#7694)
* feat(meta): add write route policy to region route with backward compatibility Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(meta): use partition_expr compatibility accessor in repartition matching Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): introduce staging partition rule enum for repartition instructions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(datanode): plumb staging partition rule enum through heartbeat handlers Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): mark pending-deallocate regions as reject-all during merge staging Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(partition): exclude reject-all regions from write partitioning Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): store staging partition rule enum in region state Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(mito): reject writes in staging when partition rule is reject-all Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): send enter staging instruction with reject-all Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(repartition): preserve reject-all on exit, merge enter-staging instructions, and allow staged bulk writes Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: refactor to ignore all writes Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename StagingPartitionRule to StagingPartitionDirective across staging flow Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add comments Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: clippy Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: nit Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
 dennis zhuangandCopilot
|
6d998c043e |
feat: implements anomaly_score_iqr, anomaly_score_mad etc. (#7681)
* feat: implements anomaly_score_iqr, anomaly_score_mad and anomaly_score_zscore functions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: forgot result file Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: return NaN and zero for scale=0 Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: comment Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: signature, cast and degenerate case Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: tweak comment Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: format Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat: update min samples threshold Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: missing closing quote Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: mock engine Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix: oudated comment and test name Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: todo Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * chore: apply suggestions Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Weny Xu
|
d0c610f3c7 |
feat: add partial_drop to DropRequest (#7597)
* feat: add `partial_drop` to `DropRequest` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: handle non-partial-drop drop task Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: remove files immediately Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
discord9
|
67e51b4573 |
feat: gc worker on dropped region (#7537)
* feat: allow clean up for dropped region Signed-off-by: discord9 <discord9@163.com> * clippy Signed-off-by: discord9 <discord9@163.com> * pcr Signed-off-by: discord9 <discord9@163.com> * fix: get access layer correct Signed-off-by: discord9 <discord9@163.com> * chore: invalid gc args Signed-off-by: discord9 <discord9@163.com> * chore: fix test Signed-off-by: discord9 <discord9@163.com> * feat: more defend check Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * feat: messy impl of drop region Signed-off-by: discord9 <discord9@163.com> * feat: add dropped region GC handling module and integrate with GcScheduler Signed-off-by: discord9 <discord9@163.com> * refactor: simplify access layer creation Signed-off-by: discord9 <discord9@163.com> * c Signed-off-by: discord9 <discord9@163.com> * fix: path type Signed-off-by: discord9 <discord9@163.com> * feat: gc handle drop Signed-off-by: discord9 <discord9@163.com> * chore: use proper const Signed-off-by: discord9 <discord9@163.com> * fix: recursive list when check empty dir Signed-off-by: discord9 <discord9@163.com> * per review Signed-off-by: discord9 <discord9@163.com> * refactor: with gc only delete if metadata region Signed-off-by: discord9 <discord9@163.com> * feat: add batch_get_table_route method to SchedulerCtx and MockSchedulerCtx Signed-off-by: discord9 <discord9@163.com> * chore: comment Signed-off-by: discord9 <discord9@163.com> * refactor: retry delete method Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
Weny Xu
|
e7c19a87cd |
fix(datanode): fetch region engine via get_engine for sync/remap operations (#7594)
fix(datanode): fetch region engine via get_engine for sync/remap Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
653740b42a |
feat(meta): add repartition timeouts and metrics (#7587)
* feat(meta): add repartition timeouts and metrics Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add logs and remove default value Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhi
|
007e6cc860 |
chore: batch requests in metric engine (#7535)
* chore: batch requests in metric engine Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: fix clippy Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix: merge multiple schemas Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: add tracing span Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: add metrics for batch put Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: add sparse and dense encoding test case Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: avoid allocation of vec Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: table_id_for_row Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: algorithm to reduce hash lookups by using array indexing Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
Weny Xu
|
2ae20daa62 |
feat: add sync region instruction for repartition procedure (#7562)
* feat: add sync region instruction for repartition procedure This commit introduces a new sync region instruction and integrates it into the repartition procedure flow, specifically for metric engine tables. Changes: - Add SyncRegion instruction type and SyncRegionsReply in instruction.rs - Implement SyncRegionHandler in datanode to handle sync region requests - Add SyncRegion state in repartition procedure to sync newly allocated regions - Integrate sync region step after enter_staging_region for metric engine tables - Add sync_region flag and allocated_region_ids to PersistentContext - Make SyncRegionFromRequest serializable for instruction transmission - Add test utilities and mock support for sync region operations The sync region step is conditionally executed based on the table engine type, ensuring that newly allocated regions in metric engine tables are properly synced from their source regions before proceeding with manifest remapping. Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: add logs Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(repartition): improve staging region handling and support metric engine repartition - Reorder sync region flow: move SyncRegion from EnterStagingRegion to RepartitionStart to sync before applying staging - Add ExitStaging metadata update state to properly clear staging leader info after repartition completes - Update build_template_from_raw_table_info to optionally skip metric engine internal columns when creating region requests - Fix region state transition: set_dropping now expects specific state (Staging or Writable) for proper validation - Adjust region drop and copy handlers to handle staging regions correctly - Add comprehensive test cases for metric engine SPLIT/MERGE partition operations on physical tables with logical tables - Improve logging for table route updates, region drops, and repartition operations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: removes code duplication Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: update result Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: refine comments Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add error strategy support for flush region and flush pending deallocate regions - **Add `ErrorStrategy` enum** in `procedure/utils.rs`: - Supports `Ignore` and `Retry` strategies for error handling - Refactor `flush_region` to accept `error_strategy` parameter - Extract `handle_flush_region_reply` helper function for better code organization - **Add pending deallocate region support**: - Add `pending_deallocate_region_ids` field to `PersistentContext` - Implement `flush_pending_deallocate_regions` in `EnterStagingRegion` state - Flush pending deallocate regions before entering staging regions to ensure data consistency - **Update error handling**: - `flush_leader_region`: Use `ErrorStrategy::Ignore` to skip unreachable datanodes - `sync_region`: Use `ErrorStrategy::Retry` for critical operations - `enter_staging_region`: Use `ErrorStrategy::Retry` when flushing pending deallocate regions This change improves the robustness of the repartition procedure by: 1. Providing flexible error handling strategies for flush operations 2. Ensuring pending deallocate regions are properly flushed before repartitioning 3. Preventing data inconsistency during region migration Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: compile Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
f3e2d333e4 |
feat(repartition): implement region allocation for repartition procedure (#7534)
* refactor: rename WalOptionsAllocator to WalProvider The name "WalOptionsAllocator" was misleading because: - For RaftEngine variant, it doesn't actually allocate anything - The actual allocation logic lives in KafkaTopicPool "WalProvider" better describes its role as providing WAL options based on the configured WAL backend (RaftEngine or Kafka). Changes: - Rename `WalOptionsAllocator` to `WalProvider` - Rename `WalOptionsAllocatorRef` to `WalProviderRef` - Rename `build_wal_options_allocator` to `build_wal_provider` - Rename module `wal_options_allocator` to `wal_provider` - Rename error types: `BuildWalOptionsAllocator` -> `BuildWalProvider`, `StartWalOptionsAllocator` -> `StartWalProvider` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(meta): extract allocator traits from TableMetadataAllocator Refactor TableMetadataAllocator to use trait-based dependency injection for better testability and separation of concerns. Changes: - Add `ResourceIdAllocator` trait to abstract ID allocation - Add `WalOptionsAllocator` trait to abstract WAL options allocation - Implement traits for `Sequence` and `WalProvider` - Remove duplicate `allocate_region_wal_options` function - Rename `table_id_sequence` to `table_id_allocator` for consistency - Rename `TableIdSequenceHandler` to `TableIdAllocatorHandler` Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta): add max_region_number tracking to PhysicalTableRouteValue Add `max_region_number` field to track the highest region number ever allocated for a table. This value only increases when regions are added and never decreases when regions are dropped, ensuring unique region numbers across the table's lifetime. Changes: - Add `max_region_number` field to `PhysicalTableRouteValue` - Implement custom `Deserialize` for backward compatibility - Update `update_region_routes` to maintain max_region_number - Calculate max_region_number from region_routes in `new()` Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: extract TableRouteAllocator trait from TableMetadataAllocator - Add TableRouteAllocator trait for abstracting region route allocation - Implement blanket impl for all PeerAllocator types - Add PeerAllocator impl for Arc<T> to support trait object delegation - Update TableMetadataAllocator to use TableRouteAllocatorRef Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: rename TableRouteAllocator to RegionRoutesAllocator - Rename table_route.rs to region_routes.rs - Rename TableRouteAllocator trait to RegionRoutesAllocator - Rename wal_option.rs to wal_options.rs for consistency - Update TableMetadataAllocator to use new naming Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(meta-srv): implement region allocation for repartition procedure This commit implements the region allocation phase of the repartition procedure, which handles allocating new regions when a table needs to be split into more partitions. Key changes: - Refactor `RegionRoutesAllocator::allocate` to accept `(region_number, partition_expr)` tuples for more flexible region number assignment - Simplify `AllocationPlanEntry` by removing `regions_to_allocate` and `regions_to_deallocate` fields (now derived from source/target counts) - Add `convert_allocation_plan_to_repartition_plan` function to handle allocation, equal, and deallocation cases - Fix `RepartitionPlanEntry::allocate_regions()` to return target regions (was incorrectly returning source regions) - Implement complete `AllocateRegion` state with: - Region route allocation via `RegionRoutesAllocator` - WAL options allocation via `WalOptionsAllocator` - Operating region registration for concurrency control - Region creation on datanodes via `CreateTableExecutor` - Table route metadata update - Add `TableRouteValue::max_region_number()` helper method - Add comprehensive unit tests for plan conversion and allocation logic Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
Weny Xu
|
2f242927a8 |
feat(repartition): implement region deallocation for repartition procedure (#7522)
* feat: implement deallocate regions for repartition procedure Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(metric-engine): add force flag to drop physical regions with associated logical regions Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: update table metadata after deallocating regions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: update proto Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
jeremyhiandCopilot
|
898e84898c |
feat!: make heartbeat config only in metasrv (#7510)
* feat: make heartbeat config only in metasrv Signed-off-by: jeremyhi <fengjiachun@gmail.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: refine config doc Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: make the heartbeat setup simple Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: by comment Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: revert config Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat: proto update Signed-off-by: jeremyhi <fengjiachun@gmail.com> * chore: fix sqlness wrong cfg Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Weny Xu
|
2d756b24c8 |
feat: implement RemapManifest and ApplyStagingManifest for repartition procedure (#7509)
* feat: add RemapManifest and ApplyStagingManifest heartbeat handler Signed-off-by: WenyXu <wenymedia@gmail.com> * feat: add `RemapManifest` and `ApplyStagingManifest` states for repartition Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> * chore: apply suggestions from CR Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
