* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record catalog and schema in slow queries
Add catalog and schema context to slow query records while appending the new columns after existing fields to preserve column order.
- `src/common/frontend/src/slow_query_event.rs`: extend `SlowQueryEvent` schema and rows with `catalog_name` and `schema_name`, and cover append-only ordering.
- `src/catalog/src/process_manager.rs`: carry catalog and schema through `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: capture context for SQL, plan, and PromQL slow query timers.
- `tests-integration/tests/sql.rs`: assert MySQL and PostgreSQL slow query records include catalog and schema.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: address slow query review comment
Use `String::clone` when writing slow query catalog and schema values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep slow query schema only
Remove the slow query `catalog_name` column and keep `schema_name` as a non-null tag dimension.
- `src/common/frontend/src/slow_query_event.rs`: expose only `schema_name` in `SlowQueryEvent` rows and mark it as a tag.
- `src/catalog/src/process_manager.rs`: stop carrying catalog context in `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: pass only schema context to slow query timers.
- `tests-integration/tests/sql.rs`: assert slow query records include `schema_name` without `catalog_name`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: schema name semantic should be field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(json2): type hint
* test(datatypes): add JsonSettings serde round-trip test
* minor refactor
This reverts commit 7ff5a5249a09be5396536284fe822b5761ef4e6a.
* fix: code review
* chore: declare GreptimeDB Enterprise License for enterprise-gated sources
The `enterprise`-feature-gated sources (triggers, mito2 extension) were
excluded from the Apache-2.0 header check but carried no license of their
own. Declare a separate GreptimeDB Enterprise License and enforce it.
- Add LICENSE-ENTERPRISE (open-core split; core stays Apache-2.0).
- Add an Enterprise License header to each enterprise source file.
- Add licenserc-enterprise.toml and a second hawkeye step in CI to enforce
the Enterprise header on exactly those files.
- Cross-reference the two complementary file lists; document the layout in
licenses/README.md and the README License section.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: reference per-customer Enterprise Agreement instead of a terms URL
There is no public enterprise-terms page; each customer signs an individually
negotiated agreement. Point the license at a "separate written commercial
agreement with GrepTime Inc." and direct readers to the existing contact page.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(metric-engine): report query load under physical region id
Propagate the physical region ID through the scanner, record batch
stream, and query engine so that query-load metrics (CPU time, scanned
bytes) are attributed to the correct physical region rather than always
to the logical region.
- `src/store-api/src/region_engine.rs` — add `query_load_region_id` to
`ScannerProperties` and `set_query_load_region_id` to `RegionScanner`
trait
- `src/mito2/src/read/seq_scan.rs`,
`src/mito2/src/read/series_scan.rs`,
`src/mito2/src/read/unordered_scan.rs` — implement the new trait
method on each scanner
- `src/common/recordbatch/src/adapter.rs` — carry region id through
`RecordBatchStreamAdapter` into `RecordBatchMetrics`
- `src/table/src/table/scan.rs` — expose region id on `RegionScanExec`
- `src/query/src/datafusion.rs` — extract region id from the physical
plan and set it on the output stream
- `src/query/src/dist_plan/merge_scan.rs` — use metrics-contained region
id in query-load reporting, falling back to the logical region id
- `src/metric-engine/src/engine/read.rs` — set region id on the metric
engine scanner
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): satisfy clippy for query load region id
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): ignore missing query load region ids
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
* Initial plan
* fix: guard structured json alignment to fix Clippy CI failure
- Add `is_structured_json_field` function that only returns true for
fields with both JSON extension type AND Struct Arrow data type
- Replace all usages of `is_json_extension_type` / `has_json_extension_field`
with `is_structured_json_field` to prevent legacy JSONB binary columns
from entering structured JSON alignment paths
- Fix logic in `FlatProjectionMapper::new_with_read_columns` to guard
JSON type hint concretization for JSON2 columns only
- Fix `create_column` in show_create_table.rs to only emit JSON structure
settings for JSON2 columns
- Move `mod tests` to end of flat_projection.rs to fix clippy::items_after_test_module
- Add tests for legacy JSON behavior
* fix: guard narrow_read_columns_by_json_type_hint with is_structured_json_field check
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* feat: expose region read load through Prometheus metrics and heartbeat
Introduce region-level query load tracking (CPU time and scanned bytes)
collected by `RegionScanExec`, exposed via Prometheus metrics and optionally
reported through heartbeat region stats.
- **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add
`greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`,
and `greptime_mito_region_written_bytes_since_open` gauge metrics.
- **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`,
`src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with
`IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle
management (init, reset, remove-on-drop).
- **RegionStatistic** (`src/store-api/src/region_engine.rs`,
`src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and
`query_scanned_bytes` fields.
- **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from
metadata and data regions.
- **Heartbeat** (`src/datanode/src/heartbeat.rs`,
`src/common/meta/src/datanode.rs`): Relay region query load via heartbeat
`RegionStat`; add test.
- **Query engine** (`src/query/src/options.rs`,
`src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`,
`src/query/src/dist_plan/merge_scan.rs`,
`src/query/src/dist_plan/analyzer.rs`,
`src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config;
wire `RegionScanExec` to accumulate CPU time and scanned bytes.
- **Table scan** (`src/table/src/table/scan.rs`,
`src/table/src/table/metrics.rs`): Wire table scan metrics.
- **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`,
`config/frontend.example.toml`, `config/config.md`): Add example config and
documentation for `enable_region_query_load_report`.
- **Tests** (`src/mito2/src/engine/basic_test.rs`,
`src/mito2/src/engine/close_test.rs`,
`src/cmd/tests/load_config_test.rs`,
`src/flow/src/adapter.rs`): Add unit tests for region query load reporting
and metric cleanup on region close; set default config values.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move region read load report config from query layer to mito engine
Move the `enable_region_query_load_report` setting from query-level config
(`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config
(`MitoConfig`), and expose it through the `RegionScanner` trait instead
of `ScanRequest`/`PrepareRequest`.
- Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs`
- Scan region plumbing: `src/mito2/src/read/scan_region.rs`
- RegionScanner trait: `src/store-api/src/region_engine.rs`
- Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs`
- RegionScanExec: `src/table/src/table/scan.rs`
- Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs`
- Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs`
- Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move region query load report config from MitoConfig to LoggingOptions
Relocate the `enable_region_query_load_report` setting from
`MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`),
and thread it into `MitoEngineBuilder` instead of reading from
the engine config directly. This makes the region read-load
reporting a per-node logging/observability concern rather than
a per-engine storage setting.
- `config/config.md`
- `config/datanode.example.toml`
- `config/standalone.example.toml`
- `src/common/telemetry/src/logging.rs`
- `src/datanode/src/datanode.rs`
- `src/mito2/src/config.rs`
- `src/mito2/src/engine.rs`
- `src/mito2/src/region.rs`
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: report region query load on stream drop instead of stream end
Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()`
to `Drop::drop()` so that region query load is reported even when the
stream is dropped prematurely (not just when fully consumed).
Affected files:
- `src/table/src/table/scan.rs`
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: make region query load reporting configurable
Introduce `enable_region_query_load_report` flag to optionally report
per-region `query_cpu_time` and `query_scanned_bytes` metrics instead
of always creating them. When disabled, the Prometheus gauges are not
created (`None`), avoiding metric churn for workloads that do not
need query-level load tracking.
- `src/common/meta/src/datanode.rs` — Placeholder fields for query load
- `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset
- `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener`
- `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop`
- `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open
- `src/mito2/src/worker/handle_create.rs` — Pass flag on region create
- `src/mito2/src/worker/handle_open.rs` — Pass flag on region open
- `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder`
- `src/mito2/src/test_util.rs` — Test helpers for both modes
- `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases
- `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove elapsed_compute metric from scan stream
The elapsed_compute metric conflated poll-wait time with actual CPU
computation, making it misleading. Removed the metric and its
recording path from StreamMetrics and StreamWithMetricWrapper.
Added a test asserting that poll duration is not reported as
elapsed_compute.
- `src/table/src/table/metrics.rs` — removed elapsed_compute field,
builder, and record_elapsed_compute method
- `src/table/src/table/scan.rs` — removed record_elapsed_compute
call; added SlowRecordBatchStream test helper and
wrapper_poll_time_is_not_elapsed_compute test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable region query load report for compaction scans
Compaction scans are internal operations initiated by the engine,
not user queries. Disable region query load reporting when the
scan input is marked as compaction to avoid misleading load metrics.
- `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report`
to `false` when compaction is enabled; add unit test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test: add `enable_per_region_metrics` config to HTTP integration test
- Enable per-region metrics config in HTTP test setup
\`tests-integration/tests/http.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove region query load reporting tests and helpers
Remove the region query load reporting feature from the codebase,
including tests, test utilities, and helper infrastructure that were
part of this now-deprecated functionality.
Specifically:
- Remove region query load reporting tests from
`src/mito2/src/engine/basic_test.rs` and
`src/table/src/table/scan.rs`, and the region close metrics test
from `src/mito2/src/engine/close_test.rs`
- Remove region query load report test utilities and simplify engine
construction helpers in `src/mito2/src/test_util.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf: avoid disabled region query load timing
Summary:
- Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled.
- Keep region query-load CPU accounting active only when reporting is enabled.
Files:
- `src/table/src/table/scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move per-region query load reporting from storage to query engine
Move `enable_per_region_metrics` from datanode to frontend config and
migrate query load tracking (CPU time, scanned bytes) from mito2
storage engine to the query engine's distributed scan planner. The
storage-level metrics plumbing and `enable_region_query_load_report`
flag are removed from mito2, `ScanInput`, `ScanRegion`, and
`RegionScanner`. Query-level metrics are now collected in
`merge_scan.rs` via `scan_region_load`.
- `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes`
metrics, `enable_region_query_load_report` plumbing from engine,
region, opener, scanner types, workers
- `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes`
from `RegionStatistic`
- `src/metric-engine/` -- Remove query load fields from
`get_region_statistic`
- `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`;
wire through planner, optimizer, merge scan with `scan_region_load`
metrics
- `src/frontend/` -- Pass `enable_per_region_metrics` into
`QueryOptions`
- `src/common/meta/` -- Remove TODO for query load fields
- `config/` -- Move `enable_per_region_metrics` from datanode to
frontend and standalone example configs
- `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode
config test
- `src/flow/` -- Add `enable_per_region_metrics` default to flownode
options
- `src/table/` -- Remove unused query load fields from scan
- `src/datanode/` -- Remove
`with_enable_region_query_load_report` calls
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove obsolete mito write load metric
Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path.
Related files:
- \`src/mito2/src/metrics.rs\`
- \`src/mito2/src/region.rs\`
- \`src/mito2/src/region/opener.rs\`
- \`src/mito2/src/region_write_ctx.rs\`
- \`src/mito2/src/engine/basic_test.rs\`
- \`src/mito2/src/worker.rs\`
- \`src/mito2/src/config.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: change region query load metrics from gauge to counter
Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from
`IntGaugeVec` to `IntCounterVec` since these values are monotonically
increasing and do not need gauge semantics. Update corresponding `add`
calls to `inc_by` in merge scan reporting.
Files:
- `src/store-api/src/metrics.rs` — metric type and label changes
- `src/query/src/dist_plan/merge_scan.rs` — caller adaptation
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: pass ReadItem directly to report_region_query_load
Move `region_scan_load` call to the caller, so `report_region_query_load`
accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`.
- `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call,
remove stale test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: ensure region query load is reported on MergeScanExec drop
Remove the `enable_per_region_metrics` parameter from `report_region_query_load`
so region load metrics are always emitted. Add a `Drop` impl for
`MergeScanExec` that reports sub-stage metrics when the executor is
dropped, covering edge cases where per-region metric emission was
missed. Add a unit test verifying CPU time and scanned bytes are
recorded on drop.
Affected file: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: gate region query load reporting
Guard drop-time region query load reporting with the configured per-region metrics flag.
Related files:
- \`src/query/src/dist_plan/merge_scan.rs\`
Symbols:
- \`MergeScanExec::drop\`
- \`enable_per_region_metrics\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clean region query load metrics on drop
Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry.
Related files:
- \`src/mito2/src/region.rs\`
Symbols:
- \`MitoRegion::drop\`
- \`REGION_QUERY_CPU_TIME\`
- \`REGION_QUERY_SCANNED_BYTES\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
* feat: track scan output bytes and use them for read costing
Track the actual byte count of each record batch produced by
`RegionScanExec` and use it in place of the aggregated per-plan
`memory_usage` as the `table_scan` cost input. This avoids double
counting bytes that flow through multiple operators.
A named constant `REGION_SCAN_EXEC_NAME` exposes the plan node name
so downstream metric parsers remain correct if the struct is renamed.
Affected files:
- `src/table/src/table/metrics.rs` -- add `output_bytes` counter
- `src/table/src/table/scan.rs` -- record bytes, export name constant
- `src/query/src/dist_plan/merge_scan.rs` -- consume scan bytes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove misleading `mem_used` gauge from scan metrics
The `mem_used` gauge was used with `add()` on every output batch, but a
RegionScan does not hold memory across polls — each batch is yielded
immediately to the upstream operator. The gauge semantics were
incorrect (cumulative `add()` on a gauge) and the value duplicated
`output_bytes` anyway.
Affected files:
- `src/table/src/table/metrics.rs` — drop `mem_used` field and methods
- `src/table/src/table/scan.rs` — remove `record_mem_usage` call
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: use stable plan names for scan byte metrics
Expose `plan_name` alongside rendered plan text so scan byte extraction no longer depends on EXPLAIN formatting. Keep old serialized metrics compatible by defaulting missing `plan_name`.
Affected files:
- `src/common/recordbatch/src/adapter.rs` -- add `plan_name` to `PlanMetrics` and populate it from `ExecutionPlan::name`
- `src/query/src/dist_plan/merge_scan.rs` -- match `REGION_SCAN_EXEC_NAME` through `plan_name`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve merge scan CPU read cost
Record read cost whenever stream metrics are available, default scan bytes to zero, and aggregate scan output bytes across region scan nodes.
Files: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: correct doc comment in `StreamMetrics`
Fix doc comment on `StreamMetrics::new()` to reference the correct
struct name (`StreamMetrics`) instead of the old `MemoryUsageMetrics`.
- `src/table/src/table/metrics.rs` — fix struct name in doc comment
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add datanode runtime options
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add datanode runtime handles
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: wire datanode runtimes into region server
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: route datanode ingestion to ingestion runtime
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add datanode query runtime stream bridge
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: route datanode reads to query runtime
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add datanode global runtimes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: use common datanode runtimes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: run mito scan tasks on query runtime
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: split datanode runtime options
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: share global fallback for datanode runtimes
Use the global runtime as the fallback for datanode query and ingestion
runtimes when datanode-specific pools are not initialized. This avoids
creating unused datanode worker pools in non-datanode services.
Files:
- `src/common/runtime/src/global.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: docs
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: forward query runtime stream metrics
Forward inner stream metrics through the datanode query runtime bridge so
`EXPLAIN ANALYZE` can report plan metrics after stream polling moves to the
query runtime.
Files:
- `src/datanode/src/query_stream.rs`
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: route metric batch puts to ingest runtime
Run the optimized metric batch put path on the datanode ingest runtime so
metric ingestion does not bypass runtime isolation.
Files:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: abort query producer on stream drop
Abort the datanode query runtime producer when the returned read stream is
dropped so cancelled clients do not leave query work running in the
background.
Files:
- `src/datanode/src/query_stream.rs`
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: simplify query stream bridge setup
Create the inner read stream before spawning the datanode query runtime
producer so setup does not use an extra task and initialization channel.
Files:
- `src/datanode/src/region_server.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/runtime-priority:
### Update Datanode Runtime Options and Region Server Logic
- **`global.rs`**: Adjusted `datanode_ingest_rt_size` to utilize all available CPUs for improved performance.
- **`region_server.rs`**: Simplified the collection of `put_requests` and optimized the `put_regions_batch` call for better efficiency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/runtime-priority:
### Remove Redundant Checks and Simplify Code
- **`global.rs`**: Removed the assertion check for already initialized global runtimes to streamline the initialization process.
- **`region_server.rs`**: Simplified the extraction of `Put` requests by removing unnecessary cloning and restructuring the iterator logic.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: remove redundant spawn_datanode_query in RegionServer::handle_read
The outer `spawn_datanode_query` wrapped `handle_read_inner` on the
same runtime, creating a nested spawn that consumed query runtime
threads unnecessarily under concurrent read load. The gRPC handler
already provides runtime isolation, so the inner call is sufficient.
- `src/datanode/src/region_server.rs` — inline `handle_read_inner`
directly instead of spawning onto the datanode query runtime
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: resolve test mismatch and redundant spawn in handle_remote_read
- `src/common/runtime/src/global.rs` — update test assertion to match
default `datanode_ingest_rt_size` of `cpus` instead of `1`
- `src/datanode/src/region_server.rs` — inline `handle_remote_read_inner`
directly instead of spawning onto the datanode query runtime
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: rename datanode runtimes
Summary:
- Rename datanode runtime APIs from `datanode_query` and `datanode_ingest` to `query` and `ingest`.
- Rename runtime config keys from `datanode_query_rt_size` and `datanode_ingest_rt_size` to `query_rt_size` and `ingest_rt_size`.
- Update config docs, example config, and config-loading coverage.
Files:
- `src/common/runtime/src/global.rs`
- `src/common/runtime/src/lib.rs`
- `src/cmd/tests/load_config_test.rs`
- `src/datanode/src/region_server.rs`
- `src/mito2/src/read/pruner.rs`
- `src/mito2/src/read/range_cache.rs`
- `src/mito2/src/read/scan_region.rs`
- `src/mito2/src/read/series_scan.rs`
- `config/datanode.example.toml`
- `config/config.md`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: consolidate runtime options
Summary:
- Embed datanode runtime sizes in shared `RuntimeOptions` and remove the extra `GreptimeOptions` runtime type parameter.
- Use the unified `RuntimeOptions` for datanode global and datanode-specific runtime initialization.
- Update datanode runtime config coverage and ingest runtime default documentation.
Files:
- `src/common/runtime/src/global.rs`
- `src/common/runtime/src/lib.rs`
- `src/cmd/src/options.rs`
- `src/cmd/src/datanode.rs`
- `src/cmd/src/datanode/builder.rs`
- `src/cmd/tests/load_config_test.rs`
- `config/datanode.example.toml`
- `config/config.md`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: guard against double initialization of datanode runtimes
Add an assertion in `init_datanode_runtimes` to panic when global runtimes
are already initialized, preventing silent overwrites.
- `src/common/runtime/src/global.rs` — assert guard in `init_datanode_runtimes`
and test `test_set_datanode_runtimes_panics_after_global_runtimes_initialized`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: table semantic layer information_schema view (Phase 3)
Add `information_schema.table_semantics`, a queryable view over the table
semantic layer. One row per table that carries at least one
`greptime.semantic.*` option: the signal-agnostic keys
(signal_type/source/pipeline/metadata_quality) are promoted to columns and
the remaining signal-specific keys are folded into a `semantic_options`
JSON string. Tables with no semantic key are excluded.
Stacked on Phase 2.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: address PR review on table_semantics
- fold JSON serialization failure into None instead of unwrap/panic
- drop per-row Vec allocation in predicate eval; use a fixed array
- align RFC view name with the shipped `table_semantics`
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update results
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: support defer_on_missing_source for pending flow creation
Add `defer_on_missing_source` flow option that allows creating flows
even when source tables do not yet exist. The flow enters a pending
state and is automatically activated when source tables become available.
Key changes:
- New `FlowStatus::PendingSources` and fields in `FlowInfoValue` for
unresolved source table names and last activation error
- `defer_on_missing_source` create-time-only option: stripped from
runtime/flownode `CreateRequest` but preserved in metadata for
SQL round-trip (`SHOW CREATE FLOW`, `information_schema.flows`)
- `CreateFlowProcedure` creates pending metadata when sources are
missing and `defer_on_missing_source=true`; falls back to
`FlowType::Batching` for missing-source flows
- `PendingFlowReconcileManager` in meta-srv periodically checks
pending flows and activates them when source tables resolve
- `ActivatePendingFlowProcedure` handles activation: allocates peers,
creates flows on flownodes, updates metadata, invalidates cache
- `OR REPLACE` properly handles pending<->active transitions,
including peer allocation and flownode flow teardown
- `FlowMetadataAllocator::alloc_peers` for peer allocation at
activation time
- Validated flow options: only `defer_on_missing_source` allowed;
unknown options rejected
- Known issue: standalone mode does not support flownodes, so
pending flow flush/sink behavior covered only in distributed
sqlness; operator and meta unit tests cover activation logic
Tests:
- operator `determine_flow_type_for_source_state` (3 passed)
- common-meta `create_flow` (19 passed) including replacement
- common-meta `activate_flow` (4 passed)
- meta-srv `flow` (11 passed)
- sqlness: `flow_pending` covers create/replace/round-trip
Signed-off-by: discord9 <discord9@163.com>
* chore: simplify pending flow PR scope
Reduce PR #8124 to the metadata-only MVP after complexity review.
Changes:
- Remove automatic activation procedure and meta-srv reconcile wiring
- Remove activation tests and activation-only metadata fields
- Reject cross-state pending<->active `OR REPLACE` transitions for now
- Keep pending metadata creation and SQL round-trip behavior
- Allow `DROP FLOW` for pending flows without routes
- Reduce flow_pending sqlness to metadata/round-trip/drop coverage only
Deferred follow-ups are documented locally in `.tmp/tasks/pending-defer-semantics/deferred-followups.md` and intentionally not committed.
Tests:
- `cargo test -p operator determine_flow_type_for_source_state`
- `cargo test -p common-meta create_flow`
- `cargo test -p common-meta drop_flow`
- `cargo sqlness bare --test-filter flow_pending --bins-dir /mnt/nvme_rust/rust-targets/pending_defer/debug`
Signed-off-by: discord9 <discord9@163.com>
* test: cover pending flow metadata edge cases
Signed-off-by: discord9 <discord9@163.com>
* test: fix pending flow metadata test lint
Signed-off-by: discord9 <discord9@163.com>
* docs: document pending flow metadata fields
Signed-off-by: discord9 <discord9@163.com>
* chore: more sleep when test
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
* feat: expose node info for placement selectors
Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.
Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.
Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: skip datanode node info without stats
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: avoid unnecessary workload clones
Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.
Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: require frontend origin address
Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.
Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: reuse resolved frontend address
Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.
Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve datanode lease liveness
Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.
Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Remove stale datanode lease helper
- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/env-based-region-selector-oss: simplify lease discovery
- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
