* fix: record catalog and schema in slow queries
Add catalog and schema context to slow query records while appending the new columns after existing fields to preserve column order.
- `src/common/frontend/src/slow_query_event.rs`: extend `SlowQueryEvent` schema and rows with `catalog_name` and `schema_name`, and cover append-only ordering.
- `src/catalog/src/process_manager.rs`: carry catalog and schema through `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: capture context for SQL, plan, and PromQL slow query timers.
- `tests-integration/tests/sql.rs`: assert MySQL and PostgreSQL slow query records include catalog and schema.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: address slow query review comment
Use `String::clone` when writing slow query catalog and schema values.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: keep slow query schema only
Remove the slow query `catalog_name` column and keep `schema_name` as a non-null tag dimension.
- `src/common/frontend/src/slow_query_event.rs`: expose only `schema_name` in `SlowQueryEvent` rows and mark it as a tag.
- `src/catalog/src/process_manager.rs`: stop carrying catalog context in `SlowQueryTimer`.
- `src/frontend/src/instance.rs`: pass only schema context to slow query timers.
- `tests-integration/tests/sql.rs`: assert slow query records include `schema_name` without `catalog_name`.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: schema name semantic should be field
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix: typo
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(agents): add per-crate guides, architecture invariants, and generated-files list
Add agent/contributor navigation docs modeled on the AGENTS.md convention:
- Per-crate AGENTS.md for hot crates (mito2, metric-engine, flow, frontend,
meta-srv): module map, read/write paths, change-coupling points, test
commands, and gotchas.
- .agents/architecture-invariants.md: repo-wide rules that clippy and the
style guide do not cover (format compatibility, crate layering, async
runtimes, error handling, experimental gating, the DataFusion fork).
- .agents/generated-files.md: tool-generated artifacts that must not be
hand-edited (sqlness .result, config.md, dashboards, build.rs output, proto).
- Anchor the .gitignore CLAUDE.md/AGENTS.md rules to the repo root so per-crate
AGENTS.md files are tracked while root-level personal config stays ignored.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: update crate AGENTS.md and fix config.md path
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* docs(agents): fix DataFusion patch layout and SQL query lifecycle order
Address review feedback on #8346:
- architecture-invariants: the DataFusion sub-crates pin an exact crates.io
version in [workspace.dependencies] and are redirected to the fork rev in
[patch.crates-io]; the two sections hold different forms, not the same rev.
- frontend: the SQL query lifecycle runs the pre_parsing/post_parsing
interceptors around parsing, before the per-statement permission check.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: expose region read load through Prometheus metrics and heartbeat
Introduce region-level query load tracking (CPU time and scanned bytes)
collected by `RegionScanExec`, exposed via Prometheus metrics and optionally
reported through heartbeat region stats.
- **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add
`greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`,
and `greptime_mito_region_written_bytes_since_open` gauge metrics.
- **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`,
`src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with
`IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle
management (init, reset, remove-on-drop).
- **RegionStatistic** (`src/store-api/src/region_engine.rs`,
`src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and
`query_scanned_bytes` fields.
- **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from
metadata and data regions.
- **Heartbeat** (`src/datanode/src/heartbeat.rs`,
`src/common/meta/src/datanode.rs`): Relay region query load via heartbeat
`RegionStat`; add test.
- **Query engine** (`src/query/src/options.rs`,
`src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`,
`src/query/src/dist_plan/merge_scan.rs`,
`src/query/src/dist_plan/analyzer.rs`,
`src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config;
wire `RegionScanExec` to accumulate CPU time and scanned bytes.
- **Table scan** (`src/table/src/table/scan.rs`,
`src/table/src/table/metrics.rs`): Wire table scan metrics.
- **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`,
`config/frontend.example.toml`, `config/config.md`): Add example config and
documentation for `enable_region_query_load_report`.
- **Tests** (`src/mito2/src/engine/basic_test.rs`,
`src/mito2/src/engine/close_test.rs`,
`src/cmd/tests/load_config_test.rs`,
`src/flow/src/adapter.rs`): Add unit tests for region query load reporting
and metric cleanup on region close; set default config values.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move region read load report config from query layer to mito engine
Move the `enable_region_query_load_report` setting from query-level config
(`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config
(`MitoConfig`), and expose it through the `RegionScanner` trait instead
of `ScanRequest`/`PrepareRequest`.
- Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs`
- Scan region plumbing: `src/mito2/src/read/scan_region.rs`
- RegionScanner trait: `src/store-api/src/region_engine.rs`
- Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs`
- RegionScanExec: `src/table/src/table/scan.rs`
- Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs`
- Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs`
- Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move region query load report config from MitoConfig to LoggingOptions
Relocate the `enable_region_query_load_report` setting from
`MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`),
and thread it into `MitoEngineBuilder` instead of reading from
the engine config directly. This makes the region read-load
reporting a per-node logging/observability concern rather than
a per-engine storage setting.
- `config/config.md`
- `config/datanode.example.toml`
- `config/standalone.example.toml`
- `src/common/telemetry/src/logging.rs`
- `src/datanode/src/datanode.rs`
- `src/mito2/src/config.rs`
- `src/mito2/src/engine.rs`
- `src/mito2/src/region.rs`
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: report region query load on stream drop instead of stream end
Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()`
to `Drop::drop()` so that region query load is reported even when the
stream is dropped prematurely (not just when fully consumed).
Affected files:
- `src/table/src/table/scan.rs`
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: make region query load reporting configurable
Introduce `enable_region_query_load_report` flag to optionally report
per-region `query_cpu_time` and `query_scanned_bytes` metrics instead
of always creating them. When disabled, the Prometheus gauges are not
created (`None`), avoiding metric churn for workloads that do not
need query-level load tracking.
- `src/common/meta/src/datanode.rs` — Placeholder fields for query load
- `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset
- `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener`
- `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop`
- `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open
- `src/mito2/src/worker/handle_create.rs` — Pass flag on region create
- `src/mito2/src/worker/handle_open.rs` — Pass flag on region open
- `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder`
- `src/mito2/src/test_util.rs` — Test helpers for both modes
- `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases
- `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove elapsed_compute metric from scan stream
The elapsed_compute metric conflated poll-wait time with actual CPU
computation, making it misleading. Removed the metric and its
recording path from StreamMetrics and StreamWithMetricWrapper.
Added a test asserting that poll duration is not reported as
elapsed_compute.
- `src/table/src/table/metrics.rs` — removed elapsed_compute field,
builder, and record_elapsed_compute method
- `src/table/src/table/scan.rs` — removed record_elapsed_compute
call; added SlowRecordBatchStream test helper and
wrapper_poll_time_is_not_elapsed_compute test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable region query load report for compaction scans
Compaction scans are internal operations initiated by the engine,
not user queries. Disable region query load reporting when the
scan input is marked as compaction to avoid misleading load metrics.
- `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report`
to `false` when compaction is enabled; add unit test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test: add `enable_per_region_metrics` config to HTTP integration test
- Enable per-region metrics config in HTTP test setup
\`tests-integration/tests/http.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove region query load reporting tests and helpers
Remove the region query load reporting feature from the codebase,
including tests, test utilities, and helper infrastructure that were
part of this now-deprecated functionality.
Specifically:
- Remove region query load reporting tests from
`src/mito2/src/engine/basic_test.rs` and
`src/table/src/table/scan.rs`, and the region close metrics test
from `src/mito2/src/engine/close_test.rs`
- Remove region query load report test utilities and simplify engine
construction helpers in `src/mito2/src/test_util.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf: avoid disabled region query load timing
Summary:
- Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled.
- Keep region query-load CPU accounting active only when reporting is enabled.
Files:
- `src/table/src/table/scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move per-region query load reporting from storage to query engine
Move `enable_per_region_metrics` from datanode to frontend config and
migrate query load tracking (CPU time, scanned bytes) from mito2
storage engine to the query engine's distributed scan planner. The
storage-level metrics plumbing and `enable_region_query_load_report`
flag are removed from mito2, `ScanInput`, `ScanRegion`, and
`RegionScanner`. Query-level metrics are now collected in
`merge_scan.rs` via `scan_region_load`.
- `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes`
metrics, `enable_region_query_load_report` plumbing from engine,
region, opener, scanner types, workers
- `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes`
from `RegionStatistic`
- `src/metric-engine/` -- Remove query load fields from
`get_region_statistic`
- `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`;
wire through planner, optimizer, merge scan with `scan_region_load`
metrics
- `src/frontend/` -- Pass `enable_per_region_metrics` into
`QueryOptions`
- `src/common/meta/` -- Remove TODO for query load fields
- `config/` -- Move `enable_per_region_metrics` from datanode to
frontend and standalone example configs
- `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode
config test
- `src/flow/` -- Add `enable_per_region_metrics` default to flownode
options
- `src/table/` -- Remove unused query load fields from scan
- `src/datanode/` -- Remove
`with_enable_region_query_load_report` calls
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove obsolete mito write load metric
Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path.
Related files:
- \`src/mito2/src/metrics.rs\`
- \`src/mito2/src/region.rs\`
- \`src/mito2/src/region/opener.rs\`
- \`src/mito2/src/region_write_ctx.rs\`
- \`src/mito2/src/engine/basic_test.rs\`
- \`src/mito2/src/worker.rs\`
- \`src/mito2/src/config.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: change region query load metrics from gauge to counter
Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from
`IntGaugeVec` to `IntCounterVec` since these values are monotonically
increasing and do not need gauge semantics. Update corresponding `add`
calls to `inc_by` in merge scan reporting.
Files:
- `src/store-api/src/metrics.rs` — metric type and label changes
- `src/query/src/dist_plan/merge_scan.rs` — caller adaptation
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: pass ReadItem directly to report_region_query_load
Move `region_scan_load` call to the caller, so `report_region_query_load`
accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`.
- `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call,
remove stale test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: ensure region query load is reported on MergeScanExec drop
Remove the `enable_per_region_metrics` parameter from `report_region_query_load`
so region load metrics are always emitted. Add a `Drop` impl for
`MergeScanExec` that reports sub-stage metrics when the executor is
dropped, covering edge cases where per-region metric emission was
missed. Add a unit test verifying CPU time and scanned bytes are
recorded on drop.
Affected file: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: gate region query load reporting
Guard drop-time region query load reporting with the configured per-region metrics flag.
Related files:
- \`src/query/src/dist_plan/merge_scan.rs\`
Symbols:
- \`MergeScanExec::drop\`
- \`enable_per_region_metrics\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clean region query load metrics on drop
Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry.
Related files:
- \`src/mito2/src/region.rs\`
Symbols:
- \`MitoRegion::drop\`
- \`REGION_QUERY_CPU_TIME\`
- \`REGION_QUERY_SCANNED_BYTES\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer identity (Phase 1)
Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.
Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
(signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
defined now, populated by later phases), value constants, the internal
`greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
(metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
the v1 model), logs (log/opentelemetry), and Prometheus remote write
(metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
`_services` / `_operations` lookup tables keep the unstamped context and carry no
semantic identity (cross-table relationships are out of scope).
Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.
Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: prom batcher not cover and white list for semantic keys/values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: typo
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: global switch for creating table automatically
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: make auto_create_table as comment by default
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: respect gloabl switch for metric engine
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: provide query information for post_execute interceptor
* test: update for tests-integration
* feat: make interceptor available to prometheus serialization
* feat: revert post_execute change
* feat: add expr to pre_execute and remove serialization interceptor
* chore: lint
* feat: expose node info for placement selectors
Return `NodeInfo` from `PeerDiscovery` methods and keep OSS selectors mapping back to `Peer`.
Carry `__greptime_origin_frontend.addr` from frontend create-table DDLs into selector `extensions`, and thread `PeerAllocContext` through table-route allocation.
Persist datanode `NodeInfo` when heartbeat stats are absent so collected env vars remain available after restart.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: skip datanode node info without stats
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: avoid unnecessary workload clones
Skip workload cloning for inactive nodes and for active node-info lookups without workload filters.
Files: `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: require frontend origin address
Require `StatementExecutor` to carry a concrete frontend origin address and always attach it to meta DDL query contexts.
Files: `src/operator/src/statement.rs`, `src/operator/src/statement/ddl.rs`, `src/operator/src/utils.rs`, `src/frontend/src/instance/builder.rs`, `src/frontend/src/heartbeat.rs`, `src/flow/src/server.rs`, `src/cmd/src/standalone.rs`, `src/cmd/src/flownode.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: reuse resolved frontend address
Resolve the frontend peer address once in the frontend builder, store it on the instance, and reuse it for heartbeat and flow invoker origins.
Files: `src/frontend/src/instance/builder.rs`, `src/frontend/src/instance.rs`, `src/frontend/src/heartbeat.rs`, `src/cmd/src/frontend.rs`, `src/cmd/src/standalone.rs`, `src/frontend/src/frontend.rs`, `src/frontend/src/heartbeat/tests.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve datanode lease liveness
Filter active datanode node infos through lease timestamps and workloads while preserving node info fields such as reported env vars.
Files: `src/meta-srv/src/discovery/utils.rs`, `src/meta-srv/src/discovery/lease.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Remove stale datanode lease helper
- `discovery`: remove the obsolete `alive_datanodes` helper and related tests in `src/meta-srv/src/discovery/utils.rs` and `src/meta-srv/src/discovery/lease.rs`
- `integration`: update cluster and standalone setup paths in `tests-integration/src/cluster.rs` and `tests-integration/src/standalone.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/env-based-region-selector-oss: simplify lease discovery
- `lease-discovery`: simplify logic and remove unused utilities in `src/meta-srv/src/discovery/lease.rs` and `src/meta-srv/src/discovery/utils.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: support reporting env vars in heartbeat messages to metasrv
Add `heartbeat_env_vars` config option for datanode and frontend. When
configured, the specified environment variable values are read at startup
and sent to metasrv in every heartbeat via the `extensions` map. Metasrv
extracts and stores them in `NodeInfo` for use in routing decisions
(e.g. AZ-aware region placement).
- Add `EnvVars` helper in `common/meta/src/datanode.rs` following the
existing `GcStat` extension pattern with `into_extensions`/`from_extensions`
- Add `env_vars: HashMap<String, String>` field to `NodeInfo` in
`common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat
- Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`,
`FrontendOptions`, and `StandaloneOptions`
- Inject env vars into heartbeat `extensions` in both datanode and frontend
heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`)
- Extract env vars from `req.extensions` in all three metasrv
`CollectXxxClusterInfoHandler`s
- Update `NodeInfo` construction sites in `meta-client`,
`discovery/lease.rs`, and `standalone/information_extension.rs`
- Update expected TOML output in `tests-integration/tests/http.rs`
- Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: address heartbeat env review feedback
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: log error on deserialization failure
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: send heartbeat env vars once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: resend heartbeat env vars after reconnect
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: keep env vars in every heartbeat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(server): describe EXPLAIN statements so bind parameters work
`do_describe_inner` only planned `Insert`/`Query`/`Delete`, so
`EXPLAIN` and `EXPLAIN ANALYZE` fell through to the non-plan branch
and had no parameter-type inference. At Bind time the Postgres
handler then reported `unsupported_parameter_type` even though the
inner query would have worked on its own.
Recurse one level into `Statement::Explain` so that an EXPLAIN
wrapping a plannable statement goes through the same describe path.
Adds a tokio-postgres integration test that exercises
`EXPLAIN`/`EXPLAIN ANALYZE` over the extended query protocol.
Fixes#8029
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
* refactor(server): extract plannable-inner check into closure
Reduce duplication between the direct match and the EXPLAIN inner match
by factoring out is_inner_plannable. Behaviour unchanged.
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
---------
Signed-off-by: BootstrapperSBL <yvanwww@gmail.com>
* feat/auto-schema-align:
- **Error Handling Improvements**:
- Removed `CatalogSnafu` context from various `.await` calls in `dashboard.rs`, `influxdb.rs`, `jaeger.rs`, `prometheus.rs`, `event.rs`, and `pipeline.rs` to streamline error handling.
- **Prometheus Store Enhancements**:
- Added support for auto-creating tables and adding missing Prometheus tag columns in `prom_store.rs` and `pending_rows_batcher.rs`.
- Introduced `PendingRowsSchemaAlterer` trait for schema alterations in `pending_rows_batcher.rs`.
- **Test Additions**:
- Added tests for new Prometheus store functionalities in `prom_store.rs` and `pending_rows_batcher.rs`.
- **Error Message Improvements**:
- Enhanced error messages for catalog access in `error.rs`.
- **Server Configuration Updates**:
- Updated server configuration to include Prometheus store options in `server.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* reformat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add DataTypes Error Handling and Column Renaming Logic
- **`error.rs`**: Introduced a new `DataTypes` error variant to handle errors from `datatypes::error::Error`. Updated `ErrorExt` implementation to include `DataTypes`.
- **`pending_rows_batcher.rs`**: Added functions `find_prom_special_column_names` and `rename_prom_special_columns_for_existing_schema` to handle renaming of special Prometheus columns. Updated `build_prom_create_table_schema` to simplify error handling with
`ConcreteDataType`.
- **Tests**: Added a test case `test_rename_prom_special_columns_for_existing_schema` to verify the renaming logic for Prometheus special columns.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- Refactored `PendingRowsBatcher` to accommodate Prometheus record batches:
- Introduced `accommodate_record_batch_for_target_schema` to normalize incoming record batches against existing table schemas.
- Removed `collect_missing_prom_tag_columns` and `rename_prom_special_columns_for_existing_schema` in favor of the new function.
- Added `unzip_logical_region_schema` to extract schema components.
- Updated tests in `pending_rows_batcher.rs`:
- Added tests for `accommodate_record_batch_for_target_schema` to verify handling of missing tag columns and renaming of special columns.
- Ensured error handling for missing timestamp and field columns in target schema.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Summary
- **Enhancement in Table Creation Logic**: Updated `prom_store.rs` to modify the handling of `table_options` during table creation. Specifically, `table_options` are now extended differently based on the `AutoCreateTableType`. For `Physical` tables, enforced
`sst_format=flat` to optimize pending-rows writes by leveraging bulk memtables.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Enhance Performance Monitoring in `pending_rows_batcher.rs`
- Added performance monitoring timers to various stages of the `PendingRowsBatcher` process, including schema cache checks, table resolution, schema creation, and record batch alignment.
- Improved schema handling by adding timers around schema alteration and missing column addition processes.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Enhance Concurrent Write Handling**: Introduced `FlushRegionWrite` and `FlushWriteResult` structs to manage region writes and their results. Added `flush_region_writes_concurrently` function to handle concurrent flushing of region writes based on
`should_dispatch_concurrently` logic in `pending_rows_batcher.rs`.
- **Testing Enhancements**: Added tests for concurrent dispatching of region writes and the logic for determining concurrent dispatch in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add Histogram for Flush Stage Elapsed Time
- **`metrics.rs`**: Introduced a new `HistogramVec` named `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to track the elapsed time of pending rows batch flush stages.
- **`pending_rows_batcher.rs`**: Replaced instances of `PENDING_ROWS_BATCH_INGEST_STAGE_ELAPSED` with `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` to measure the elapsed time for various flush stages, including `flush_write_region`, `flush_concat_table_batches`,
`flush_resolve_table`, `flush_fetch_partition_rule`, `flush_split_record_batch`, `flush_filter_record_batch`, `flush_resolve_region_leader`, and `flush_encode_ipc`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Add design doc for physical table batching in PendingRowsBatcher
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Add implementation plan for physical table batching in PendingRowsBatcher
* feat/auto-schema-align:
### Commit Message
**Enhance Metric Engine with Physical Batch Processing**
- **Add `metric-engine` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `metric-engine` as a workspace dependency.
- **Expose Batch Modifier Functions**: Changed visibility of `TagColumnInfo`, `compute_tsid_array`, and `modify_batch_sparse` in `batch_modifier.rs` to public, and made `batch_modifier` a public module in `lib.rs`.
- **Implement Physical Batch Processing**:
- Added functions `bulk_insert_physical_region` and `bulk_insert_logical_region` in `bulk_insert.rs` to handle physical and logical batch insertions.
- Updated `pending_rows_batcher.rs` to attempt physical batch processing before falling back to logical processing, including new functions `flush_batch_physical` and `flush_batch_per_logical_table`.
- **Enhance Testing**:
- Added tests for physical region passthrough and empty batch handling in `bulk_insert.rs`.
- Introduced `with_mito_config` in `test_util.rs` for customized test environments.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Batch Processing for Table Creation and Alteration
- **`prom_store.rs`**:
- Added `create_tables_if_missing_batch` and
`add_missing_prom_tag_columns_batch` methods to handle batch creation of tables
and batch alteration to add missing tag columns.
- Implemented logic to determine missing tables and columns, and perform batch
operations accordingly.
- **`pending_rows_batcher.rs`**:
- Updated `PendingRowsBatcher` to utilize batch methods for creating tables an
adding missing columns.
- Enhanced logic to resolve table schemas and accommodate record batches after
batch operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf: concurrent catalog lookups and eliminate redundant concat_batches on ingest path
Replace sequential catalog_manager.table() calls with concurrent
futures::future::join_all in align_table_batches_to_region_schema.
This affects all three lookup loops: initial table resolution,
post-create resolution, and post-alter schema refresh. Reduces
O(N) sequential RPC latency to O(1) wall-clock time for requests
with many distinct logical tables (e.g. Prometheus remote_write).
Remove the per-logical-table concat_batches in flush_batch_physical.
Instead of merging all chunks of a table into one RecordBatch before
calling modify_batch_sparse, apply modify_batch_sparse directly to
each chunk and collect all modified chunks for a single final concat.
This eliminates one full data copy per logical table on the flush path.
* refactor: extract Prometheus schema alignment helpers into prom_row_builder module
Move six functions and their eight unit tests from pending_rows_batcher.rs
(~2386 lines) into a new prom_row_builder.rs module (~776 lines), leaving
the batcher at ~1665 lines focused on flush/worker machinery.
Extracted functions:
- accommodate_record_batch_for_target_schema (normalize incoming batch
against existing table schema)
- unzip_logical_region_schema (extract ts/field/tag columns)
- build_prom_create_table_schema (build ColumnSchema vec for table creation)
- align_record_batch_to_schema (reorder/fill/cast columns to target schema)
- rows_to_record_batch (convert proto Rows to Arrow RecordBatch)
- build_arrow_array (build Arrow arrays from proto values)
Cleaned up 12 now-unused imports from pending_rows_batcher.rs.
* feat/auto-schema-align:
### Enhance `PendingRowsBatcher` and `prom_row_builder` for Efficient Schema Handling
- **`pending_rows_batcher.rs`:**
- Refactored `submit` method to integrate table batch building and alignment into a single method `build_and_align_table_batches`.
- Removed intermediate `RecordBatch` creation, optimizing the process by directly converting proto `RowInsertRequests` into aligned `RecordBatch`es.
- Enhanced schema handling by identifying missing columns directly from proto schemas.
- **`prom_row_builder.rs`:**
- Introduced `rows_to_aligned_record_batch` for direct conversion of proto `Rows` into aligned `RecordBatch`es.
- Added `identify_missing_columns_from_proto` to detect absent tag columns without intermediate `RecordBatch`.
- Implemented `build_prom_create_table_schema_from_proto` to construct table schemas directly from proto schemas.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Add elapsed time metrics for bulk insert operations
- Updated `bulk_insert` method in `bulk_insert.rs` to record elapsed time metrics using `MITO_OPERATION_ELAPSED` for both physical and logical regions.
- Added a new test `test_bulk_insert_records_elapsed_metric` to verify that the elapsed time metric is recorded correctly during bulk insert operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* remove flush per logical region
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `flush_batch` and `flush_batch_physical` functions**
- Removed unused `catalog` and `schema` variables from `flush_batch` in `pending_rows_batcher.rs`.
- Updated `flush_batch_physical` to directly use `ctx.current_catalog()` and `ctx.current_schema()` for resolving table names.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Remove Unused Function and Associated Test
- **File:** `src/servers/src/prom_row_builder.rs`
- Removed the unused function `build_prom_create_table_schema` which was responsible for building a `Vec<ColumnSchema>` from an Arrow schema.
- Deleted the associated test `test_build_prom_create_table_schema_from_request_schema` that validated the removed function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Remove Test**: Deleted the `test_bulk_insert_records_elapsed_metric` test from `bulk_insert.rs`.
- **Refactor Table Resolution**: Introduced `TableResolutionPlan` struct and refactored table resolution logic in `pending_rows_batcher.rs`.
- **Enhance Table Handling**: Added functions for collecting non-empty table rows, unique table schemas, and handling table creation and alteration in `pending_rows_batcher.rs`.
- **Add Tests**: Implemented tests for `collect_non_empty_table_rows` and `collect_unique_table_schemas` in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor Error Handling**: Updated error handling in `pending_rows_batcher.rs` and `prom_row_builder.rs` to use `Snafu` error context for more descriptive error messages.
- **Remove Unused Functionality**: Eliminated the `rows_to_record_batch` function and related test in `prom_row_builder.rs` as it was redundant.
- **Simplify Function Return Types**: Modified `rows_to_aligned_record_batch` in `prom_row_builder.rs` to return only `RecordBatch` without missing columns, simplifying the function's interface and related tests.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Add Helper Function for Table Options in `prom_store.rs`
- Introduced `fill_metric_physical_table_options` function to encapsulate logic for setting table options, ensuring the use of flat SST format and physical table metadata.
- Updated `Instance` implementation to utilize the new helper function for setting table options.
- Added a unit test `test_metric_physical_table_options_forces_flat_sst_format` to verify the correct application of table options.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor `PendingRowsBatcher`**: Simplified worker retrieval logic in `get_or_spawn_worker` method by using a more concise conditional check.
- **Metrics Update**: Added `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED` metric in `pending_rows_batcher.rs`.
- **Remove Unused Code**: Deleted multiple test functions related to record batch alignment and schema preparation in `pending_rows_batcher.rs` and `prom_row_builder.rs`.
- **Function Visibility Change**: Made `build_prom_create_table_schema_from_proto` public in `prom_row_builder.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: remove plan
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Refactor and Simplify Schema Alteration Logic
- **Removed Unused Methods**: Deleted `create_table_if_missing` and `add_missing_prom_tag_columns` methods from `PendingRowsSchemaAlterer` trait in `prom_store.rs` and `pending_rows_batcher.rs`.
- **Error Handling Improvement**: Enhanced error handling in `create_tables_if_missing_batch` method to return a specific error message for unsupported `AutoCreateTableType` in `prom_store.rs`.
- **Visibility Change**: Made `as_str` method public in `AutoCreateTableType` enum in `insert.rs` to support external access.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Message
Improve safety in `prom_row_builder.rs`
- Updated `unzip_logical_region_schema` to use `saturating_sub` for safer capacity calculation of `tag_columns`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
Add TODO comments for future improvements in `pending_rows_batcher.rs`
- Added a TODO comment to consider bounding the `flush_region_writes_concurrently` function.
- Added a TODO comment to potentially limit the maximum rows to concatenate in the `flush_batch_physical` function.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Message
Enhance error handling in `pending_rows_batcher.rs`
- Updated `collect_unique_table_schemas` to return a `Result` type, enabling error handling for duplicate table names.
- Modified the function to return an error when duplicate table names are found in `table_rows`.
- Adjusted test cases to handle the new `Result` return type in `collect_unique_table_schemas`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Refactor `partition_columns` Method**: Updated the `partition_columns` method in `multi_dim.rs`, `partition.rs`, and `splitter.rs` to return a slice reference instead of a cloned vector, improving performance by avoiding unnecessary cloning.
- **Enhance Partition Handling**: Added functions `collect_tag_columns_and_non_tag_indices` and `strip_partition_columns_from_batch` in `pending_rows_batcher.rs` to manage partition columns more efficiently, including stripping partition columns from record batches.
- **Update Tests**: Modified existing tests and added new ones in `pending_rows_batcher.rs` to verify the functionality of partition column handling, ensuring correct behavior of the new methods.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Schema Handling and Validation in `pending_rows_batcher.rs`
- **Schema Validation Enhancements**:
- Added checks for essential columns (`timestamp`, `value`) in `collect_tag_columns_and_non_tag_indices`.
- Introduced `PHYSICAL_REGION_ESSENTIAL_COLUMN_COUNT` to ensure minimum column count in `strip_partition_columns_from_batch`.
- Improved error handling for unexpected data types and duplicated columns.
- **Function Modifications**:
- Updated `strip_partition_columns_from_batch` to project essential columns without lookup.
- Modified `flush_batch_physical` to use `essential_col_indices` instead of `non_tag_indices`.
- **Test Enhancements**:
- Added tests for schema validation, including checks for unexpected data types and duplicated columns.
- Verified correct projection of essential columns in `strip_partition_columns_from_batch`.
Files affected: `pending_rows_batcher.rs`, `tests`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
- **Add `smallvec` Dependency**: Updated `Cargo.lock` and `Cargo.toml` to include `smallvec` as a workspace dependency.
- **Refactor Function**: Renamed `collect_tag_columns_and_non_tag_indices` to `columns_taxonomy` in `pending_rows_batcher.rs` and updated its return type to use `SmallVec`.
- **Update Tests**: Modified test cases in `pending_rows_batcher.rs` to reflect changes in function name and return type.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `pending_rows_batcher.rs` to Simplify Table ID Handling**
- Updated `TableBatch` struct to use `TableId` directly instead of `Option<u32>` for `table_id`.
- Simplified logic in `flush_batch_physical` by removing the check for `None` in `table_id`.
- Adjusted related logic in `start_worker` to accommodate the change in `table_id` handling.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Enhance Batch Processing Logic
- **`pending_rows_batcher.rs`**:
- Moved column taxonomy resolution inside the loop to handle schema variations across batches.
- Added checks to skip processing if both tag columns and essential column indices are empty.
- **Tests**:
- Added `test_modify_batch_sparse_with_taxonomy_per_batch` to verify batch modification logic with varying schemas.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Remove Primary Key Column Check in `pending_rows_batcher.rs`
- Removed the check for the primary key column and other essential column names in the function `strip_partition_columns_from_batch` within `pending_rows_batcher.rs`.
- Simplified the logic by eliminating the validation of column order against expected essential names.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Refactor error handling and iteration in `otlp.rs` and `pending_rows_batcher.rs`
- **`otlp.rs`**: Simplified error handling by removing `CatalogSnafu` context when awaiting table retrieval.
- **`pending_rows_batcher.rs`**: Streamlined iteration over tables by removing unnecessary `into_iter()` calls, improving code readability and efficiency.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore/metrics-for-bulk:
Add timing metrics for batch processing in `pending_rows_batcher.rs`
- Introduced `modify_elapsed` and `columns_taxonomy_elapsed` to measure time spent in `modify_batch_sparse` and `columns_taxonomy` functions.
- Updated `flush_batch_physical` to record these metrics using `PENDING_ROWS_BATCH_FLUSH_STAGE_ELAPSED`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Commit Summary
- **Remove Unused Code**: Eliminated the `#[allow(dead_code)]` attribute from the `compute_tsid_array` function in `batch_modifier.rs`.
- **Error Handling Improvement**: Enhanced error handling in `flush_batch_physical` function by adjusting the `match` block in `pending_rows_batcher.rs`.
- **Simplify Logic**: Streamlined the logic in `rows_to_aligned_record_batch` by removing unnecessary type casting in `prom_row_builder.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
**Refactor `flush_batch_physical` in `pending_rows_batcher.rs`:**
- Moved partition column stripping logic to a single location before processing region batches.
- Updated the use of `combined_batch` to `stripped_batch` for consistency in batch processing.
- Removed redundant partition column stripping logic within the region batch loop.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/auto-schema-align:
### Update `batch_modifier.rs` Documentation and Parameter Naming
- Enhanced documentation for `compute_tsid_array` and `modify_batch_sparse` functions to clarify their logic and parameters.
- Renamed parameter `non_tag_column_indices` to `extra_column_indices` in `modify_batch_sparse` for better clarity.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: metric batch 2s PoC
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: max_concurrent_flushes
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: work channel size
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(servers): add metrics and logs for pending rows batch flush
Add the `FLUSH_ELAPSED` histogram metric to track the duration of pending
rows batch flushes in the Prometheus store protocol handler. This provides
better observability into the performance and latency of the batcher.
Also update telemetry by:
- Recording elapsed time for both successful and failed flush operations.
- Adding an informational log upon successful flush including row count and duration.
- Including elapsed time in error logs when a flush fails.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): implement columnar batching for pending rows
Refactor PendingRowsBatcher to use columnar batching for the metrics
store. Incoming RowInsertRequests are now converted to RecordBatches,
partitioned, and flushed via BulkInsert requests to datanodes.
- Enhance MultiDimPartitionRule to handle scalar boolean predicates.
- Add metrics for tracking flush failures and dropped rows.
- Update dependencies to support columnar batching in servers.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): add backpressure for pending rows
Implement backpressure in PendingRowsBatcher by limiting in-flight
requests with a semaphore and making the submission wait for the flush
result. This ensures Prometheus write requests are throttled and only
return once the data has been successfully flushed to datanodes.
- Add max_inflight_requests to PromStoreOptions.
- Use oneshot channels to notify submitters of flush completion.
- Limit concurrent requests using a new inflight_semaphore.
- Update PendingRowsBatcher::submit to wait for the flush outcome.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add stage-level metrics for bulk ingestion
Introduce histograms to track the elapsed time of various stages in the
metric engine bulk insert path and the server's pending rows batcher.
This provides better observability into the performance bottlenecks
of the ingestion pipeline.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* - `src/metric-engine/src/engine/bulk_insert.rs`: Removed the fallback mechanism that converted record batches to rows when bulk inserts were unsupported, along with related helper functions and unused imports.
- `src/operator/src/insert.rs`: Removed an unused import (`common_time::TimeToLive::Instant`).
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): columnar Prom remote write
Optimize the Prometheus remote write path by allowing direct conversion
from decoded Prometheus samples to Arrow RecordBatches. This bypasses
intermediate row-based representations when `PendingRowsBatcher` is
active and no pipeline is used, improving ingestion efficiency.
- Implement `as_record_batch_groups` in `TablesBuilder` and `PromWriteRequest`.
- Add `submit_prom_record_batch_groups` to `PendingRowsBatcher`.
- Introduce `DecodedPromWriteRequest` in `prom_store`.
- Implement row-to-RecordBatch conversion logic in `prom_row_builder`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Revert "feat(servers): columnar Prom remote write"
This reverts commit efbb63c12a3e7fcec03858ea0351efd94fec8242.
* refactor(servers): improve row to RecordBatch conversion
- Use `snafu::ensure` for row validation in `rows_to_record_batch`.
- Add explicit type hint for `MutableVector` to improve clarity.
- Reorganize and clean up imports in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(servers): use arrow builders for row conversion
This commit optimizes the conversion from `api::v1::Rows` to `RecordBatch`
by using Arrow builders directly. This avoids the overhead of
`MutableVector` and `common_recordbatch`, leading to better performance
in the `pending_rows_batcher`.
Additionally, the `#[allow(dead_code)]` attribute is removed from
`modify_batch_sparse` in the metric engine as it is now utilized.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(metric-engine): optimize batch modification
Optimize `modify_batch_sparse` by reusing buffers, using Arrow
builders, and employing fast-path encoding methods. This reduces
allocations and avoids redundant downcasting and serializer overhead.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-support-bulk:
**Add Environment Variable for Batch Sync Control**
- `pending_rows_batcher.rs`: Introduced an environment variable `PENDING_ROWS_BATCH_SYNC` to control the synchronization behavior of batch processing. If set to true, the function will wait for the flush result; otherwise, it will return immediatel
with the total rows count.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* wip
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: update and fix clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: failing test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Commit Message
Remove Unused Code and Simplify Error Handling
- **`src/error.rs`**: Removed the `BatcherQueueFull` error variant and its associated logic, simplifying the error handling by removing unused code.
- **`src/http/prom_store.rs`**: Eliminated the `try_decompress` function, streamlining the decompression logic by directly using `snappy_decompress` in `decode_remote_read_request`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: parse PENDING_ROWS_BATCH_SYNC once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: revert unrelated changes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* **Refactor Prometheus Write Handling**
- **`prom_store.rs`**: Introduced `pre_write` method in `PromStoreProtocolHandler` to handle pre-write checks for Prometheus remote write requests. Updated `write` method to utilize `pre_write`.
- **`server.rs`**: Modified `PendingRowsBatcher` initialization to conditionally create a batcher based on `with_metric_engine` flag.
- **`http/prom_store.rs`**: Integrated `pre_write` checks before submitting requests to `PendingRowsBatcher`.
- **`query_handler.rs`**: Added `pre_write` method to `PromStoreProtocolHandler` trait for pre-write operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
- **Fix Label Typo**: Corrected a typo in the label value from `"flush_wn ite_region"` to `"flush_write_region"` in `pending_rows_batcher.rs`.
- **Refactor Array Building Logic**: Introduced a macro `build_array!` to streamline the construction of `ArrayRef` for different data types, reducing code duplication in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* format toml
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Update PromStore and PendingRowsBatcher Configuration
- **`prom_store.rs`**: Set `pending_rows_flush_interval` to `Duration::ZERO` to disable automatic flushing.
- **`pending_rows_batcher.rs`**: Enhance validation to disable the batcher when `flush_interval` is zero or configuration values like `max_batch_rows`, `max_concurrent_flushes`, `worker_channel_capacity`, or `max_inflight_requests` are zero, preventing potential panics or deadlocks.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Update `pending_rows_flush_interval` to Zero
- **Files Modified**:
- `src/frontend/src/service_config/prom_store.rs`
- `tests-integration/tests/http.rs`
- **Key Changes**:
- Updated `pending_rows_flush_interval` from `Duration::from_secs(2)` to `Duration::ZERO` in `prom_store.rs`.
- Changed `pending_rows_flush_interval` configuration from `"2s"` to `"0s"` in `http.rs`.
These changes set the flush interval to zero, potentially affecting how frequently pending rows are flushed.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
**Add Worker Management Enhancements**
- **`metrics.rs`**: Introduced `PENDING_WORKERS` gauge to track active pending rows batch workers.
- **`pending_rows_batcher.rs`**:
- Added worker idle timeout logic with `WORKER_IDLE_TIMEOUT_MULTIPLIER`.
- Implemented worker management functions: `spawn_worker`, `remove_worker_if_same_channel`, and `should_close_worker_on_idle_timeout`.
- Enhanced worker lifecycle management to handle idle workers and ensure proper cleanup.
- **Tests**: Added unit tests for worker removal and idle timeout logic.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: jeremyhi <fengjiachun@gmail.com>
chore: remove GrpcQueryHandler::put_record_batch, we should use GrpcQueryHandler::handle_put_record_batch_stream instead
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: initial function rewriter for json_get
* feat: make sure rewrite rule is applied
* feat: keep analyzer's default rules
* feat: implement rewriter for arrow_cast
* test: add unit test for tht rewriter
* chore: format
* refactor: extract some more functions
* Apply suggestion from @waynexia
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
---------
Co-authored-by: Ruihang Xia <waynestxia@gmail.com>
