* feat: accept x-greptime-pipeline-name header on /events/logs
The /events/logs (and /logs/ingest) endpoint previously only read the
pipeline name from the `pipeline_name` query parameter, while the
OTLP/Elasticsearch/Splunk log ingestion endpoints already accept it via
the `x-greptime-pipeline-name` header. This inconsistency is unfriendly
to users.
Make `log_ingester` resolve the pipeline name from the
`x-greptime-pipeline-name` header (and the deprecated
`x-greptime-log-pipeline-name`), falling back to the query parameter.
The header takes precedence, consistent with how other pipeline options
(e.g. `x-greptime-pipeline-params`) outrank their query-parameter
counterparts.
Closes#6095
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
* address review: prefer non-deprecated pipeline-name header
When both pipeline-name headers are present, resolve the non-deprecated
`x-greptime-pipeline-name` before the deprecated
`x-greptime-log-pipeline-name`, and cover the precedence with a test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
---------
Signed-off-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: BootstrapperSBL <yvanwww01@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat: expose region read load through Prometheus metrics and heartbeat
Introduce region-level query load tracking (CPU time and scanned bytes)
collected by `RegionScanExec`, exposed via Prometheus metrics and optionally
reported through heartbeat region stats.
- **Region metrics** (`src/mito2/src/metrics.rs`, `src/store-api/src/metrics.rs`): Add
`greptime_mito_region_query_cpu_time`, `greptime_mito_region_query_scanned_bytes`,
and `greptime_mito_region_written_bytes_since_open` gauge metrics.
- **MitoRegion** (`src/mito2/src/region.rs`, `src/mito2/src/region/opener.rs`,
`src/mito2/src/region_write_ctx.rs`): Replace `AtomicU64` `written_bytes` with
`IntGauge`; add `query_cpu_time`/`query_scanned_bytes` fields with lifecycle
management (init, reset, remove-on-drop).
- **RegionStatistic** (`src/store-api/src/region_engine.rs`,
`src/store-api/src/storage/requests.rs`): Add `query_cpu_time` and
`query_scanned_bytes` fields.
- **Metric-engine** (`src/metric-engine/src/utils.rs`): Aggregate query load from
metadata and data regions.
- **Heartbeat** (`src/datanode/src/heartbeat.rs`,
`src/common/meta/src/datanode.rs`): Relay region query load via heartbeat
`RegionStat`; add test.
- **Query engine** (`src/query/src/options.rs`,
`src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`,
`src/query/src/dist_plan/merge_scan.rs`,
`src/query/src/dist_plan/analyzer.rs`,
`src/query/src/dummy_catalog.rs`): Add `enable_region_query_load_report` config;
wire `RegionScanExec` to accumulate CPU time and scanned bytes.
- **Table scan** (`src/table/src/table/scan.rs`,
`src/table/src/table/metrics.rs`): Wire table scan metrics.
- **Config** (`config/standalone.example.toml`, `config/datanode.example.toml`,
`config/frontend.example.toml`, `config/config.md`): Add example config and
documentation for `enable_region_query_load_report`.
- **Tests** (`src/mito2/src/engine/basic_test.rs`,
`src/mito2/src/engine/close_test.rs`,
`src/cmd/tests/load_config_test.rs`,
`src/flow/src/adapter.rs`): Add unit tests for region query load reporting
and metric cleanup on region close; set default config values.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move region read load report config from query layer to mito engine
Move the `enable_region_query_load_report` setting from query-level config
(`QueryOptions`/`DistPlannerOptions`) into the mito2 storage engine config
(`MitoConfig`), and expose it through the `RegionScanner` trait instead
of `ScanRequest`/`PrepareRequest`.
- Mito config: `src/mito2/src/config.rs`, `src/mito2/src/engine.rs`
- Scan region plumbing: `src/mito2/src/read/scan_region.rs`
- RegionScanner trait: `src/store-api/src/region_engine.rs`
- Scanner impls: `src/mito2/src/read/seq_scan.rs`, `src/mito2/src/read/series_scan.rs`, `src/mito2/src/read/unordered_scan.rs`
- RegionScanExec: `src/table/src/table/scan.rs`
- Removed from query layer: `src/query/src/options.rs`, `src/query/src/dist_plan/analyzer.rs`, `src/query/src/query_engine/state.rs`, `src/query/src/datafusion.rs`, `src/query/src/dummy_catalog.rs`
- Removed from test/config: `src/query/src/dist_plan/analyzer/test.rs`, `src/flow/src/adapter.rs`, `src/cmd/tests/load_config_test.rs`, `src/store-api/src/storage/requests.rs`
- Config docs: `config/config.md`, `config/datanode.example.toml`, `config/frontend.example.toml`, `config/standalone.example.toml`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move region query load report config from MitoConfig to LoggingOptions
Relocate the `enable_region_query_load_report` setting from
`MitoConfig` to `LoggingOptions` (as `enable_per_region_metrics`),
and thread it into `MitoEngineBuilder` instead of reading from
the engine config directly. This makes the region read-load
reporting a per-node logging/observability concern rather than
a per-engine storage setting.
- `config/config.md`
- `config/datanode.example.toml`
- `config/standalone.example.toml`
- `src/common/telemetry/src/logging.rs`
- `src/datanode/src/datanode.rs`
- `src/mito2/src/config.rs`
- `src/mito2/src/engine.rs`
- `src/mito2/src/region.rs`
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: report region query load on stream drop instead of stream end
Move `report_region_query_load()` from `StreamWithMetricWrapper::poll_next()`
to `Drop::drop()` so that region query load is reported even when the
stream is dropped prematurely (not just when fully consumed).
Affected files:
- `src/table/src/table/scan.rs`
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: make region query load reporting configurable
Introduce `enable_region_query_load_report` flag to optionally report
per-region `query_cpu_time` and `query_scanned_bytes` metrics instead
of always creating them. When disabled, the Prometheus gauges are not
created (`None`), avoiding metric churn for workloads that do not
need query-level load tracking.
- `src/common/meta/src/datanode.rs` — Placeholder fields for query load
- `src/mito2/src/region.rs` — Make query metrics `Option<IntGauge>`, conditional create/remove/reset
- `src/mito2/src/region/opener.rs` — Thread flag through `RegionOpener`
- `src/mito2/src/worker.rs` — Thread flag through `WorkerGroup`/`WorkerStarter`/`RegionWorkerLoop`
- `src/mito2/src/worker/handle_catchup.rs` — Pass flag on region open
- `src/mito2/src/worker/handle_create.rs` — Pass flag on region create
- `src/mito2/src/worker/handle_open.rs` — Pass flag on region open
- `src/mito2/src/engine.rs` — Pass flag from `MitoEngineBuilder`
- `src/mito2/src/test_util.rs` — Test helpers for both modes
- `src/mito2/src/engine/basic_test.rs` — Cover disabled and preserve cases
- `src/mito2/src/engine/close_test.rs` — Adapt to optional metrics
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove elapsed_compute metric from scan stream
The elapsed_compute metric conflated poll-wait time with actual CPU
computation, making it misleading. Removed the metric and its
recording path from StreamMetrics and StreamWithMetricWrapper.
Added a test asserting that poll duration is not reported as
elapsed_compute.
- `src/table/src/table/metrics.rs` — removed elapsed_compute field,
builder, and record_elapsed_compute method
- `src/table/src/table/scan.rs` — removed record_elapsed_compute
call; added SlowRecordBatchStream test helper and
wrapper_poll_time_is_not_elapsed_compute test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: disable region query load report for compaction scans
Compaction scans are internal operations initiated by the engine,
not user queries. Disable region query load reporting when the
scan input is marked as compaction to avoid misleading load metrics.
- `src/mito2/src/read/scan_region.rs` — set `enable_region_query_load_report`
to `false` when compaction is enabled; add unit test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* test: add `enable_per_region_metrics` config to HTTP integration test
- Enable per-region metrics config in HTTP test setup
\`tests-integration/tests/http.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove region query load reporting tests and helpers
Remove the region query load reporting feature from the codebase,
including tests, test utilities, and helper infrastructure that were
part of this now-deprecated functionality.
Specifically:
- Remove region query load reporting tests from
`src/mito2/src/engine/basic_test.rs` and
`src/table/src/table/scan.rs`, and the region close metrics test
from `src/mito2/src/engine/close_test.rs`
- Remove region query load report test utilities and simplify engine
construction helpers in `src/mito2/src/test_util.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf: avoid disabled region query load timing
Summary:
- Avoid per-poll `Instant::now` and elapsed-time accumulation when `enable_region_query_load_report` is disabled.
- Keep region query-load CPU accounting active only when reporting is enabled.
Files:
- `src/table/src/table/scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: move per-region query load reporting from storage to query engine
Move `enable_per_region_metrics` from datanode to frontend config and
migrate query load tracking (CPU time, scanned bytes) from mito2
storage engine to the query engine's distributed scan planner. The
storage-level metrics plumbing and `enable_region_query_load_report`
flag are removed from mito2, `ScanInput`, `ScanRegion`, and
`RegionScanner`. Query-level metrics are now collected in
`merge_scan.rs` via `scan_region_load`.
- `src/mito2/` -- Remove `query_cpu_time`, `query_scanned_bytes`
metrics, `enable_region_query_load_report` plumbing from engine,
region, opener, scanner types, workers
- `src/store-api/` -- Remove `query_cpu_time`, `query_scanned_bytes`
from `RegionStatistic`
- `src/metric-engine/` -- Remove query load fields from
`get_region_statistic`
- `src/query/` -- Add `enable_per_region_metrics` to `QueryOptions`;
wire through planner, optimizer, merge scan with `scan_region_load`
metrics
- `src/frontend/` -- Pass `enable_per_region_metrics` into
`QueryOptions`
- `src/common/meta/` -- Remove TODO for query load fields
- `config/` -- Move `enable_per_region_metrics` from datanode to
frontend and standalone example configs
- `src/cmd/tests/` -- Add `enable_per_region_metrics` to flownode
config test
- `src/flow/` -- Add `enable_per_region_metrics` default to flownode
options
- `src/table/` -- Remove unused query load fields from scan
- `src/datanode/` -- Remove
`with_enable_region_query_load_report` calls
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove obsolete mito write load metric
Remove obsolete mito-side region written-bytes metric plumbing that is not needed by the frontend read-load reporting path.
Related files:
- \`src/mito2/src/metrics.rs\`
- \`src/mito2/src/region.rs\`
- \`src/mito2/src/region/opener.rs\`
- \`src/mito2/src/region_write_ctx.rs\`
- \`src/mito2/src/engine/basic_test.rs\`
- \`src/mito2/src/worker.rs\`
- \`src/mito2/src/config.rs\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: change region query load metrics from gauge to counter
Change `REGION_QUERY_CPU_TIME` and `REGION_QUERY_SCANNED_BYTES` from
`IntGaugeVec` to `IntCounterVec` since these values are monotonically
increasing and do not need gauge semantics. Update corresponding `add`
calls to `inc_by` in merge scan reporting.
Files:
- `src/store-api/src/metrics.rs` — metric type and label changes
- `src/query/src/dist_plan/merge_scan.rs` — caller adaptation
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: pass ReadItem directly to report_region_query_load
Move `region_scan_load` call to the caller, so `report_region_query_load`
accepts the already-computed `ReadItem` instead of `RecordBatchMetrics`.
- `src/query/src/dist_plan/merge_scan.rs` — update signature, inline call,
remove stale test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: ensure region query load is reported on MergeScanExec drop
Remove the `enable_per_region_metrics` parameter from `report_region_query_load`
so region load metrics are always emitted. Add a `Drop` impl for
`MergeScanExec` that reports sub-stage metrics when the executor is
dropped, covering edge cases where per-region metric emission was
missed. Add a unit test verifying CPU time and scanned bytes are
recorded on drop.
Affected file: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: gate region query load reporting
Guard drop-time region query load reporting with the configured per-region metrics flag.
Related files:
- \`src/query/src/dist_plan/merge_scan.rs\`
Symbols:
- \`MergeScanExec::drop\`
- \`enable_per_region_metrics\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clean region query load metrics on drop
Remove per-region query load metric labels when a region is dropped so stale label series do not remain in the registry.
Related files:
- \`src/mito2/src/region.rs\`
Symbols:
- \`MitoRegion::drop\`
- \`REGION_QUERY_CPU_TIME\`
- \`REGION_QUERY_SCANNED_BYTES\`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Signed-off-by: Lei Huang <lei@huang.to>
Signed-off-by: Lei, Huang <huanglei@qiyi.com>
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer identity (Phase 1)
Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.
Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
(signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
defined now, populated by later phases), value constants, the internal
`greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
(metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
the v1 model), logs (log/opentelemetry), and Prometheus remote write
(metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
`_services` / `_operations` lookup tables keep the unstamped context and carry no
semantic identity (cross-table relationships are out of scope).
Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.
Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: prom batcher not cover and white list for semantic keys/values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: typo
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: global switch for creating table automatically
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: make auto_create_table as comment by default
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: respect gloabl switch for metric engine
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: switch partition tree to bulk
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: keep partition tree memtable for migration test
Restore PartitionTreeMemtable construction when memtable.type=partition_tree
is explicit, and move the sparse-encoding bulk override into the default
(no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on
reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a
metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse,
so the test actually exercises a PartitionTreeMemtable in phase 1 and
verifies WAL replay into the new BulkMemtable on reopen without flushing.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: drop partition tree memtable from runtime
Re-apply the unconditional sparse-encoding override in
`MemtableBuilderProvider::builder_for_options` and route the
`MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation
warning. After this change, `PartitionTreeMemtableBuilder` is no longer
reachable from the engine runtime; benchmarks still reference the type.
Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the
`put_sparse_rows` helper added in the previous commit — that test only
existed to validate the PartitionTree -> Bulk reopen migration and is
unnecessary now that the override is in place.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move timestamp_array_to_i64_slice into read module
Relocate the timestamp_array_to_i64_slice helper from
memtable/partition_tree/data.rs to the read module so that the read
path no longer depends on the partition_tree internals. All call sites
(both inside and outside the partition_tree module) now import from
crate::read.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests
The time_partition tests use the memtable builder purely as a generic
backend for the TimePartitions write/scan paths; nothing in them is
specific to the partition-tree memtable. Switch the seven affected
tests to TimeSeriesMemtableBuilder so the tests no longer depend on
PartitionTreeMemtableBuilder.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): delete PartitionTreeMemtable implementation
The runtime already falls back to BulkMemtable for the PartitionTree
variant. Drop the now-unreachable implementation, its metrics, the
partition_tree benchmarks, the metric-engine Unsupported fallback in
bulk_insert.rs, and the test helpers that only existed for the deleted
module.
MemtableOptions::PartitionTree, its parsing, the runtime fallback, the
store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures
remain so existing region options keep round-tripping.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito-codec): drop skip_partition_column parameter
PartitionTreeMemtable was the only caller passing
skip_partition_column=true; every other caller passes false. Now that
the partition_tree module is gone, the parameter is uniformly false
and the guard branch is dead. Drop the parameter from the trait method
and both impls, remove the guard and the is_partition_column helper,
and update the four remaining call sites in mito2 plus the bench.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): remove unused MemtableConfig enum
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fmt code
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: remove unused variant
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update test_config_api
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: remove unused memtable test helpers
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: address review comment
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: support bulk memtable options
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: sanitize config
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: remove partition tree options from region options
Move primary_key_encoding to the top level
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: make ssts test datetime replaced text stable
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update sqlness result
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: validate_enum_options consider bulk memtable
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: pass region id when parsing region options
Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with
`try_from_options(region_id, options_map)` so the legacy partition_tree
fallback can log the affected region. The fallback now also overrides
the SST format to flat in addition to clearing the memtable type.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: align sst_format with bulk memtable on parse and open
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: support reporting env vars in heartbeat messages to metasrv
Add `heartbeat_env_vars` config option for datanode and frontend. When
configured, the specified environment variable values are read at startup
and sent to metasrv in every heartbeat via the `extensions` map. Metasrv
extracts and stores them in `NodeInfo` for use in routing decisions
(e.g. AZ-aware region placement).
- Add `EnvVars` helper in `common/meta/src/datanode.rs` following the
existing `GcStat` extension pattern with `into_extensions`/`from_extensions`
- Add `env_vars: HashMap<String, String>` field to `NodeInfo` in
`common/meta/src/cluster.rs` with `#[serde(default)]` for backward compat
- Add `heartbeat_env_vars: Vec<String>` config field to `DatanodeOptions`,
`FrontendOptions`, and `StandaloneOptions`
- Inject env vars into heartbeat `extensions` in both datanode and frontend
heartbeat tasks (`datanode/src/heartbeat.rs`, `frontend/src/heartbeat.rs`)
- Extract env vars from `req.extensions` in all three metasrv
`CollectXxxClusterInfoHandler`s
- Update `NodeInfo` construction sites in `meta-client`,
`discovery/lease.rs`, and `standalone/information_extension.rs`
- Update expected TOML output in `tests-integration/tests/http.rs`
- Add unit tests for `EnvVars` round-trip and `NodeInfo` backward compat
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: address heartbeat env review feedback
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: log error on deserialization failure
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: send heartbeat env vars once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: resend heartbeat env vars after reconnect
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* revert: keep env vars in every heartbeat
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <leih@nvidia.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(operator): allow last_row merge_mode when append_mode is enabled
- Update RegionOptions::validate to allow last_row merge_mode with append_mode.
- Update fill_table_options_for_create to automatically set merge_mode to last_row when append_mode is enabled for LastNonNull table type.
- Add unit tests in mito2 and operator to verify options validation and table creation.
- Add integration test for InfluxDB write with append mode hint.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix(operator): simplify append mode options
Group `LastNonNull` auto-create options in a single append-mode branch.
Files:
- `src/operator/src/insert.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: sqlness
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: support alter from primary_key to flat
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: alter flat to primary_key
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: change default_experimental_flat_format to true
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: compute channel size from splitted batch size
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: add tests for split and channel size
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: always set sst_format from manifest on region open
sanitize_region_options did not set options.sst_format when the
default (PrimaryKey) matched the manifest value, leaving it as None
after reopen. This caused the alter format change to appear lost.
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: fix tests
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: show create table after alteration
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor!: rename default_experimental_flat_format to default_flat_format
The flat format is no longer experimental. Remove "experimental" from
the config field name, doc comments, and all references.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fix clippy
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: metric batch 2s PoC
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: max_concurrent_flushes
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* chore: work channel size
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* feat(servers): add metrics and logs for pending rows batch flush
Add the `FLUSH_ELAPSED` histogram metric to track the duration of pending
rows batch flushes in the Prometheus store protocol handler. This provides
better observability into the performance and latency of the batcher.
Also update telemetry by:
- Recording elapsed time for both successful and failed flush operations.
- Adding an informational log upon successful flush including row count and duration.
- Including elapsed time in error logs when a flush fails.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): implement columnar batching for pending rows
Refactor PendingRowsBatcher to use columnar batching for the metrics
store. Incoming RowInsertRequests are now converted to RecordBatches,
partitioned, and flushed via BulkInsert requests to datanodes.
- Enhance MultiDimPartitionRule to handle scalar boolean predicates.
- Add metrics for tracking flush failures and dropped rows.
- Update dependencies to support columnar batching in servers.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): add backpressure for pending rows
Implement backpressure in PendingRowsBatcher by limiting in-flight
requests with a semaphore and making the submission wait for the flush
result. This ensures Prometheus write requests are throttled and only
return once the data has been successfully flushed to datanodes.
- Add max_inflight_requests to PromStoreOptions.
- Use oneshot channels to notify submitters of flush completion.
- Limit concurrent requests using a new inflight_semaphore.
- Update PendingRowsBatcher::submit to wait for the flush outcome.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: add stage-level metrics for bulk ingestion
Introduce histograms to track the elapsed time of various stages in the
metric engine bulk insert path and the server's pending rows batcher.
This provides better observability into the performance bottlenecks
of the ingestion pipeline.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* - `src/metric-engine/src/engine/bulk_insert.rs`: Removed the fallback mechanism that converted record batches to rows when bulk inserts were unsupported, along with related helper functions and unused imports.
- `src/operator/src/insert.rs`: Removed an unused import (`common_time::TimeToLive::Instant`).
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat(servers): columnar Prom remote write
Optimize the Prometheus remote write path by allowing direct conversion
from decoded Prometheus samples to Arrow RecordBatches. This bypasses
intermediate row-based representations when `PendingRowsBatcher` is
active and no pipeline is used, improving ingestion efficiency.
- Implement `as_record_batch_groups` in `TablesBuilder` and `PromWriteRequest`.
- Add `submit_prom_record_batch_groups` to `PendingRowsBatcher`.
- Introduce `DecodedPromWriteRequest` in `prom_store`.
- Implement row-to-RecordBatch conversion logic in `prom_row_builder`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* Revert "feat(servers): columnar Prom remote write"
This reverts commit efbb63c12a3e7fcec03858ea0351efd94fec8242.
* refactor(servers): improve row to RecordBatch conversion
- Use `snafu::ensure` for row validation in `rows_to_record_batch`.
- Add explicit type hint for `MutableVector` to improve clarity.
- Reorganize and clean up imports in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(servers): use arrow builders for row conversion
This commit optimizes the conversion from `api::v1::Rows` to `RecordBatch`
by using Arrow builders directly. This avoids the overhead of
`MutableVector` and `common_recordbatch`, leading to better performance
in the `pending_rows_batcher`.
Additionally, the `#[allow(dead_code)]` attribute is removed from
`modify_batch_sparse` in the metric engine as it is now utilized.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* perf(metric-engine): optimize batch modification
Optimize `modify_batch_sparse` by reusing buffers, using Arrow
builders, and employing fast-path encoding methods. This reduces
allocations and avoids redundant downcasting and serializer overhead.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/metric-engine-support-bulk:
**Add Environment Variable for Batch Sync Control**
- `pending_rows_batcher.rs`: Introduced an environment variable `PENDING_ROWS_BATCH_SYNC` to control the synchronization behavior of batch processing. If set to true, the function will wait for the flush result; otherwise, it will return immediatel
with the total rows count.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* wip
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: update and fix clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: failing test
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Commit Message
Remove Unused Code and Simplify Error Handling
- **`src/error.rs`**: Removed the `BatcherQueueFull` error variant and its associated logic, simplifying the error handling by removing unused code.
- **`src/http/prom_store.rs`**: Eliminated the `try_decompress` function, streamlining the decompression logic by directly using `snappy_decompress` in `decode_remote_read_request`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: parse PENDING_ROWS_BATCH_SYNC once
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* chore: revert unrelated changes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* **Refactor Prometheus Write Handling**
- **`prom_store.rs`**: Introduced `pre_write` method in `PromStoreProtocolHandler` to handle pre-write checks for Prometheus remote write requests. Updated `write` method to utilize `pre_write`.
- **`server.rs`**: Modified `PendingRowsBatcher` initialization to conditionally create a batcher based on `with_metric_engine` flag.
- **`http/prom_store.rs`**: Integrated `pre_write` checks before submitting requests to `PendingRowsBatcher`.
- **`query_handler.rs`**: Added `pre_write` method to `PromStoreProtocolHandler` trait for pre-write operations.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
- **Fix Label Typo**: Corrected a typo in the label value from `"flush_wn ite_region"` to `"flush_write_region"` in `pending_rows_batcher.rs`.
- **Refactor Array Building Logic**: Introduced a macro `build_array!` to streamline the construction of `ArrayRef` for different data types, reducing code duplication in `pending_rows_batcher.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* format toml
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Update PromStore and PendingRowsBatcher Configuration
- **`prom_store.rs`**: Set `pending_rows_flush_interval` to `Duration::ZERO` to disable automatic flushing.
- **`pending_rows_batcher.rs`**: Enhance validation to disable the batcher when `flush_interval` is zero or configuration values like `max_batch_rows`, `max_concurrent_flushes`, `worker_channel_capacity`, or `max_inflight_requests` are zero, preventing potential panics or deadlocks.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
### Update `pending_rows_flush_interval` to Zero
- **Files Modified**:
- `src/frontend/src/service_config/prom_store.rs`
- `tests-integration/tests/http.rs`
- **Key Changes**:
- Updated `pending_rows_flush_interval` from `Duration::from_secs(2)` to `Duration::ZERO` in `prom_store.rs`.
- Changed `pending_rows_flush_interval` configuration from `"2s"` to `"0s"` in `http.rs`.
These changes set the flush interval to zero, potentially affecting how frequently pending rows are flushed.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* picking-pending-rows-batcher:
**Add Worker Management Enhancements**
- **`metrics.rs`**: Introduced `PENDING_WORKERS` gauge to track active pending rows batch workers.
- **`pending_rows_batcher.rs`**:
- Added worker idle timeout logic with `WORKER_IDLE_TIMEOUT_MULTIPLIER`.
- Implemented worker management functions: `spawn_worker`, `remove_worker_if_same_channel`, and `should_close_worker_on_idle_timeout`.
- Enhanced worker lifecycle management to handle idle workers and ensure proper cleanup.
- **Tests**: Added unit tests for worker removal and idle timeout logic.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: clippy
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
Co-authored-by: jeremyhi <fengjiachun@gmail.com>
* feat: impl vector index scan in storage
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: fallback to read remote blob when blob not found
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: refactor encoding and decoding and apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: license
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add apply_with_k tests
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: forgot to align nulls when the vector column is not in the batch
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: add test for vector column is not in a batch while buiilding
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: impl vector index building
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: supports flat format
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: add vector_index feature to test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: apply suggestions
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: apply suggestions from copilot
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat/allow-one-to-many-pipeline:
### Enhance Pipeline Processing for One-to-Many Transformations
- **Support One-to-Many Transformations**:
- Updated `processor.rs`, `etl.rs`, `vrl_processor.rs`, and `greptime.rs` to handle one-to-many transformations by allowing VRL processors to return arrays, expanding each element into separate rows.
- Introduced `transform_array_elements` and `values_to_rows` functions to facilitate this transformation.
- **Error Handling Enhancements**:
- Added new error types in `error.rs` to handle cases where array elements are not objects and for transformation failures.
- **Testing Enhancements**:
- Added tests in `pipeline.rs` to verify one-to-many transformations, single object processing, and error handling for non-object array elements.
- **Context Management**:
- Modified `ctx_req.rs` to clone `ContextOpt` when adding rows, ensuring correct context management during transformations.
- **Server Pipeline Adjustments**:
- Updated `pipeline.rs` in `servers` to handle transformed outputs with one-to-many row expansions, ensuring correct row padding and request formation.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
Add one-to-many VRL pipeline test in `http.rs`
- Introduced `test_pipeline_one_to_many_vrl` to verify VRL processor's ability to expand a single input row into multiple output rows.
- Updated `http_tests!` macro to include the new test.
- Implemented test scenarios for single and multiple input rows, ensuring correct data transformation and row count validation.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Add Tests for VRL Pipeline Transformations
- **File:** `src/pipeline/src/etl.rs`
- Added tests for one-to-many VRL pipeline expansion to ensure multiple output rows from a single input.
- Introduced tests to verify backward compatibility for single object output.
- Implemented tests to confirm zero rows are produced from empty arrays.
- Added validation tests to ensure array elements must be objects.
- Developed tests for one-to-many transformations with table suffix hints from VRL.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Enhance Pipeline Transformation with Per-Row Table Suffixes
- **`src/pipeline/src/etl.rs`**: Updated `TransformedOutput` to include per-row table suffixes, allowing for more flexible routing of transformed data. Modified `PipelineExecOutput` and related methods to
handle the new structure.
- **`src/pipeline/src/etl/transform/transformer/greptime.rs`**: Enhanced `values_to_rows` to support per-row table suffix extraction and application.
- **`src/pipeline/tests/common.rs`** and **`src/pipeline/tests/pipeline.rs`**: Adjusted tests to validate the new per-row table suffix functionality, ensuring backward compatibility and correct behavior in
one-to-many transformations.
- **`src/servers/src/pipeline.rs`**: Modified `run_custom_pipeline` to process transformed outputs with per-row table suffixes, grouping rows by `(opt, table_name)` for insertion.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Update VRL Processor Type Checks
- **File:** `vrl_processor.rs`
- **Changes:** Updated type checking logic to use `contains_object()` and `contains_array()` methods instead of `is_object()` and `is_array()`. This change ensures
compatibility with VRL type inference that may return multiple possible types.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
- **Enhance Error Handling**: Added new error types `ArrayElementMustBeObjectSnafu` and `TransformArrayElementSnafu` to improve error handling in `etl.rs` and `greptime.rs`.
- **Refactor Error Usage**: Moved error usage declarations in `transform_array_elements` and `values_to_rows` functions to the top of the file for better organization in `etl.rs` and `greptime.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Update `greptime.rs` to Enhance Error Handling
- **Error Handling**: Modified the `values_to_rows` function to handle invalid array elements based on the `skip_error` parameter. If `skip_error` is true, invalid elements are skipped; otherwise, an error is returned.
- **Testing**: Added unit tests in `greptime.rs` to verify the behavior of `values_to_rows` with different `skip_error` settings, ensuring correct processing of valid objects and appropriate error handling for invalid elements.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Commit Summary
- **Enhance `TransformedOutput` Structure**: Refactored `TransformedOutput` to use a `HashMap` for grouping rows by `ContextOpt`, allowing for per-row configuration options. Updated methods in `PipelineExecOutput` to support the new structure (`src/pipeline/src/etl.rs`).
- **Add New Transformation Method**: Introduced `transform_array_elements_to_hashmap` to handle array inputs with per-row `ContextOpt` in `HashMap` format (`src/pipeline/src/etl.rs`).
- **Update Pipeline Execution**: Modified `run_custom_pipeline` to process `TransformedOutput` using the new `HashMap` structure, ensuring rows are grouped by `ContextOpt` and table name (`src/servers/src/pipeline.rs`).
- **Add Tests for New Structure**: Implemented tests to verify the functionality of the new `HashMap` structure in `TransformedOutput`, including scenarios for one-to-many mapping, single object input, and empty arrays (`src/pipeline/src/etl.rs`).
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Refactor `values_to_rows` to Return `HashMap` Grouped by `ContextOpt`
- **`etl.rs`**:
- Updated `values_to_rows` to return a `HashMap` grouped by `ContextOpt` instead of a vector.
- Adjusted logic to handle single object and array inputs, ensuring rows are grouped by their `ContextOpt`.
- Modified functions to extract rows from default `ContextOpt` and apply table suffixes accordingly.
- **`greptime.rs`**:
- Enhanced `values_to_rows` to handle errors gracefully with `skip_error` logic.
- Added logic to group rows by `ContextOpt` for array inputs.
- **Tests**:
- Updated existing tests to validate the new `HashMap` return structure.
- Added a new test to verify correct grouping of rows by per-element `ContextOpt`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Refactor and Enhance Error Handling in ETL Pipeline
- **Refactored Functionality**:
- Replaced `transform_array_elements` with `transform_array_elements_by_ctx` in `etl.rs` to streamline transformation logic and improve error handling.
- Updated `values_to_rows` in `greptime.rs` to use `or_default` for cleaner code.
- **Enhanced Error Handling**:
- Introduced `unwrap_or_continue_if_err` macro in `etl.rs` to allow skipping errors based on pipeline context, improving robustness in data processing.
These changes enhance the maintainability and error resilience of the ETL pipeline.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/allow-one-to-many-pipeline:
### Update `Row` Handling in ETL Pipeline
- **Refactor `Row` Type**: Introduced `RowWithTableSuffix` type alias to simplify handling of rows with optional table suffixes across the ETL pipeline.
- **Modify Function Signatures**: Updated function signatures in `etl.rs` and `greptime.rs` to use `RowWithTableSuffix` for better clarity and consistency.
- **Enhance Test Coverage**: Adjusted test logic in `greptime.rs` to align with the new `RowWithTableSuffix` type, ensuring correct grouping and processing of rows by TTL.
Files affected: `etl.rs`, `greptime.rs`.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/change-tsid-gen:
perf(metric-engine): replace mur3 with fxhash for faster TSID generation
- Switches from mur3::Hasher128 to fxhash::FxHasher for TSID hashing
- Pre-computes label-name hash when no nulls are present, avoiding redundant work
- Adds fast-path for rows without nulls; falls back to slow path otherwise
- Updates Cargo.toml and lockfile to reflect dependency change
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/change-tsid-gen:
fix: only check primary-key labels for null when re-using cached hash
- Rename has_null() → has_null_labels() and restrict the check to the
primary-key columns so that non-label NULLs do not force a full
TSID re-computation.
- Update expected hashes in tests to match the new logic.
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/change-tsid-gen:
test: add comprehensive TSID generation tests for label ordering and null handling
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/change-tsid-gen:
bench: add criterion benchmark for TSID generator
- Compare original mur3 vs current fxhash fast/slow paths
- Test 2, 5, 10 label sets plus null-value slow path
- Add mur3 & criterion dev-deps; register bench target
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/change-tsid-gen:
test: stabilize metric-engine tests by fixing non-deterministic row order
- Add ORDER BY to SELECTs in TTL tests to ensure consistent output
- Update expected __tsid values after hash function change
- Swap expected OTLP metric rows to match new ordering
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat/change-tsid-gen:
refactor: simplify Default impls and remove redundant code
- Replace manual Default for TsidGenerator with derive
- Remove unnecessary into_iter() call
- Simplify Option::unwrap_or_else to unwrap_or
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
discord9
Yvan Wang
shuiyisong
Yash Agrawal
Lei, HUANG
LFC
Ning Sun
dennis zhuang
Yingwen
Han
QuakeWang
Weny Xu
jeremyhi
Ruihang Xia
fys