* feat: support ALTER TABLE SET auto_flush_interval
Closes#8394.
Add a new SetRegionOption::AutoFlushInterval variant so that the
per-table auto flush interval can be changed on existing tables
via 'ALTER TABLE t SET ...', following up the CREATE TABLE path
from #8357.
- region_request.rs: parse 'auto_flush_interval' with humantime and
map it to the new variant.
- metadata.rs: persist the value (or remove it, if None) in
TableOptions.extra_options using the same humantime string format
the engine already expects.
- handle_alter.rs: apply the new interval in
handle_alter_region_options_fast (no memtable flush needed, same
pattern as Ttl) and group the variant with Ttl/Twsc in
new_region_options_on_empty_memtable.
Tests:
- Two unit tests in metadata.rs covering set and unset-to-None.
- A new sqlness case alter_auto_flush_interval.sql covering
create-then-alter, alter-then-alter, invalid duration, and
alter on a table that already had auto_flush_interval at create
time.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: validate auto_flush_interval > 0 in ALTER SET path
Gemini code assist flagged that the request parser accepted a zero
duration, leaving the rejection to the downstream RegionOptions
validation which only fires on next flush. Reject it at parse time
so users get the error immediately at the ALTER TABLE statement.
Also add a '0s' error case to the sqlness test.
Signed-off-by: srivtx <crypticcc101@gmail.com>
* fix: handle SET 'auto_flush_interval' = NULL and add checked-in .result
Address the rest of fengjiachun's review on #8403:
1. Empty value in ALTER SET clears the override (parallels Ttl).
'ALTER TABLE t SET ... = NULL' comes through as value = ''; we
now return AutoFlushInterval(None) so the override is removed
from TableOptions.extra_options, matching the Ttl pattern.
2. Add a unit test in region_request.rs covering the four cases
(valid, empty-clears, zero-rejected, garbage-rejected).
3. Generate and check in alter_auto_flush_interval.result via
'cargo sqlness-runner bare -t alter_auto_flush_interval'. Both
the standalone and distributed sqlness jobs now pass locally,
and the test extension covers the NULL-clears path end to end.
Signed-off-by: srivtx <crypticcc101@gmail.com>
---------
Signed-off-by: srivtx <crypticcc101@gmail.com>
* feat: report region read load in heartbeat
Signed-off-by: WenyXu <wenymedia@gmail.com>
* feat: expose region query stats in information schema
Signed-off-by: WenyXu <wenymedia@gmail.com>
* chore: update sqlness result
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: record region query stats on stream drop
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix: keep region query cpu stats in nanoseconds
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
* fix(metric-engine): report query load under physical region id
Propagate the physical region ID through the scanner, record batch
stream, and query engine so that query-load metrics (CPU time, scanned
bytes) are attributed to the correct physical region rather than always
to the logical region.
- `src/store-api/src/region_engine.rs` — add `query_load_region_id` to
`ScannerProperties` and `set_query_load_region_id` to `RegionScanner`
trait
- `src/mito2/src/read/seq_scan.rs`,
`src/mito2/src/read/series_scan.rs`,
`src/mito2/src/read/unordered_scan.rs` — implement the new trait
method on each scanner
- `src/common/recordbatch/src/adapter.rs` — carry region id through
`RecordBatchStreamAdapter` into `RecordBatchMetrics`
- `src/table/src/table/scan.rs` — expose region id on `RegionScanExec`
- `src/query/src/datafusion.rs` — extract region id from the physical
plan and set it on the output stream
- `src/query/src/dist_plan/merge_scan.rs` — use metrics-contained region
id in query-load reporting, falling back to the logical region id
- `src/metric-engine/src/engine/read.rs` — set region id on the metric
engine scanner
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): satisfy clippy for query load region id
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(query): ignore missing query load region ids
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
Co-authored-by: Lei, HUANG <ratuthomm@gmail.com>
* feat: track scan output bytes and use them for read costing
Track the actual byte count of each record batch produced by
`RegionScanExec` and use it in place of the aggregated per-plan
`memory_usage` as the `table_scan` cost input. This avoids double
counting bytes that flow through multiple operators.
A named constant `REGION_SCAN_EXEC_NAME` exposes the plan node name
so downstream metric parsers remain correct if the struct is renamed.
Affected files:
- `src/table/src/table/metrics.rs` -- add `output_bytes` counter
- `src/table/src/table/scan.rs` -- record bytes, export name constant
- `src/query/src/dist_plan/merge_scan.rs` -- consume scan bytes
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* refactor: remove misleading `mem_used` gauge from scan metrics
The `mem_used` gauge was used with `add()` on every output batch, but a
RegionScan does not hold memory across polls — each batch is yielded
immediately to the upstream operator. The gauge semantics were
incorrect (cumulative `add()` on a gauge) and the value duplicated
`output_bytes` anyway.
Affected files:
- `src/table/src/table/metrics.rs` — drop `mem_used` field and methods
- `src/table/src/table/scan.rs` — remove `record_mem_usage` call
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: use stable plan names for scan byte metrics
Expose `plan_name` alongside rendered plan text so scan byte extraction no longer depends on EXPLAIN formatting. Keep old serialized metrics compatible by defaulting missing `plan_name`.
Affected files:
- `src/common/recordbatch/src/adapter.rs` -- add `plan_name` to `PlanMetrics` and populate it from `ExecutionPlan::name`
- `src/query/src/dist_plan/merge_scan.rs` -- match `REGION_SCAN_EXEC_NAME` through `plan_name`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: preserve merge scan CPU read cost
Record read cost whenever stream metrics are available, default scan bytes to zero, and aggregate scan output bytes across region scan nodes.
Files: `src/query/src/dist_plan/merge_scan.rs`
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* fix: correct doc comment in `StreamMetrics`
Fix doc comment on `StreamMetrics::new()` to reference the correct
struct name (`StreamMetrics`) instead of the old `MemoryUsageMetrics`.
- `src/table/src/table/metrics.rs` — fix struct name in doc comment
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
---------
Signed-off-by: Lei, HUANG <mrsatangel@gmail.com>
* feat: table semantic layer per-table enrichment (Phase 2)
Phase 2 of the table semantic layer, plus a vocabulary trim so the layer only
records what a machine consumer cannot cheaply recover on its own.
Per-table metric enrichment (OTLP), via an internal per-table channel:
- A `SemanticIndex` accumulator records, per emitted table, the declared metric
keys: type / unit / temporality / metadata_quality=declared / original_name.
Conflicting single-valued keys collapse to `mixed`/`unknown`.
- Recording happens at the `encode_metrics` level where the base name, metric
type, and proto fields are all in scope, so histogram/summary fan-out gets the
correct per-subtable type (`_bucket`=histogram, `_sum`/`_count`=counter)
without threading state through every encoder.
- The index is serialized onto the `greptime.internal.semantic.per_table_index`
context extension; `apply_per_table_semantic_options` folds each table's keys
into its options at auto-create time.
- `trace.conventions` is refined from the request's resource/scope `schema_url`s
(concrete when uniform, else `mixed`/`unknown`).
Vocabulary trimmed to only meaningful keys. Kept: signal_type, source, pipeline,
trace.conventions, metric.{type,unit,temporality,metadata_quality,original_name}.
Dropped: metric.monotonic (a function of type), trace.has_events/has_links
(constant + derivable from columns), log.severity_scheme/body_format (constant /
derivable, and body_format cost an O(rows) scan), resource/scope lineage
(restates columns / collector-config concern), source_version (no cheap
non-constant value today). Prometheus carries type/unit in the metric name by
convention, so it gets identity only — no inferred enrichment.
Identity (signal_type + source) extended to the remaining ingest protocols so
the discovery view is complete: InfluxDB and OpenTSDB (metric), Loki and
Elasticsearch (log). These protocols carry no type/unit metadata, so identity is
all that applies.
Tests: unit coverage for the accumulator, per-metric-type fan-out, and trace
conventions; integration goldens updated for the OTLP metric/trace SHOW CREATE
output and the new Loki identity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: validate the option value
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: table semantic layer identity (Phase 1)
Attach a thin layer of semantic metadata to ingested tables via the existing
`table_options` slot, so machine consumers (LLM agents, alert/dashboard builders,
MCP servers, ETL) can align a table with the observability concept it stands for
without guessing from column names. See docs/rfcs/2026-05-28-table-semantic-layer.md.
Phase 1 (identity) only:
- New `table::requests::semantic` module: the `greptime.semantic.*` vocabulary
(signal/source/source_version/pipeline + trace/metric/log/resource-scope keys,
defined now, populated by later phases), value constants, the internal
`greptime.internal.semantic.per_table_index` transport key (reserved for Phase 2,
deliberately outside the public namespace), and `is_semantic_option_key`.
- `validate_table_option` accepts the `greptime.semantic.*` prefix, so the keys are
valid both on the auto-create path and on explicit `CREATE TABLE ... WITH (...)`.
- `fill_table_options_for_create` copies every semantic ctx extension into the new
table's options (prefix passthrough alongside the fixed allowlist).
- Frontend stamps identity on the context at each ingest entry: OTLP metrics
(metric/opentelemetry), traces (+pipeline, has_events/has_links/conventions for
the v1 model), logs (log/opentelemetry), and Prometheus remote write
(metric/prometheus, metadata_quality=inferred). OTLP metric metadata_quality is
left for Phase 2 (declared).
- Trace identity is stamped only on the main span table; the derived
`_services` / `_operations` lookup tables keep the unstamped context and carry no
semantic identity (cross-table relationships are out of scope).
Semantic options appear in SHOW CREATE TABLE (like table_data_model /
otlp_metric_compat) and in information_schema, so an LLM inspecting a table sees its
semantics directly.
Tests: unit (validation prefix + internal-key rejection, ctx passthrough) and
integration assertions that the common keys land for OTLP metrics (metric-engine
logical table), traces, logs, and Prometheus remote write; SHOW CREATE goldens
updated.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: prom batcher not cover and white list for semantic keys/values
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: typo
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: switch partition tree to bulk
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: keep partition tree memtable for migration test
Restore PartitionTreeMemtable construction when memtable.type=partition_tree
is explicit, and move the sparse-encoding bulk override into the default
(no explicit memtable.type) arm so phase 2's memtable.type=bulk wins on
reopen. Rewrite test_reopen_time_series_sparse_memtable_with_bulk to use a
metric-engine-shaped schema and sparse-encoded rows with WriteHint::Sparse,
so the test actually exercises a PartitionTreeMemtable in phase 1 and
verifies WAL replay into the new BulkMemtable on reopen without flushing.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: drop partition tree memtable from runtime
Re-apply the unconditional sparse-encoding override in
`MemtableBuilderProvider::builder_for_options` and route the
`MemtableOptions::PartitionTree` arm to `BulkMemtable` with a deprecation
warning. After this change, `PartitionTreeMemtableBuilder` is no longer
reachable from the engine runtime; benchmarks still reference the type.
Remove `test_reopen_time_series_sparse_memtable_with_bulk` and the
`put_sparse_rows` helper added in the previous commit — that test only
existed to validate the PartitionTree -> Bulk reopen migration and is
unnecessary now that the override is in place.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): move timestamp_array_to_i64_slice into read module
Relocate the timestamp_array_to_i64_slice helper from
memtable/partition_tree/data.rs to the read module so that the read
path no longer depends on the partition_tree internals. All call sites
(both inside and outside the partition_tree module) now import from
crate::read.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito2): use TimeSeriesMemtableBuilder in time_partition tests
The time_partition tests use the memtable builder purely as a generic
backend for the TimePartitions write/scan paths; nothing in them is
specific to the partition-tree memtable. Switch the seven affected
tests to TimeSeriesMemtableBuilder so the tests no longer depend on
PartitionTreeMemtableBuilder.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): delete PartitionTreeMemtable implementation
The runtime already falls back to BulkMemtable for the PartitionTree
variant. Drop the now-unreachable implementation, its metrics, the
partition_tree benchmarks, the metric-engine Unsupported fallback in
bulk_insert.rs, and the test helpers that only existed for the deleted
module.
MemtableOptions::PartitionTree, its parsing, the runtime fallback, the
store-api MEMTABLE_PARTITION_TREE_* constants, and the SQL fixtures
remain so existing region options keep round-tripping.
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor(mito-codec): drop skip_partition_column parameter
PartitionTreeMemtable was the only caller passing
skip_partition_column=true; every other caller passes false. Now that
the partition_tree module is gone, the parameter is uniformly false
and the guard branch is dead. Drop the parameter from the trait method
and both impls, remove the guard and the is_partition_column helper,
and update the four remaining call sites in mito2 plus the bench.
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore(mito2): remove unused MemtableConfig enum
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: fmt code
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: remove unused variant
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update test_config_api
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: remove unused memtable test helpers
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: address review comment
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: support bulk memtable options
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: sanitize config
Signed-off-by: evenyag <realevenyag@gmail.com>
* feat: remove partition tree options from region options
Move primary_key_encoding to the top level
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: make ssts test datetime replaced text stable
Signed-off-by: evenyag <realevenyag@gmail.com>
* test: update sqlness result
Signed-off-by: evenyag <realevenyag@gmail.com>
* chore: validate_enum_options consider bulk memtable
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: pass region id when parsing region options
Replace the `TryFrom<&HashMap>` impl for `RegionOptions` with
`try_from_options(region_id, options_map)` so the legacy partition_tree
fallback can log the affected region. The fallback now also overrides
the SST format to flat in addition to clearing the memtable type.
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix: align sst_format with bulk memtable on parse and open
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
* refactor: remove the `RawTableMeta` and `RawTableInfo` to make codes more concise
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
* fix ci
Signed-off-by: luofucong <luofc@foxmail.com>
---------
Signed-off-by: luofucong <luofc@foxmail.com>
* feat: impl vector index query
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat: remove VectorSearchRule and merge it into scan hint rule
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor: vector search hint
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test: join and subquery
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: clippy when feature disabled
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: push hint only when column is non-nullable or an explicit IS NOT NULL filter exists
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix: transformed = true
Co-authored-by: Yingwen <realevenyag@gmail.com>
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: remove adpater vector hint
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* chore: revert transformed
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
fys
Yingwen
Lei, HUANG
Ruihang Xia
jeremyhi
sribatsha dash
Weny Xu
shuiyisong
discord9
dennis zhuang
Ning Sun
discord9
LFC
Sicong Hu