I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Split out of #9868, where it did not belong: the rename is not part of
the wiring.
## What's changed and what's your intention?
`datatypes::series::MetricName` did not hold a metric name. It held the
whole
PromQL-visible label set of one series — the `__name__` value plus its
ordinary
labels — with methods for label lookup, rewriting, keeping and removing
that
operate on the whole set. The name made every call site read as if a
series were
identified by its metric name, which it is not: two series share a
metric name
and differ in their other labels.
Rename the type to `SeriesLabels` so the name says what it holds, and
state in
its doc comment what it is *not*:
- not a canonical matching/grouping identity — matching drops `__name__`
and
compares the remaining labels canonically;
- not a storage identity — that is the read key / primary key.
The rename is mechanical: 58 files, +290/−279. No field, method, or
behaviour
changes. Identifiers that merely *begin* with the old name are untouched
—
`MetricNameLabel`, `MetricNameDiscovery`,
`PrometheusMetricNamesQueryPlan` keep
their names, since they refer to a metric name in the narrow sense and
are
correct as they are.
### Limitations
None. This is a rename; the type's representation and semantics are
unchanged.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and what's your intention?
**Summarize your change**
The native PromQL runtime addressed a node by the path that reached it.
Every
execution walked that path again to find the node's state, every metric
export
walked it to name a node, and the state itself lived in tables keyed by
that
path. This removes `NodePath` and the tables keyed by it, and replaces
them with
one recursive runtime tree: each node owns its children, its operator
state, its
metrics, and — once settled — nothing it no longer needs.
**How it works**
`NativeExecutionState` now holds a `RuntimeTree { root: RuntimeNode }`.
A
`RuntimeNode` carries its own `Box`ed children, its `Option` payload
(the
evaluator, the retained operands, the fold, the collected input), its
lifecycle
state and a shared `Arc<NodeMetrics>`, so there is no side table to look
a node
up in. Pulling, settling, flushing after a subtree closes, EOF, errors
and drop
all recurse through that tree.
An `InputId` is assigned only to the leaves that actually receive a
feed, by a
depth-first walk of the built tree. A scalar leaf takes no id, so no
node is
addressed by position and no flat registry stands in for the topology.
The feed
API is `push_raw(InputId, chunk)` / `push_evaluated(InputId, chunk)`;
there is no
`push_at` shim and no `input_count` nobody calls.
`EXPLAIN` and the per-node metrics are derived from the same tree: a
node's name
is produced only when a metrics set is exported, and the export walks
the tree at
that moment rather than consulting a stored path. `ANALYZE` reads the
same
runtime tree.
**Limitations**
- A node kind the runtime does not execute yet is still rejected when
the
execution is built, naming the kind. That set is unchanged by this PR.
- This is a runtime-layout change only: the compiled `NativeTree` stays
immutable
and carries no state, so it can still be used to build independent
executions.
- No persisted format, wire format or protocol changes.
**API or data compatibility**
No. Nothing here is persisted or sent over the wire; `InputId` is an
in-process
identity. Data and schema compatibility are unaffected.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels.
### Verification
- `cargo nextest run -p promql -p query` — **1829 passed, 2 skipped**
- `cargo clippy -p promql -p query --all-targets --all-features -- -D
warnings` — clean
- `cargo check -p promql --all-targets` and `-p query --all-targets` —
clean
- `hawkeye check` — no missing headers
Apply only the type rename to the surviving recursive runtime and tests; preserve the removal of global final-label occupancy and speculative split-series admission guards.
Signed-off-by: discord9 <discord9@163.com>
Wires the native PromQL route into the production planner so a real
PromQL statement reaches `FusedSeriesExec` and runs. Scope is a
standalone, single-region point of contact: `rate(x[5m])` and
`abs(rate(x[5m]))` execute through the node, and every shape the
compiler does not accept keeps the plan the existing planner builds.
Depends on the node `FusedSeries` / `FusedSeriesExec`, which landed in
`narthex` as #9820.
## The route
A statement is offered to `native_compile::compile` at its root inside
`PromPlanner`, before the expression is split into the nodes a range
function and the math functions over it would have shared. The route is
offered only when **both** hold: the engine-wide
`query.experimental_native_promql` option (retained in
`QueryEngineState`) and a per-request query-context extension of the
same name. Either alone keeps the existing plan, so a client cannot turn
the route on against the server's configuration.
A refused shape is the route declining the query, not an approximation:
planning continues exactly as it would with the flag off.
## Getting the node to the region
**Placement.** `check_extension_plan` answered `Unsupported` for the
node, which the analyzer treats as a boundary. It now answers
commutative when the input is exactly the table scan (the region builds
the node from the scan's own entrypoint and cannot look through a filter
or projection), the node binds a metric name, and no node of its tree
folds across series.
**Encodability.** `should_expand` asks whether a plan encodes to
Substrait before it asks whether a node commutes. That is right for a
region reached over the wire and wrong for standalone, whose region
server is in the same process. The engine now knows whether its regions
are reached in process, and the exception is scoped to a plan whose
**root** is `FusedSeries`; every other plan keeps the ordinary
whole-plan fast path and the per-node check. Both transports are covered
by the analyzer tests.
**Scan hint.** A region only prepares its series scan for a per-series
request. The existing `ScanHintRule` now attaches `PerSeries` to the
scan that is the immediate input of a fused node; the duplicate helper
that used to do it from the planner is deleted.
**Time bounds.** The native scan is the selector's own scan, so it gets
the same bounds the legacy selector path computes:
`build_time_index_filter` with the selector range, `offset` and `@`
anchor, pushed into `TableScan.filters` so the plan stays a bare
`TableScan`. Without this the scan read the table's whole retention.
## Cross-region correctness
A region evaluates the whole tree over the series its own scan returns,
so a series split across storage regions loses window contributions
instead of failing — `rate(cpu[2m])` on a time-partitioned table returns
nothing where the complete series answers `1.0`.
`single_partition_scan()` only covers scan partitions inside one region.
The route therefore declines unless **every partition column of the
table is one of the selector's own tag columns**: no partition columns
is a single region, and a tag-determined partition key cannot split one
series. Anything else, including internal columns, falls back to the
existing plan. No exchange and no region-count probing.
## Verification
Real standalone, `experimental_native_promql = true`, physical plan from
`datafusion::physical_planner=debug`:
```
FusedSeriesExec: rate(cpu[5m])
NativeTree
root Window rate [300000ms] ...
SeriesScanExec ...
```
- `rate(cpu[5m])` and `abs(rate(cpu[5m]))` return the same series,
timestamps and values as the existing path on a dense series, a sparse
one and a two-point series.
**Multi-partition, with a flush in the middle.** `[query] parallelism =
4`, 8 series with a `shard` tag written in two rounds with `ADMIN
FLUSH_TABLE` between them, so each series' samples span a memtable and
an SST. The physical plan reports `SeriesScanExec: ... partitions=4` and
`FusedSeriesExec`, and for both `rate(cpu[5m])` and `abs(rate(cpu[5m]))`
all 40 emitted rows match the legacy plan position by position
(timestamps, values, `host`/`shard` labels). A series split across
partitions would have made `rate` lose the earlier sample, so the
row-for-row match is the check that each series stayed in one scan
partition.
**Scan parallelism is regression-tested in the workspace, not just at
the seam.** Moving the range-assignment rule into `table::table::scan`
initially broke two callers that no single-crate test covered
(`src/cmd/src/datanode/scanbench.rs` and `parallelize_scan`'s own test
module); `cargo check --workspace --all-targets` is what caught them,
and it now passes along with `query --lib` 897 passed.
- `/v1/prometheus/api/v1/query` and `/query_range` responses are
identical between the two routes.
- `/v1/promql` now emits the same column order as the existing path
(time index first).
Sqlness, standalone and distributed: `distinct`, `latest_per_series`,
`neumann` pass.
Focused unit tests, 13 passed / 0 failed (`native_fused_*` and the
request-opt-in parser after the admission fix; the analyzer and
scan-hint checks below are separate invocations): `native_fused_*`
(including the time-partitioned fallback and the tag-partitioned
still-native case), the request-opt-in parser, both analyzer
encodability checks under **both** transports, the ordinary-plan
fast-path assertion, and the three `ScanHintRule` cases. Clippy `-p
query --all-targets -D warnings` and fmt clean.
## Admission: what the route refuses
The region's raw-series preflight refuses an append-mode table and any
time index that is not millisecond, and the physical layer turns that
refusal into a planning error rather than a fallback. The logical route
therefore declines those tables itself, so such a query keeps the
existing plan instead of failing after the logical plan is already
fixed. Both cases have fallback tests asserting the native and flag-off
plans are identical.
## What the route refuses, and why that is a fallback rather than an
error
Every refusal below happens while the logical plan is still this
function's to choose, so the query keeps the plan the existing planner
builds. That matters because once a `FusedSeries` exists the logical
plan is fixed, and the physical layer can only report an error - it
cannot undo the choice.
- **A shape the compiler refuses**, a table it cannot bind, a remaining
label matcher, `__field__`, a subquery (the selector walk does not enter
one), or more than one selector.
- **A tree that reads the whole vector at a point** (`Aggregate`,
vector-vector `Binary`, and `Vector`: `sort`, `absent`, `topk`,
`histogram_quantile`, ...). One executor runs per scan partition, and a
partition holds only part of a metric's series.
- **A table whose own settings the region cannot frame**: a
non-millisecond time index, append mode, a partition column that is not
a tag, a field that is not numeric (a decimal is not: the region asks
`ConcreteDataType::is_numeric`, which excludes them), or an engine
without a raw-series scanner. Mito's series scanner is the one that
implements the raw path; a file-engine table is read by a
`SinglePartitionScanner`, which cannot frame a series. A metric-engine
logical table is refused too: it reads through a mito scanner, but the
planner only reaches this node with a bare scan when it could not
resolve the physical table, and that table's `TableInfo` carries no
`append_mode` - so the region's own eligibility could not be checked
here. That is a false negative that costs the query its native route,
not an error.
- **A plan that cannot reach a region**: regions reached over the wire
(a distributed frontend) and a PromQL statement embedded in a hybrid
CTE, where the node would sit under a SQL root and the in-process
encodability exception - which is scoped to a plan whose root is the
node - would not apply.
Two shapes fall back for a simpler reason, and are recorded as limits
rather than fixed here because supporting them would widen this POC to
scalar broadcast: a negation or a binary operation mentioning a selector
(`-m`, `rate(m[5m]) * 100`) is not found by the selector walk, and a
statement whose bounds precede the Unix epoch cannot be represented as a
`SystemTime`. Both answer correctly on the existing path.
- **`TQL EXPLAIN` / `TQL ANALYZE` still print the existing plan.** Those
statements wrap the expression in an explain extension before this route
sees it, and the selector walk does not enter a wrapper, so the route
declines. The query itself answers the same way; what differs is that
the displayed plan does not show the node the same statement executes
through `TQL EVAL`. Closing that needs the wrapper recognised and
re-applied around a native plan, which is deliberately not part of this
POC.
## Known limits, stated plainly
- **The value column is named `val`, not the expression-derived name**
the legacy projection carries
(`prom_rate(ts_range,val,ts,Int64(300000))`). Prometheus API output is
unaffected because it reads column roles, not names; the
greptimedb-shaped `/v1/promql` response shows the difference.
- **Scan parallelism is preserved.** The fused leaf needs a *prepared*
scan, because `ParallelizeScan` runs after physical planning and cannot
see a `SeriesScanExec`. Preparation now uses the same range-assignment
rule the parallelizer uses —
`table::table::scan::assign_partition_ranges`, moved there and shared by
both callers — with the session's `target_partitions`, and it leaves an
already-prepared scan untouched. The inherited single-partition
restriction in `resolve_named_series_binding` is gone, so a
multi-partition scan resolves its binding; the series producer still
keeps one series inside one scan partition, so each partition feeds
`FusedSeriesExec` with whole series.
- **Each series must stay inside one region.** The guard declines the
route unless every partition column is one of the selector's own tag
columns, so a table partitioned by time, or by anything that is not a
series label, falls back to the existing plan instead of answering
wrongly. A tag-partitioned table is accepted and does use the native
route, because a tag-determined partition key cannot split one series.
The guard is a sufficient condition, not a proof for every partitioning
scheme: it has not been exercised against a concurrent repartition
lifecycle.
- **Anything with a label matcher falls back**, because a matcher that
survives preprocessing selects fewer rows than the scan below the node
reads.
- **Distributed is unsupported**: in a frontend the metric scan is
rewritten to `MergeScanLogicalPlan`, the node is not placed there, and
the flag is off by default.
- **No aggregates, scalars or subquery execution.**
Preserve the POC removal of global output-label occupancy and speculative split-series guards while adopting InputId leaf routing and execution-owned metric handles. Let NativeExecutionState own its recursive root directly.
Signed-off-by: discord9 <discord9@163.com>
Remove plan-global label-set occupancy and the speculative split-series tree guard. Keep the native evaluator, finality, memory budgets, errors and ordinary RecordBatch output local to each scan partition.
Narrow native admission to selectors and the small unary POC subset. Document that cross-partition final-label conflicts and semantic merging are outside this POC, and cover real rate/abs(rate) execution across dense and sparse storage spanning SST and memtable data.
Signed-off-by: discord9 <discord9@163.com>
Keep native queries available for unflushed memtables and SSTs with incomplete columnar series or samples coverage by selecting the ordinary raw-series reader. Preserve structural validation and propagate artifact read errors instead of retrying them through legacy execution.
Cover all four incomplete-readiness states in public PromQL queries, both storage scan modes and ordering, complete prepared-source execution, and artifact error propagation.
Signed-off-by: discord9 <discord9@163.com>
Register each execution's node metric handles once in the physical plan's metrics set. Reading metrics no longer retains or locks runtime topology. Keep historical counters alive after execution drop and cover repeated reads and executions.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The histogram-function and range-modifier tests that landed on the branch use
the execution entry points as they were before the runtime took an input
position, so they name the leaf they push at.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The runtime takes an input position now, so the tests that a label rewrite
brought in name the leaf they push at, and the map import no longer carries the
operation type the runtime reads through `MapOp`.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
A node was identified by the path that reached it, so every execution and
every metric export walked that path again, and a node's own state lived in
tables keyed by it. This replaces that with one recursive runtime tree: each
node owns its children, its operator state, its metrics and, once settled,
nothing it no longer needs.
An `InputId` is assigned only to the leaves that actually receive a feed, by a
depth-first walk of the built tree, so a scalar leaf takes no id and no node is
addressed by position. Pulling, settling, flushing after a subtree closes,
EOF, errors and drop all recurse through the same tree.
Signed-off-by: discord9 <discord9@163.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Native operator kernels, NEW-EVAL-MOD (#27).
## What's changed and what's your intention?
Prometheus v3.15.0 adds two experimental range selector modifiers behind
`EnableExtendedRangeSelectors`: `anchored` and `smoothed`. This adds
them to the range evaluator as `RangeWindow.modifier:
Option<RangeModifier>`.
- **What they read.** `anchored` also reads the statement's lookback
before the window start. `smoothed` reads the lookback before the start
and after the end. The evaluator keeps samples that much longer and
widens a smoothed window's end.
- **Kernels** follow `extendedRate`, `pickOrInterpolate*`,
`interpolate`, `correctForCounterResets` and `pickFirstSampleIndices`:
- `rate`, `increase` and `delta` take the start value from the last
sample at or before the start. For `smoothed` they interpolate both
ends, a counter that reset between the two samples restarts from zero,
and nothing is extrapolated.
- `resets` and `changes` (anchored only) count from that anchor sample.
- **Allowed functions.** These match `AnchoredSafeFunctions` /
`SmoothedSafeFunctions`; any other function is rejected with Prometheus'
message. Evaluated input (subqueries) is refused; the parser does not
allow modifiers there either.
- **When a smoothed point is decided.** A smoothed end needs the first
sample at or after the point. A point is therefore decided once every
field holds such a sample, or once the lookback past it has arrived. The
chunk's last row alone is not enough: a stale last row is dropped, and a
field can be empty in it.
- **Not here:**
- Native histogram samples under a modifier are refused with
`NotImplemented`. Prometheus runs `extendedHistogramRate` there, and
returns nothing with a warning for mixed windows.
- No compiler wiring: promql-parser 0.10 lexes the keywords but has no
grammar or AST for them.
- A scan feeding a modified window has to read the extra lookback; the
`RangeWindow` doc says so.
- **Display:** `rate [5m] smoothed lookback 300000ms ...`.
### Tests
- Golden `range_eval/modifier_cases.txt` (43 KB, 100 cases) from
`promql-compat/cmd/narthex-golden-rangemod`. It has 50 random cases and
50 cases with whole-second times that put samples exactly on the window
and lookback bounds. Each case is pushed as one chunk and as one chunk
per sample, and the cases for unsupported functions expect the
rejection.
- A smoothed `delta` whose first chunk ends on a stale row must wait for
the next sample to interpolate the end. This case fails when readiness
comes from the chunk's last row.
- Display of a modified window.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Native tree operator wiring, native histogram functions (#25). Follows
#9845 → #9846 → #9848 (all merged).
## What's changed and what's your intention?
`histogram_count`, `histogram_sum`, `histogram_avg`, `histogram_stddev`
and `histogram_stdvar` now compile into the native tree and run in the
existing `Map` node.
- **Plan.** `MapOp::Histogram(NativeHistogramFunction)`. These are
pointwise per series like math (Prometheus `simpleHistogramFunc`), so no
new node or output type is needed.
- **Runtime.** `map_chunk` calls the existing `map_native_histogram`
kernel: only histogram positions produce a value, float positions become
holes, and only `__name__` is dropped (`__type__`/`__unit__` stay,
unlike math). When the op empties a closing chunk, the closing series is
named the same way. `map_chunk` now takes `&Arc<SeriesSchema>` to call
the kernel.
- **Kernel.** `map_native_histogram` checks its input against the schema
with `validate_grid_series`, as `map_math` does. It used to check only
that the axis is a grid, and a series with more or fewer fields than the
schema indexed out of bounds.
- **Compiler.** `histogram_function` lists the five names; one vector
argument. `histogram_quantile` / `histogram_fraction` also read classic
bucket series across series, so they are not maps and are not wired
here.
- **Display.** The PromQL name.
Shared files touched: `native/plan.rs` (`MapOp`), `native/execute.rs`
(`map_chunk`), `native/display.rs`, `native_compile.rs`.
### Tests
- Runtime: a chunk holding histogram, float, histogram maps to values at
the two histogram positions with the name dropped; a closing float chunk
is emptied and keeps `__type__`.
- Kernel: a series that does not fit the schema is refused instead of
panicking.
- Compiler: the five names bind their kernel into a map.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
A doc comment on the categorizer was followed by a blank line, and the test
table builder takes eight arguments. Both are what CI's `-D warnings` turns
into errors; neither changes behaviour.
Signed-off-by: discord9 <discord9@163.com>
discord9
discord9
dennis zhuang