Preserve the POC removal of global output-label occupancy and speculative split-series guards while adopting InputId leaf routing and execution-owned metric handles. Let NativeExecutionState own its recursive root directly.
Signed-off-by: discord9 <discord9@163.com>
Remove plan-global label-set occupancy and the speculative split-series tree guard. Keep the native evaluator, finality, memory budgets, errors and ordinary RecordBatch output local to each scan partition.
Narrow native admission to selectors and the small unary POC subset. Document that cross-partition final-label conflicts and semantic merging are outside this POC, and cover real rate/abs(rate) execution across dense and sparse storage spanning SST and memtable data.
Signed-off-by: discord9 <discord9@163.com>
Keep native queries available for unflushed memtables and SSTs with incomplete columnar series or samples coverage by selecting the ordinary raw-series reader. Preserve structural validation and propagate artifact read errors instead of retrying them through legacy execution.
Cover all four incomplete-readiness states in public PromQL queries, both storage scan modes and ordering, complete prepared-source execution, and artifact error propagation.
Signed-off-by: discord9 <discord9@163.com>
Register each execution's node metric handles once in the physical plan's metrics set. Reading metrics no longer retains or locks runtime topology. Keep historical counters alive after execution drop and cover repeated reads and executions.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The histogram-function and range-modifier tests that landed on the branch use
the execution entry points as they were before the runtime took an input
position, so they name the leaf they push at.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
The runtime takes an input position now, so the tests that a label rewrite
brought in name the leaf they push at, and the map import no longer carries the
operation type the runtime reads through `MapOp`.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
A node was identified by the path that reached it, so every execution and
every metric export walked that path again, and a node's own state lived in
tables keyed by it. This replaces that with one recursive runtime tree: each
node owns its children, its operator state, its metrics and, once settled,
nothing it no longer needs.
An `InputId` is assigned only to the leaves that actually receive a feed, by a
depth-first walk of the built tree, so a scalar leaf takes no id and no node is
addressed by position. Pulling, settling, flushing after a subtree closes,
EOF, errors and drop all recurse through the same tree.
Signed-off-by: discord9 <discord9@163.com>
A doc comment on the categorizer was followed by a blank line, and the test
table builder takes eight arguments. Both are what CI's `-D warnings` turns
into errors; neither changes behaviour.
Signed-off-by: discord9 <discord9@163.com>
A plan holding a node with no Substrait serializer was always cut at that node,
because the analyzer asks whether a plan encodes before it asks whether a node
commutes with the region split. That is right for a region reached over the wire:
the plan is encoded into the request and has to decode on the other side.
Standalone does not use that path. Its region server is the one built in the same
process, its query handler passes the request straight to `RegionServer::handle_read`,
and the plan stays a `LogicalPlan` from the frontend to the region. Encoding it is
work nobody reads, and refusing to place a node because it cannot be encoded
refuses a plan that would have travelled intact.
So the engine now knows whether its regions are reached in process, and the
encodability gate applies only when they are not. The capability is set where it
can be stated truthfully - the standalone bootstrap, which creates both the
frontend and the region server it reads - and it defaults to false everywhere
else, so an ordinary build and every other deployment keeps today's behaviour
byte for byte.
Skipping the gate skips one early return and adds no reason to expand:
`Categorizer::check_plan` still runs and still decides, and a plan that does not
encode for any other reason still expands at the node that caused it. Two tests
hold those two halves apart.
Signed-off-by: discord9 <discord9@163.com>
The arm above answers that a fused node commutes with the region split, and it
listed "no series is split across regions" among its reasons. That is not
established anywhere. A region evaluates the whole tree over the series its own
scan returns, so a series whose samples reach two regions is evaluated per
fragment, and a range function over a fragment is not the range function over the
series.
What the placement does have is three proofs about the node - its input is the
scan, it binds a metric name, and no node of its tree folds across series. What it
does not have is any check that the series stays in one region: the storage
partition key is whatever `partition_column_names()` names and may be a column
that varies within one series, and this categorizer runs before region selection,
so there is no selected-region vector here to consult either.
So the comment now says that plainly, names the deployment the placement is
correct under, and says what would make it unconditional. A single-region
deployment satisfies it; a partitioned table over several regions would answer
wrongly rather than fail.
Signed-off-by: discord9 <discord9@163.com>
A `FusedSeries` node is built for one metric's series, and a region holds a subset
of those series with none split across regions, so the node commutes with the
region split when what it reads is per-series. It says so now, and the analyzer
may place it inside the merge scan.
Three conditions, each drawn from the conversion that has to succeed there:
- The node's input is exactly the table scan. The region builds the node from the
scan's own raw-series entrypoint and cannot look through a filter, a projection
or a sort in between, so a node over anything else has nothing to read where it
would be placed.
- The node binds a metric name, because the region resolves the series of one
named metric.
- No node of its tree folds across series. A region holds part of a metric's
series, so an aggregate or a binary operation evaluated there would consume a
partial input and return a wrong answer without saying so. Every operation the
compiler emits today - input, instant, window, map - reads one series at a
time, so this refuses nothing buildable; it is the conservative answer for a
tree built some other way.
A scan below such a node also asks for the per-series distribution now. That is
what makes a region prepare its series scan, and only that path packs a series'
chunks into the frame the node reads.
Both are inert without a `FusedSeries` node, so an ordinary query keeps its plan
and its scan request.
Not yet reached: the analyzer asks whether a plan encodes before it asks whether
a node commutes, and this node has no Substrait serializer, so the plan is
expanded at the node and the region still receives the bare scan. A test pins
that boundary so it fails the day the node can cross it.
Signed-off-by: discord9 <discord9@163.com>
Nothing called `native_compile::compile` or built a `FusedSeries`, so the native
tree and its node were library code with tests and no query behind them. This
adds the route, behind a per-query flag.
The statement is offered to the compiler at the root, where the expression the
reader wrote still exists. A shape the compiler refuses - `NotImplemented` - is
this route declining the query, and planning continues exactly as it would with
the flag off. Anything else is a real error.
The physical half is an arm in `PromExtensionPlanner`. Its children are already
built by then, so a metric scan arrives as the region scan, and the node asks that
scan for its named raw-series binding: `try_to_named_series_scan` resolves the
label source and ordering from the scanner itself, which is the only thing that
can name them, and returns the leaf the node reads. A scanner that cannot serve a
raw frame is reported, not silently re-planned.
Two things this needed that the route does not get for free:
- The region scan has to be prepared as one partition. `resolve_named_series_binding`
refuses a scan whose partition count is not one, and a fresh scan has not set
its partitions at all: `ParallelizeScan` runs after extension planning and would
spread it, so the node would run one evaluator per partition and split a logical
series. Preparing the ranges as one partition is what keeps `FusedSeriesExec`
over one series stream, and it is why this route does not scan in parallel yet.
- The scan needs the per-series distribution hint. That is what makes a region
prepare its series scan, and only that path packs a series' chunks into the frame
this node reads. The existing planner asks for it through a sort that
`ScanHintRule` turns into the hint; this node has no such sort below it, so the
hint is attached to the scan directly.
A query the route cannot bind faithfully keeps the existing plan rather than a
wrong one: more than one selector, a label matcher that would survive
preprocessing, an `__field__` matcher, a native histogram field, a missing table,
or a shape whose grid the compiler's statement cannot express.
Signed-off-by: discord9 <discord9@163.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Native tree operator wiring, label category (#25). Builds on #9846 (date
wiring, merged).
## What's changed and what's your intention?
`label_replace` and `label_join` now compile into the native tree and
run in the existing `Map` node.
- **Plan.** `BoundMap.ops` becomes `Arc<[MapOp]>`, with
`MapOp::Math(MathOp)` and `MapOp::Label(Arc<LabelRewrite>)`. A label
change is pointwise per series like math, so it needs no new node,
collector or output type.
- **Kernel.** `LabelRewrite` (in `map_label.rs`) is the checked,
compiled change: `replace` anchors the regex and checks the destination,
`join` checks source and destination names, `apply` rewrites one
`MetricName`. The whole-vector `label_replace` / `label_join` now go
through it and still merge same-labelset series afterwards, so the
Prometheus golden keeps covering the same code. Fields are private, so
only the two constructors build one.
- **Runtime.** `map_chunk` applies label ops to the series metric and
math ops through `map_math`. When a math op empties a closing chunk, the
labels of the closing series come from applying the remaining ops in
order (label ops rewrite, math ops drop the name). Series a label change
folds into one label set go through the `Emitted` ledger the map already
keeps: disjoint values merge, a shared point is refused.
- **Compiler.** `label_rewrite` reads the string-literal arguments.
Arguments Prometheus rejects at evaluation (invalid regex, empty label
name) fail the statement with `Plan`, as Prometheus fails the query;
everything unsupported stays `NotImplemented`.
- **Display.** `label_replace "dst" "repl" "src" "regex"` / `label_join
"dst" "sep" ["a", "b"]`.
Shared files touched: `native/plan.rs` (`MapOp`), `native/execute.rs`
(`map_chunk`, `no_row_series` now takes the metric),
`native/display.rs`, `native_compile.rs`. Test constructors move from
`MathOp` to `MapOp::Math`.
Known difference, not introduced here but now reachable: Prometheus
checks same-labelset duplicates separately for floats and histograms,
while the native ledger counts both as a value at a position. A
`label_replace` that folds a float series and a histogram series onto
one label set at the same step is refused natively.
### Tests
- Runtime: a relabel that folds two series refuses the shared point; a
closing chunk emptied mid-chain carries labels from label ops on both
sides of the emptying op; a label-only map hands histogram samples and
holes through unchanged.
- Compiler: both functions bind their change into a map; an invalid
regex fails the statement with `Plan`.
- `map_label` Prometheus golden (existing) now runs through
`LabelRewrite`.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
discord9
discord9
Dennis Zhuang