SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 176f6f6479)
Apply only the type rename to the surviving recursive runtime and tests; preserve the removal of global final-label occupancy and speculative split-series admission guards.
Signed-off-by: discord9 <discord9@163.com>
Preserve the POC removal of global output-label occupancy and speculative split-series guards while adopting InputId leaf routing and execution-owned metric handles. Let NativeExecutionState own its recursive root directly.
Signed-off-by: discord9 <discord9@163.com>
Remove plan-global label-set occupancy and the speculative split-series tree guard. Keep the native evaluator, finality, memory budgets, errors and ordinary RecordBatch output local to each scan partition.
Narrow native admission to selectors and the small unary POC subset. Document that cross-partition final-label conflicts and semantic merging are outside this POC, and cover real rate/abs(rate) execution across dense and sparse storage spanning SST and memtable data.
Signed-off-by: discord9 <discord9@163.com>
A node was identified by the path that reached it, so every execution and
every metric export walked that path again, and a node's own state lived in
tables keyed by it. This replaces that with one recursive runtime tree: each
node owns its children, its operator state, its metrics and, once settled,
nothing it no longer needs.
An `InputId` is assigned only to the leaves that actually receive a feed, by a
depth-first walk of the built tree, so a scalar leaf takes no id and no node is
addressed by position. Pulling, settling, flushing after a subtree closes,
EOF, errors and drop all recurse through the same tree.
Signed-off-by: discord9 <discord9@163.com>
A doc comment on the categorizer was followed by a blank line, and the test
table builder takes eight arguments. Both are what CI's `-D warnings` turns
into errors; neither changes behaviour.
Signed-off-by: discord9 <discord9@163.com>
A plan holding a node with no Substrait serializer was always cut at that node,
because the analyzer asks whether a plan encodes before it asks whether a node
commutes with the region split. That is right for a region reached over the wire:
the plan is encoded into the request and has to decode on the other side.
Standalone does not use that path. Its region server is the one built in the same
process, its query handler passes the request straight to `RegionServer::handle_read`,
and the plan stays a `LogicalPlan` from the frontend to the region. Encoding it is
work nobody reads, and refusing to place a node because it cannot be encoded
refuses a plan that would have travelled intact.
So the engine now knows whether its regions are reached in process, and the
encodability gate applies only when they are not. The capability is set where it
can be stated truthfully - the standalone bootstrap, which creates both the
frontend and the region server it reads - and it defaults to false everywhere
else, so an ordinary build and every other deployment keeps today's behaviour
byte for byte.
Skipping the gate skips one early return and adds no reason to expand:
`Categorizer::check_plan` still runs and still decides, and a plan that does not
encode for any other reason still expands at the node that caused it. Two tests
hold those two halves apart.
Signed-off-by: discord9 <discord9@163.com>
The arm above answers that a fused node commutes with the region split, and it
listed "no series is split across regions" among its reasons. That is not
established anywhere. A region evaluates the whole tree over the series its own
scan returns, so a series whose samples reach two regions is evaluated per
fragment, and a range function over a fragment is not the range function over the
series.
What the placement does have is three proofs about the node - its input is the
scan, it binds a metric name, and no node of its tree folds across series. What it
does not have is any check that the series stays in one region: the storage
partition key is whatever `partition_column_names()` names and may be a column
that varies within one series, and this categorizer runs before region selection,
so there is no selected-region vector here to consult either.
So the comment now says that plainly, names the deployment the placement is
correct under, and says what would make it unconditional. A single-region
deployment satisfies it; a partitioned table over several regions would answer
wrongly rather than fail.
Signed-off-by: discord9 <discord9@163.com>
A `FusedSeries` node is built for one metric's series, and a region holds a subset
of those series with none split across regions, so the node commutes with the
region split when what it reads is per-series. It says so now, and the analyzer
may place it inside the merge scan.
Three conditions, each drawn from the conversion that has to succeed there:
- The node's input is exactly the table scan. The region builds the node from the
scan's own raw-series entrypoint and cannot look through a filter, a projection
or a sort in between, so a node over anything else has nothing to read where it
would be placed.
- The node binds a metric name, because the region resolves the series of one
named metric.
- No node of its tree folds across series. A region holds part of a metric's
series, so an aggregate or a binary operation evaluated there would consume a
partial input and return a wrong answer without saying so. Every operation the
compiler emits today - input, instant, window, map - reads one series at a
time, so this refuses nothing buildable; it is the conservative answer for a
tree built some other way.
A scan below such a node also asks for the per-series distribution now. That is
what makes a region prepare its series scan, and only that path packs a series'
chunks into the frame the node reads.
Both are inert without a `FusedSeries` node, so an ordinary query keeps its plan
and its scan request.
Not yet reached: the analyzer asks whether a plan encodes before it asks whether
a node commutes, and this node has no Substrait serializer, so the plan is
expanded at the node and the region still receives the bare scan. A test pins
that boundary so it fails the day the node can cross it.
Signed-off-by: discord9 <discord9@163.com>
Nothing called `native_compile::compile` or built a `FusedSeries`, so the native
tree and its node were library code with tests and no query behind them. This
adds the route, behind a per-query flag.
The statement is offered to the compiler at the root, where the expression the
reader wrote still exists. A shape the compiler refuses - `NotImplemented` - is
this route declining the query, and planning continues exactly as it would with
the flag off. Anything else is a real error.
The physical half is an arm in `PromExtensionPlanner`. Its children are already
built by then, so a metric scan arrives as the region scan, and the node asks that
scan for its named raw-series binding: `try_to_named_series_scan` resolves the
label source and ordering from the scanner itself, which is the only thing that
can name them, and returns the leaf the node reads. A scanner that cannot serve a
raw frame is reported, not silently re-planned.
Two things this needed that the route does not get for free:
- The region scan has to be prepared as one partition. `resolve_named_series_binding`
refuses a scan whose partition count is not one, and a fresh scan has not set
its partitions at all: `ParallelizeScan` runs after extension planning and would
spread it, so the node would run one evaluator per partition and split a logical
series. Preparing the ranges as one partition is what keeps `FusedSeriesExec`
over one series stream, and it is why this route does not scan in parallel yet.
- The scan needs the per-series distribution hint. That is what makes a region
prepare its series scan, and only that path packs a series' chunks into the frame
this node reads. The existing planner asks for it through a sort that
`ScanHintRule` turns into the hint; this node has no such sort below it, so the
hint is attached to the scan directly.
A query the route cannot bind faithfully keeps the existing plan rather than a
wrong one: more than one selector, a label matcher that would survive
preprocessing, an `__field__` matcher, a native histogram field, a missing table,
or a shape whose grid the compiler's statement cannot express.
Signed-off-by: discord9 <discord9@163.com>
discord9
discord9
Dennis Zhuang