The node still framed its fields by hand: the grid spec's integers, a 0/1
marker for the metric name, a length prefix before every string, the tag
count, a format tag, and separate length-prefixed blobs for the partial fold
and the output schema, with a matching Reader on the way in. That is
per-field serialization sitting inside a change whose point was to stop
writing per-field serialization, and it kept the maintenance cost: any new
field touched both directions, and a test located a byte by computing
8 + tree.len() + 8 + 8 + 8.
One derived payload replaces it. The envelope is now a version, a payload
length, and a single Postcard-encoded `FusedSeriesPayload` holding the tree,
the grid spec, the binding's name/time index/tags, the output format, the
statement text, the optional partial fold and the output schema. Substrait
still carries the children, which are exactly the fields this format should
not be inventing. `text`, `Reader` and `MAX_TAG_COUNT` are gone; the tag bound
existed only because a hand-written length prefix needs one.
Every carried-schema check stays: the tag-order-sensitive comparison against
`state_schema(binding.tags)` for a partial export, exact equality with
`series_frame_output_schema` for a frame node, and the full column-role
validation for plain rows. Decode also requires the payload to be consumed
exactly, so a byte appended after a valid payload is rejected rather than
ignored -- `postcard::from_bytes` would have accepted it, which is why the
sub-payload framing had that hole.
Two tests were reading the format rather than the behaviour:
The depth-limit test asserted that `RUST_MIN_STACK` was unset, but CI sets it
to 8 MiB, so the assertion failed before the decoder ran and the test could
never do its job there. It now runs on an explicit 2 MiB worker stack, which
is the stack a normal worker has, and asserts from inside that thread that 128
nodes decode and 129 are rejected.
The integration test located the embedded version by assuming an offset and
comparing raw bytes. The framing change made that offset stale and it failed.
It now finds the envelope, decodes the version word and asserts the number,
checks the length word against the payload, and prints the observed bytes when
either assertion trips.
Signed-off-by: discord9 <discord9@163.com>
`native/wire.rs` encoded `NativeTree` by hand: fixed node tags, fixed field
layouts, both directions written out, and a `check_shape` that narrowed the
transport to `Input -> Instant|Window -> Map(Math*)`. The serialization layer
therefore decided which plans could exist. `abs(ceil(x))` compiled but could
not be transported, and every new node kind cost a second codec path plus a
new whitelist entry. Widening the whitelist to nested maps maintained that
interpreter rather than removing it.
The file is deleted. The plan types derive serde and the codec is postcard
1.1.3 -- bincode is unusable here because RUSTSEC-2025-0141 marks the whole
crate unmaintained with no patched version. postcard is maintained, has a
documented stable wire format since 1.0, and writes an f64 as its bit pattern
in eight little-endian bytes, so NaN payloads, signed zero, infinities and
subnormals survive exactly.
What remains hand-written is framing only: magic `NPRQ`, a version, a payload
length. The version is rejected before the payload is deserialized, so an
unknown version is a clean error instead of a silent misparse, and oversize,
trailing bytes and truncation are separate errors. Decoding ends at
`NativeTree::try_new`, so a tree the compiler would not build cannot be
decoded.
Three types keep written impls because they hold invariants. `EvalGrid`
serializes its `GridSpec` and rebuilds through `try_deferred` with an empty
cache -- the grid is never serialized and never materialized on decode.
`SeriesLabels` rebuilds through `try_new`, keeping the sorting and the
duplicate / `__name__`-as-a-tag rejection. `LabelRewrite` travels as its
parameters with the regex recompiled, and fails to decode if the pattern no
longer compiles; compiled internals never travel.
`FusedSeries` now carries its output schema and accepts `PlainRows` and
`partial_fold`, which it used to refuse. A carried schema is only honest if it
is checked, so decode validates every column role, the exact time type and
nullability, missing or extra value columns, and the scalar-root tag rule; a
`SeriesFrame` schema must equal the frame layout the tree derives, and a
partial export must match `state_schema(binding.tags)` including tag order.
Identity stops being display text. `FusedSeries` compared `tree.to_string()`
with a field patch, `AggregateFinalize` hashed the rendering, and
`LabelRewrite` compared its own output; all three are structural now, with
floats compared by `to_bits()` so that signed zero and NaN payloads stay
reflexive under `Eq`. `same_binding` compared `Arc::ptr_eq`, which put an
allocation address into a transportable plan; it compares the ordered fields
and the optional grid spec now, distinguishing raw from evaluated.
Tests assert the properties rather than the byte layout: a non-canonical NaN
payload round-trips by bits, an unknown version is rejected before decode even
when the payload is garbage, non-prefix trees (binary, set, aggregate, scalar,
evaluated input) round-trip, thirteen malformed schemas are each rejected with
their own message, and a multi-chunk chain over a real region is checked
against its exact sample count and contiguous timestamps.
The multi-partition integration test is split in two and labelled honestly. It
used to assert a failure, on the theory that coalescing interleaved multi-chunk
chains; its fixture's grid has one position, so no chain has more than one
chunk and that assertion was already failing at the base commit. One test now
covers the single-position shape and says it does not exercise interleaving;
the other covers a 8192-position grid that does.
Deserialization is bounded by depth, not only by bytes. The derived recursion
walks boxed nodes, and a payload under the 1 MiB limit can nest far deeper than
the stack allows, which aborts the process instead of returning an error. A
thread-local budget guards `NativeNode`'s deserializer, so a hostile payload is
rejected before a complete tree exists. `NativeTree::Deserialize` also routes
through `try_new`, so no path can materialize a tree the constructor would
refuse. Measured under postcard with no `RUST_MIN_STACK` override: 128 nodes
decode, 129 return a decode error. Postcard collapses custom serde errors to
its unit variant, so that rejection does not name the nesting limit; the other
envelope failures (version, size, trailing bytes, truncation) each keep their
own message.
Signed-off-by: discord9 <discord9@163.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
This is the wire-format half of the closed#9993 line. The raw fan-in,
aggregate split and remote-source work that #10002 also carried moves to
a separate PR on top of the public execution framework.
## What's changed and what's your intention?
### The problem
A native PromQL plan could not leave the frontend. `FusedSeries` carries
a `NativeTree`, and that tree had no wire representation, so a datanode
could only ever be asked for raw samples.
### The wire format
`src/promql/src/native/wire.rs` defines protocol version 1 for the
prefix shapes the remote route can actually produce — `Input →
Instant|Window → Map(Math*)+`, i.e. a selection under any number of math
maps — as an explicit byte layout documented in the module:
- every node kind and every operation has a **fixed tag**; the tags are
declared wire-stable and deliberately do not follow Rust discriminants,
so reordering an enum cannot change the protocol;
- floats travel as `f64::to_bits()`, so NaN payloads, signed zero,
infinities and subnormals round-trip exactly;
- the input ordering is carried as data (0 Interleaved, 1 Contiguous, 2
KeyThenTime) rather than filtered by the encoder: legality belongs to
`NativeTree::try_new` on decode, not to the transport;
- decoding rebuilds grids through their validating constructor and ends
with `NativeTree::try_new`, so a structurally invalid tree is refused at
decode; compatibility between a window modifier and its function, or a
negative `lookback_ms`, is refused later, when the datanode builds the
physical plan;
- the payload is bounded (1 MiB) and rejects unknown versions, unknown
tags, truncation and trailing bytes instead of guessing.
`Instant`'s value selector is carried as a tag (0 sample, 1 timestamp, 2
info timestamp) rather than a bool, because `InstantValue` has three
cases since `info()` became native. Anything outside the supported
shapes fails to encode *and* to decode with an error naming the
unsupported operation. `label_replace`/`label_join` are not encoded
because admission already declines them.
### Carrying it
`FusedSeries` gains `serialize`/`deserialize` (a small versioned
envelope around the tree plus its binding) and registers arms in the
existing Substrait extension serializer, so a datanode's
`DefaultPlanDecoder` already understands the node. No new transport, no
protocol change, no `greptime-proto` bump: extension payloads are opaque
bytes.
Serialization is restricted to the `SeriesFrame` output with no
`partial_fold`, because only that shape derives its output schema from
the tree and can therefore rebuild itself after decoding.
### Executing it on a datanode
The existing `PromExtensionPlanner::plan_fused_series` already knows how
to build a decoded `FusedSeries` over a `RegionScanExec`. The
integration test `native_prefix_executes_over_real_mito_region` uses a
real standalone Mito region to prove the whole path: a flushed table, an
encoded `Input → Window(rate)` prefix, `encode` → `handle_remote_read`
on the real region server, decode and execute, then unpack the returned
evaluated frame and check that the tag `host` survives, `__name__` is
dropped as `rate` requires, the grid timestamps are `3000`/`4000`, and
the rates are `20/3` and `10`. Those two numbers are not
interchangeable: a counter whose first sample is `0` cannot be
extrapolated backwards at all, while the second window's sample pattern
allows the full 1 s gap. The test also tampers with the embedded version
byte and asserts the datanode fails rather than returning a plausible
frame.
### Encode/decode agreement
A payload the encoder accepts must be one this version can decode. Three
places disagreed and are fixed here:
- **Field count.** The decoder limited an `Input`'s field count to 128
while the encoder wrote any count, so a 129-field `Input → Instant`
encoded to 983 bytes and failed to decode. Both sides now use
`MAX_FIELD_COUNT`.
- **Tag count.** `serialize` only checked that the tag count fit in a
`u32` while `deserialize` limited it to 1024. Both sides now use
`MAX_TAG_COUNT`.
- **Frame schema.** `deserialize` passed an empty schema to
`try_new_with_format`, which since #9968 *checks* a frame node's
declared schema against the layout derived from its tree instead of
overwriting it. Every serialized frame node was therefore impossible to
rebuild. `deserialize` now derives the schema from the decoded tree,
which is where it comes from in the first place.
Counts were left at their previously published values (depth 128,
operations 128, fields 128, tags 1024); only the missing encoder-side
checks and the wrong schema were added. `MAX_COUNT` is split into
`MAX_DEPTH` / `MAX_FIELD_COUNT` / `MAX_OPERATION_COUNT` so each limit
names what it bounds, and a field-count overflow no longer reports
"operation count exceeds maximum".
### Limitations
- **The frontend does not route queries through this yet.** It is a
plan-serialization and datanode-execution boundary, not a query-plan
change. Wiring the frontend to push the prefix and consume evaluated
frames is the next increment.
- Only one input source and the prefix shapes listed above are
transported; binary, vector-wide, aggregate-state and subquery trees are
not.
- No performance claim: a datanode that executes the prefix returns
evaluated frames instead of raw samples, but the frontend still gathers
every selected region into one partition.
- A selection may sit under any number of math maps, because a map is
transparent to shape: it keeps its input's samples type and grid, and
only the operation kinds are restricted. Non-math operations are still
refused at every level. The payload stays bounded by `MAX_DEPTH` (128
node levels, so a selection plus 126 maps), and both the encoder and the
decoder enforce it.
### Known limitation: the datanode boundary is not chain-atomic
The region-read merge point wraps a non-streaming plan in
`CoalescePartitionsExec`, which coalesces **per batch**. A series frame
declares `Contiguous` and one series chain spans several batches (the
packer bounds rows and flushes on `Pending`), so on a multi-partition
datanode the batches of one chain are interleaved with another's and
decoding fails with `evaluated series identity changed before its chain
closed`.
This is reproduced by `native_prefix_multi_partition_series_chains` (16
hosts × 20k samples, `rate[10s]`, 16 partitions). That test asserts the
failure, so it passes today and will turn red when the merge is fixed —
which is the signal to change it. The `parallelism: 1` path is covered
by `native_prefix_executes_over_real_mito_region`.
The fix does not belong here: the merge point should become chain-atomic
in the frontend-integration change, for example by reusing the public
`SeriesRepartitionExec`. Until then, **a multi-partition datanode read
of a series frame is not correct** — this PR is not evidence that the
datanode path is usable.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
## What's changed and your intention?
### The problem
A raw series stream reports when a series is complete by setting
`is_last` on a chunk. Its contract is "the final chunk for this series
in the input stream exposing it", but the producer only ever set it when
the next chunk of a **different** series arrived, or when the input
ended.
So a consumer could not tell a series that is finished from one that is
still being written. It had to buffer a whole series before it could act
on it, which is a cost paid whether or not the series is long.
### What this changes
`FlatSeriesBatchDivider` already emits a package only once it holds
every row of the series in it, so the record that ends such a package
also ends the last series in it. `raw::make_stream` flattened those
packages into a plain record sequence and dropped that fact. Records now
carry it:
- **The fact survives being held back.** A record held for the next
predicate group keeps its flag instead of losing it in the round trip.
- **A flagged record ends its group.** The group's end and the package's
end become the same record, so a record of the *next* package can never
be closed by the wrong boundary.
- **The tail key is taken before the residual predicates run.** A
predicate may drop the run that would have closed the series while the
package still holds all of its rows.
- **Closing a package clears `pending`, so the ordering check keeps its
own baseline.** Without it the next package's first chunk would lose the
key it must exceed and a cross-package regression would go unnoticed.
- **A series that continues past the package that completed it is
rejected**, not yielded: it would be split into two runs that both claim
to be final.
The prepared columnar source never carried the fact at all. Its
per-range readers cut complete series out, but the row-level merge
behind them re-cuts the stream by batch size, so a `Flat` package
boundary is not recoverable there. An earlier revision re-divided that
stream to restore packages; this revision drops that and yields ordinary
`RecordBatch` instead, which is why the closing below applies to the
legacy producer only. The one surviving change in `prepared.rs` skips
empty batches.
`SeriesBatch`'s doc said "batches of the same series", which contradicts
the divider that produces it (a package may hold several complete
series). Corrected.
### Limitations
- **No performance claim.** This changes *when* a consumer can observe a
complete series, not how much work the producer does. A consumer that
already buffers whole series gains nothing until it drops that
buffering.
- Package-completion information is used only under `KeyThenTime`, which
is what the one real flat producer declares: `Flat` comes from the
legacy distributor and its stream is globally ordered. A series that
reopens after the package that completed it is rejected by the
ordered-key comparison (`legacy series primary keys are not strictly
increasing`), covered by
`a_key_closed_by_a_package_must_not_continue_in_a_later_one`. Under
`Contiguous` nothing closes a flat series from package metadata, and
there is no `Contiguous` flat producer to need it.
- This does not change the `is_last` meaning or the wire representation;
it stops under-reporting it.
### Testing
Thirteen new tests. The central one,
`a_complete_package_closes_its_last_series_without_the_next_package`,
feeds one package and then never produces anything again (no next
package, no EOF, no waker). It asserts both chunks arrive with `is_last`
set. With the package fact removed the test hangs on the second chunk —
that is how the old behaviour was confirmed.
The others cover a package tail the predicate empties; a series
continued after its package, in both the same-key and after-another-key
shapes and under `KeyThenTime` (the only ordering a flat producer
declares); an ordinary `RecordBatch`, which says nothing about where a
series ends; and the batching case below.
`record_batches_separated_by_pending_share_one_predicate_pass` and the
corrected
`filtered_package_tails_do_not_close_an_incomplete_ordinary_tail` pin a
regression this revision also fixes: the fill loop awaited upstream only
when a package tail was held, so a brief `Pending` no longer splits one
predicate pass into many. Previously one hundred small record batches
separated by `Pending` cost one hundred predicate passes.
Existing tests that pin boundaries are unchanged and still pass:
`upstream_error_is_terminal_after_yielded_prefix_without_flushing_pending`,
`duplicate_or_descending_cross_batch_time_is_terminal`,
`empty_filtered_group_keeps_the_preceding_series_last_flag`.
Locally: `cargo nextest run -p mito2 -E 'test(series_scan)'` 45 passed,
`cargo clippy -p mito2 --all-targets -- -D warnings` clean, `cargo fmt
--all --check` clean. Also ran `-E 'test(series) or test(scan_test)'`
(284 passed) to cover consumers of the raw producer. No CI run claimed.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
`native_aggregate_topk_over_global_partitions` also asserted that a selected
hostname had state from two distinct original producers. That follows from
the storage scheduler, not from the merge: a scan dispatches on a channel of
capacity one and independent flushes do not fix their output partition, so a
legal schedule can leave one partition holding every series but two. The
assertion failed intermittently - `[25, 1, 1, 1]` - while the answer itself
stayed correct.
Move the requirement to a fixture that fixes which partition each contribution
arrives on, so it holds on every run. `topk_above_the_cut_sees_every_producer_
contribution` folds `topk(1, sum by (hostname)(...))` over four producers whose
contributions make the winner differ between the two grid points and require
both major producers to be summed: at point 0 producer 0 holds A=6/B=10 and
producer 1 holds A=6/C=10, so A wins on 12; at point 1 the same producers give
B=12. Two further producers contribute a small hostname each so all four
partitions carry state.
Dropping a producer or taking topk inside a producer before the merge both
answer a different winner, which the fixture catches - checked by mutating the
fixture each way and watching it fail.
The integration test keeps everything that follows from this build: the
native/legacy comparison, the hand-computed twenty rows, the two points
selecting different topk members, the producer and finalize plan shapes, and
the four non-empty producer partitions. Its now-unused partial-state recorder
is removed with it.
Signed-off-by: discord9 <discord9@163.com>
Lower MAX_DEPTH to 32. The old 128 was validated recursively, and a
debug build overflows a default test-thread stack well before reaching
it, so the depth guard itself could abort instead of refusing.
Report the refused node by its own name instead of "unsupported", which
is what every kind used to say.
Add the multi-partition datanode case. It asserts the failure the review
reproduced - a series chain cut by CoalescePartitionsExec no longer
decodes - so the limitation stays visible and turns red when the region
merge becomes chain-atomic.
Reconcile the retained-rate comment with the values the test asserts.
Signed-off-by: discord9 <discord9@163.com>
* feat(promql): name a fused tree's tags from the existing plan
A fused node declares exactly the tags the planner gives it, and reads each
one back off the series, so a label set this route cannot derive is a label
set the answer would lose. The walk over the tree derives labels for a
match, a set operation and the point-wise wrappers over them, and names
none for an aggregate's groups or a whole-vector operation - which today
declines every such statement.
Take those columns from the existing planner instead, the way the aggregate
split already does: its tag columns are what it projects for the same
statement, and they are now shared through one helper rather than written
out twice. Only the tag columns are copied, because the node names its
field columns after its own tree while the existing planner can name a
computed column after its expression.
A label rewrite is a shape the walk cannot derive at all: label_replace
adds its destination label and an empty replacement removes it, so passing
the input's labels through is an inference the rewrite contradicts. Report
no label set there. The rewrite itself keeps the existing plan, and this
does not change that: the native runtime refuses two series a rewrite gives
one label set, where that plan answers them, so admitting the shape would
turn an answer into an error. The test that holds this now covers the
two-occurrence shape as well.
An aggregate or whole-vector root does not reach this branch today: both
read across series, so they are planned by the aggregate split or fall
back. It is the prerequisite for the placement that runs such a tree in one
execution, and reaches the branch when that lands.
Signed-off-by: discord9 <discord9@163.com>
* feat(promql): run a classic histogram function above the aggregate cut
The split accepts a suffix that runs once over the fully merged series,
which is why TopK/BottomK were already admitted: the merge has folded every
partition's contribution to each group, so the series the suffix reads are
the ones the statement's input would have produced in one execution.
A classic histogram function is that kind of operation. It reads each
histogram off the bucket series that share its other labels and drops le,
and the merge has already folded each (..., le) group into one complete
series on the evaluated grid, so the buckets it reads are exactly the ones
the input would have had. Reading le off those merged series is the same
computation, not an approximation. LimitK still declines: it depends on the
storage input order, which the merge does not preserve.
This is what the o11ybench query
`histogram_quantile(0.99, sum(rate(...[5m])) by (instance, le))` needs: it
was planned by the existing path only because the split refused a
whole-vector operation above the cut.
The acceptance test runs the shape over real data and compares the rows two
executions report: the fixture spreads six bucket series over several files
and a memtable range, so the partial sums genuinely have partitions to
merge, and the native answer must equal the complete input's for quantiles
at both ends of the interpolation and for histogram_fraction. It also
asserts the native run really merged state, so a silent fall back cannot
pass it.
Signed-off-by: discord9 <discord9@163.com>
* feat(promql): read two metric logical tables as two inputs
A two-selector statement whose two sides are aggregates kept the existing
plan, even where both sides read a metric-engine logical table. The
o11ybench query `sum(max by (...) (a)) by (namespace) / sum(b) by
(namespace)` is that shape, and it fell back for three separate reasons.
First, the engine gate refused every multi-selector metric-engine
statement. The clause after it already requires each such selector to be a
recognized logical rewrite, so the removed clause refused only the case
this change handles.
Second, a metric logical table is rewritten to a scan of the physical table
restricted by `__table_id = <id>`, so two logical tables over one physical
table prepare two scans of the same table. The check deciding whether the
node binds one metric or one input per occurrence compared table names
only, so the two metrics read as one and both inputs would carry the first
one's name - a wrong answer, not a fall back. The id the rewrite already
requires is now part of that identity.
Third, a tree over two aggregates reads across series, so it is not
answered per partition, and the branch for such trees was the aggregate
split, which takes one child. A node with more than one input needs no
split: it declares one partition and reads all of every input's partitions
in one execution, so every group is folded in one place. Such a tree now
falls through to the node.
The one-selector path is unchanged: it still requires `can_split` and still
declines a tree the split cannot cut. So is the multi-selector case whose
inputs stay direct scans of one table - such a node is not region-local, as
`Categorizer::check_extension_plan` refuses a cross-series plan over more
than one scan - which is why the condition is the metric-logical-table one.
The acceptance test runs both shapes over two logical tables of one
physical table, with values that make a swapped input visibly wrong, and
compares the rows the two routes report.
Signed-off-by: discord9 <discord9@163.com>
* test(promql): log the metric pair's rows through telemetry, not stdout
The acceptance test printed its evidence with println!, which the crate
lints as an error under -D warnings, and echoed a Display value through
to_string inside the format arguments. Every other test in the file logs
through common_telemetry::info!, so this one does too.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): keep the existing plan for a tree the runtime cannot execute
The multi-input route skipped the aggregate split, and the split was what
refused a tree the runtime cannot execute: `check_cut` declines a
`quantile` and a grouping by a value field. Skipping it turned queries the
existing plan answers into errors, because the node is chosen while
planning the logical plan and a refusal the physical layer raises later has
nowhere to fall back to.
`quantile(0.5, a) / sum(b)` over two metric logical tables reported
`NotImplemented: the native runtime does not execute the Quantile { .. }
aggregate yet` where the existing plan answers it, and
`count(count by (val) (a)) / sum(b)` failed on the grouping by a value
field, which the outer `count` had hidden from the column check.
Ask the runtime before taking the route, with the same construction the
physical layer validates by, and keep the existing plan for whatever it
refuses. Asking it rather than listing the unsupported kinds is what makes
the check cover refusals this planner does not know about.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): keep the existing plan for a grouping and a selection the node cannot answer
A multi-input node skips the aggregate split, so the refusals the split
makes for reasons that are not about what the runtime computes were skipped
with it. Two of them matter here.
A grouping by the metric name keeps a label no column of the existing plan
can name. The native aggregate folds its groups by `__name__` as by any
other label, but the label set the node declares comes from the existing
planner's columns, and `__name__` is the name a raw frame carries rather
than a column of any scan - so the existing planner's `by(__name__)` finds
nothing and reports no label. The schema would then report one label set for
two groups the tree keeps apart: `sum by (__name__) (a or b)` over two
metrics would answer two rows a reader cannot tell apart, which is a series
identity lost rather than an answer given.
A selection that depends on the order its input arrives in is the other.
`limitk` keeps the first `k` series of each group in input order, while a
node with more than one input reads its inputs' partitions through a gather
that hands them over in the order they became ready: the schedule would
decide which series a query keeps. Prometheus defines the selection over a
series set. The split declines `LimitK` for the same reason. Every other
vector operation orders its input itself - `topk` by label, `sort` by value,
`limit_ratio` by a label hash - which is why only this one is asked about.
Both shapes keep the existing plan. `limitk` is not answerable by either
route today, since the existing planner has no kernel for the operator; the
point is that the native route is not the first to answer a selection the
schedule would decide.
Each guard is checked with the guard removed, so both tests are known to
fail without it.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): decline unsupported count_values groupings before execution
Signed-off-by: discord9 <discord9@163.com>
* docs(promql): scope the native capability probe guarantee
Signed-off-by: discord9 <discord9@163.com>
* test(promql): pin the same-table pair the multi-selector exclusion used to keep out
State that a Mito input joins a multi-input tree only when a second
metric logical id makes the tree multi-input, and that only LimitK's
selection depends on arrival order.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): decline ranking grouped by a value field before native execution
topk, bottomk and limitk key their groups by series labels, while the
existing plan also accepts a value field there and forms one group per
sample value. Admitted multi-input trees could therefore keep one row
where the existing plan keeps one per group. Validate the grouping when
the vector runtime is built so the statement keeps the existing plan.
Signed-off-by: discord9 <discord9@163.com>
---------
Signed-off-by: discord9 <discord9@163.com>
A payload the encoder accepts has to be one this version can decode. Three
places disagreed.
An `Input`'s field count was bounded only on the way in, so a 129-field
`Input → Instant` encoded to 983 bytes and then failed to decode. The
decoder's limit was `MAX_COUNT`, a constant that also bounds nesting depth
and `Map` operation count, which is why the missing half was easy to miss;
each limit now has its own name and both sides check it.
`FusedSeries::serialize` only checked that the tag count fit in a `u32`
while `deserialize` bounded it at 1024. Both sides now share `MAX_TAG_COUNT`.
`FusedSeries::deserialize` passed an empty schema to `try_new_with_format`,
which checks a frame node's declared schema against the layout derived from
its tree rather than overwriting it. Every serialized frame node was
therefore impossible to rebuild. The schema is derived from the decoded tree
now, which is where it comes from in the first place, so nothing is added to
the wire.
Encoding refuses an over-limit count instead of writing a payload it cannot
read back, and a field-count overflow no longer reports "operation count
exceeds maximum".
`rejects_excess_operation_count` hand-built a payload whose operation count
was read from the wrong offset: it placed an `Instant` where the decoder
expects the count, so it exercised the limit with 258 while claiming 129.
It now starts from a tree the codec accepts, asserts the bytes it edits are
the count it means, and checks the error names both the operation limit and
the value. The same is added for field count, tag count and nesting.
The integration test moves to its own file: it was written against
`tests-integration/tests/raw_series.rs`, which the raw-fan-in line creates
and this one does not carry.
Signed-off-by: discord9 <discord9@163.com>
* feat(promql): run time(), scalar(v), vector(s) and dynamic scalar broadcasts natively
Bind time() and zero-argument date functions as EvalTime, scalar(v) as ScalarOf and
vector(s) as VectorOf, and run binary operations whose scalar side varies per grid point:
the scalar is read whole, vector chunks that arrive before it are held and charged, and
both operand orders and bool comparisons follow Prometheus.
A tree holding scalar(v) runs as one execution over every scan partition, and the
planner admits it only over an unpartitioned Mito table. Data-free statements plan
without a scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): hold the labels of chunks waiting for a varying scalar
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(promql): read round and clamp parameters from varying scalars natively
Bind round, clamp, clamp_min and clamp_max whose parameters are not all constants as a
ParameterizedMap with scalar parameter nodes, run through map_math_with_parameters. The
parameters are read whole first; vector chunks that arrive before them are held and
charged, through the same hold the varying-scalar broadcast now shares. Constant
parameters keep the MathOp map.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin a parameterized map's empty closing chunk
Also correct the held-chunk docs, use same-table examples for varying map parameters,
and assert the native decline instead of the existing planner's failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): plan vector(s) natively whatever the table's field is named
The fused route declined a root field that is not a raw field, which is every root over
vector(s): it names its result `value`. Take the compiled root's fields as they are and
decline only a name the output already gives a tag or the time index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reserve a constant map parameter before building it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): leave scalar(v) to the existing planner until it has its own execution boundary
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): leave data-free statements that generate labels to the existing plan
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reserve the output point-wise kernels build before running them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(promql): run info() natively
info(v, {selector}) fell back to the existing plan, which has no info
and refuses the query ("Unsupported expr type: info", or "Cannot accept
multiple vector as function input" when a selector is given).
Compile it into the native tree and run it on the info kernel:
- Plan: NativeExpr::Info { base, info, selector }. The info side is an
instant selection reporting each selected sample's own time in
milliseconds (InstantValue::InfoTimestamp, which replaces the
BoundInstant.timestamp flag); validation refuses any other info side.
- Compiler: info_input derives the info selector the way Prometheus
does: target_info or one __name__="..." matcher, the data matchers,
and the @ and offset every selector of the first argument shares
(subquery offsets added until an @ anchors them, none when they
differ). Regex or negative name matchers, which can select several
tables, __field__/__schema__/__database__ matchers and multi-field
info tables are refused. The planner's selector walk uses the same
helper, so both bind the info table as the last input.
- Runtime: BinaryRuntime runs the kernel as PairKernel::Info, with the
retention and pool accounting of binary and set nodes; the kernel
reports its workspace through info_with_reservation.
- Planner: the info table is a second table, so it is read through its
own RawSeriesSource and the node runs on the frontend. The output adds
the info table's tags, or only the data labels the selector names.
Partitioned or non-Mito tables and a missing info table keep the
existing plan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin region-local placement of info over its own table
info(rate(target_info[5m])) reads the info table twice. Check that the
planner binds both as direct scans under one FusedSeries node bound to
the table, that the node is placed in the region, and the columns it
reports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep sample-less base series out of info's identifying patterns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): share identifying label strings in info() instead of copying them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(promql): run native trees with more than two selectors
Lift the two-selector cap of the native PromQL route. A tree of one-to-one
vector matches and set operators, nested under point-wise wrappers, is now
admitted for any number of selectors: on one table the node runs in the
region over every scan, across tables it runs on the frontend over one
RawSeriesSource per selector. Partitioned tables, aggregates, whole-vector
operations and many-to-one matches still keep the existing plan.
The dist analyzer and commutativity checks accept a same-table node with
more than two scans; without that such a plan fails in the analyzer.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover nested set operators and metric-engine multi-selector decline
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): wait for aborted partition tasks to release their inputs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 3f8699782c3116c4063ec70a3522d1f5f3bff897)
* docs(promql): say spawned partitions are released after an error, not before
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A one-to-one comparison between two selectors, such as `a > on(host) b`
or `a == bool b`, kept the existing plan: the pair admission accepted
only arithmetic operators. The kernel already pairs comparisons by the
same matching rules and only differs in the value a pair keeps (the
left sample when the comparison holds, or 1/0 with `bool`), so the
operator check is dropped. This covers one table and two, and the
point-wise wrappers around a pair, such as `(a > b) * 100`.
Adds an end-to-end case over two tables comparing explicit rows, the
legacy answer and the native placement for a filter and a `bool`
comparison.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Reuse pack_evaluated_series for fused Frame output, preserving the single native driver and PlainRows/partial timing. Reject mismatched logical frame schemas and use the reset scan plan in integration tests. Add evaluated-packer lifecycle and invalid-sequence regressions, and address the review follow-ups.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## What's changed and what's your intention?
Enable vmbench-lite 05/06/08 cross-metric arithmetic using two
independently planned native operands and the existing DataFusion binary
join, PromQL matching and arithmetic projection.
Physical shape: `SeriesScan A(N) -> Fused A(N)` and `SeriesScan B(M) ->
Fused B(M)` execute region-side; separate MergeScan branches feed the
existing frontend Join. No native Join, second matching kernel or
Exchange is added; same-table fused admission is preserved.
A bounded analyzer pass locates each direct Fused(TableScan) through
Alias/Projection/Filter/Window wrappers, rewrites only the fused
subtree, and leaves all wrappers frontend-side. This preserves global
uniqueness checks. Subquery bodies disable native admission and restore
the prior flag before returning either success or error, avoiding use of
outer evaluation bounds.
### Verification
- `make fmt-check`
- `cargo check --locked -p query -p tests-integration --all-targets`
- Focused query admission/analyzer tests: 2 passed
- `cargo nextest run --locked -p tests-integration --lib -E
'test(native_hybrid_binary_05_06_08)'`: passed, independently rerun on
final source
- `cargo clippy --locked -p query --all-targets --all-features -- -D
warnings`
- Independent source review approved.
Standalone integration asserts two independent region-local
Fused->SeriesScan branches, four nonempty partitions per input, frontend
join placement, native/legacy complete-row parity, unmatched-series
removal (05), outer scalar projection (06), and independent rate inputs
(08). Additional cases cover subquery legacy fallback, valid
reduced-label `on(job)` matching, frontend uniqueness Window plus
`prom_assert_unique_match_group` Filter above MergeScan, and meaningful
duplicate-match execution errors under both routes.
### Scope and limitations
Standalone single-region per table, multiple scan partitions. Arithmetic
and admitted one-to-one matching only. No distributed Join/Exchange,
cross-region routing, group_left/right, set operators, native subqueries
or multi-field expansion. No full workspace suite or performance
benchmark is claimed. Based directly on narthex, independent of the
global-TopK PR.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the matching labels.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## What's changed and what's your intention?
Enable vmbench-lite 09, `topk(10, sum by(hostname)(rate(...[5m])))`,
through the existing native aggregate-finalize path. Split TopK/BottomK
into the global suffix above fully merged aggregate values. Other vector
operations retain their existing admission behavior. No new physical
operator or persisted/wire format is introduced.
Physical shape: `SeriesScan(N) -> FusedSeries(N, partial sum) ->
merge/coalesce -> AggregateFinalize(global TopK) -> rows`. TopK is never
partition-local.
The standalone regression uses one region, four nonempty scan
partitions, 28 series and two evaluation timestamps with different
winning hosts. It compares complete native/legacy results and
independently expected values, checks producer/finalizer plans, and
records original partial-state streams to prove the same hostname
contributes from distinct partitions at both timestamps. Partition
output counts were [14,14,14,14].
### Verification
- `make fmt-check`
- `cargo check --locked -p promql -p tests-integration --all-targets`
- `cargo nextest run --locked -p promql -E 'test(aggregate_finalize)'`:
28 passed
- `cargo nextest run --locked -p tests-integration --lib -E
'test(native_aggregate_test)'`: 2 passed
- `cargo clippy --locked -p promql --all-targets --all-features -- -D
warnings`
- Independent source review approved; main-agent targeted execution
checks passed.
### Scope and limitations
Standalone single-region, multiple scan partitions only. No distributed
Exchange, cross-region merge/routing, or new matching/subquery support.
No full workspace suite or performance benchmark is claimed. This PR is
independent of the hybrid-binary PR and is based directly on narthex.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the matching labels.
Under `or` the right table's tags join the left side's columns, and one of
them can share a name with the left side's field or time index; building
the output schema then failed instead of leaving the query to the existing
plan. Such a union is now declined. The output-label walk also declines an
aggregate or whole-vector node instead of passing its input's labels
through, which would describe the wrong schema if one were ever reached.
Adds an end-to-end case over two tables whose tags, field and time index
are all named differently, partly flushed and partly in the memtable,
under both series scan modes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A binary or set operation between selectors of two tables, such as
`rate(a[5m]) / rate(b[5m])`, fell back to the existing plan: the native
route required every input to share one metric, time index and tag list,
and a fused node is placed inside the one region it reads.
Two tables are two regions, so such a pair now runs on the frontend. Each
input becomes a `RawSeriesSource` that rides in its own merge scan: the
region binds its scan to that table's metric, tags and fields, decodes the
labels and answers one explicit-label raw frame stream, reading its scan
partitions one after another so each series stays contiguous. The fused
node consumes those frames directly. Pairs on one table keep running in
the region; partitioned tables keep the existing plan.
A point-wise wrapper around an admitted pair, such as
`100 * (1 - a / b)`, is admitted too, on one table and across two. The
output labels are now derived from the whole tree with each input's own
tags, and the physical node takes them from the output schema instead of
narrowing by the root alone.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A range query `a or on(host) b` where one host's left series covers only
the middle steps: native must keep the left sample where it exists and
take the right sample with its own labels at the other steps, matching
the explicit rows and the legacy plan. The set test's row reader moves
to module level so both cases share it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The legacy planner refuses `and`, `or` and `unless` over several fields
(MultiFieldsNotSupported), and Prometheus has one value per sample, so
per-field presence has no reference semantics. The compiler now declines
a set operation unless each side has exactly one field, and the query
keeps the legacy plan and its error.
Errors raised by the shared two-operand runtime now name the set node
instead of calling every such node "the binary node".
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A two-scan FusedSeries over a table split into several regions has no
placement: DistPlannerAnalyzer rejects it with "only accept plan with at
most one child", so the query failed instead of falling back. This hit
the existing arithmetic pair and, since set operators now compile, set
operators too. Decline the native route for two selectors when the table
has partition columns.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Bind `and`, `or` and `unless` (with `on`/`ignoring`) to a new
`NativeExpr::Set` node instead of refusing them. The node runs the
existing MATCH-SET kernel through the binary runtime, which retains both
operands until the input ends, so a set operation reads every series of
both sides in one execution.
The set kernel now reports its workspace like the binary kernel does:
kept rows, the filter side's match groups, the sort and the merge are
charged to the node's reservation.
The compiler refuses a fill value on a set operator, which Prometheus
rejects and the parser accepts, and sides with different field counts.
Same-table set operations over two scans take the existing two-scan
native route.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The workspace denies the default Clippy warnings, and the new tests tripped
three of them without changing what any of them proves:
- an expected value was built through `scale as f64` even though the tuple
already carries an `f64`, so the conversion is spelled in the literal and the
cast is gone;
- the standalone replay printed the executed plan and the per-partition counts
even though the surrounding assertions already require four nonempty
partitions, 24 rows in total and eight closing frames, so the diagnostics
are gone rather than exempted from the workspace lint.
No production code, assertion or fixture changes.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Remove plan-global label-set occupancy and the speculative split-series tree guard. Keep the native evaluator, finality, memory budgets, errors and ordinary RecordBatch output local to each scan partition.
Narrow native admission to selectors and the small unary POC subset. Document that cross-partition final-label conflicts and semantic merging are outside this POC, and cover real rate/abs(rate) execution across dense and sparse storage spanning SST and memtable data.
Signed-off-by: discord9 <discord9@163.com>
discord9
dennis zhuang
discord9