A payload the encoder accepts has to be one this version can decode. Three
places disagreed.
An `Input`'s field count was bounded only on the way in, so a 129-field
`Input → Instant` encoded to 983 bytes and then failed to decode. The
decoder's limit was `MAX_COUNT`, a constant that also bounds nesting depth
and `Map` operation count, which is why the missing half was easy to miss;
each limit now has its own name and both sides check it.
`FusedSeries::serialize` only checked that the tag count fit in a `u32`
while `deserialize` bounded it at 1024. Both sides now share `MAX_TAG_COUNT`.
`FusedSeries::deserialize` passed an empty schema to `try_new_with_format`,
which checks a frame node's declared schema against the layout derived from
its tree rather than overwriting it. Every serialized frame node was
therefore impossible to rebuild. The schema is derived from the decoded tree
now, which is where it comes from in the first place, so nothing is added to
the wire.
Encoding refuses an over-limit count instead of writing a payload it cannot
read back, and a field-count overflow no longer reports "operation count
exceeds maximum".
`rejects_excess_operation_count` hand-built a payload whose operation count
was read from the wrong offset: it placed an `Instant` where the decoder
expects the count, so it exercised the limit with 258 while claiming 129.
It now starts from a tree the codec accepts, asserts the bytes it edits are
the count it means, and checks the error names both the operation limit and
the value. The same is added for field count, tag count and nesting.
The integration test moves to its own file: it was written against
`tests-integration/tests/raw_series.rs`, which the raw-fan-in line creates
and this one does not carry.
Signed-off-by: discord9 <discord9@163.com>
* feat(promql): run time(), scalar(v), vector(s) and dynamic scalar broadcasts natively
Bind time() and zero-argument date functions as EvalTime, scalar(v) as ScalarOf and
vector(s) as VectorOf, and run binary operations whose scalar side varies per grid point:
the scalar is read whole, vector chunks that arrive before it are held and charged, and
both operand orders and bool comparisons follow Prometheus.
A tree holding scalar(v) runs as one execution over every scan partition, and the
planner admits it only over an unpartitioned Mito table. Data-free statements plan
without a scan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): hold the labels of chunks waiting for a varying scalar
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(promql): read round and clamp parameters from varying scalars natively
Bind round, clamp, clamp_min and clamp_max whose parameters are not all constants as a
ParameterizedMap with scalar parameter nodes, run through map_math_with_parameters. The
parameters are read whole first; vector chunks that arrive before them are held and
charged, through the same hold the varying-scalar broadcast now shares. Constant
parameters keep the MathOp map.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin a parameterized map's empty closing chunk
Also correct the held-chunk docs, use same-table examples for varying map parameters,
and assert the native decline instead of the existing planner's failure.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(query): plan vector(s) natively whatever the table's field is named
The fused route declined a root field that is not a raw field, which is every root over
vector(s): it names its result `value`. Take the compiled root's fields as they are and
decline only a name the output already gives a tag or the time index.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reserve a constant map parameter before building it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* refactor(promql): leave scalar(v) to the existing planner until it has its own execution boundary
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): leave data-free statements that generate labels to the existing plan
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): reserve the output point-wise kernels build before running them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(promql): run info() natively
info(v, {selector}) fell back to the existing plan, which has no info
and refuses the query ("Unsupported expr type: info", or "Cannot accept
multiple vector as function input" when a selector is given).
Compile it into the native tree and run it on the info kernel:
- Plan: NativeExpr::Info { base, info, selector }. The info side is an
instant selection reporting each selected sample's own time in
milliseconds (InstantValue::InfoTimestamp, which replaces the
BoundInstant.timestamp flag); validation refuses any other info side.
- Compiler: info_input derives the info selector the way Prometheus
does: target_info or one __name__="..." matcher, the data matchers,
and the @ and offset every selector of the first argument shares
(subquery offsets added until an @ anchors them, none when they
differ). Regex or negative name matchers, which can select several
tables, __field__/__schema__/__database__ matchers and multi-field
info tables are refused. The planner's selector walk uses the same
helper, so both bind the info table as the last input.
- Runtime: BinaryRuntime runs the kernel as PairKernel::Info, with the
retention and pool accounting of binary and set nodes; the kernel
reports its workspace through info_with_reservation.
- Planner: the info table is a second table, so it is read through its
own RawSeriesSource and the node runs on the frontend. The output adds
the info table's tags, or only the data labels the selector names.
Partitioned or non-Mito tables and a missing info table keep the
existing plan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin region-local placement of info over its own table
info(rate(target_info[5m])) reads the info table twice. Check that the
planner binds both as direct scans under one FusedSeries node bound to
the table, that the node is placed in the region, and the columns it
reports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep sample-less base series out of info's identifying patterns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): share identifying label strings in info() instead of copying them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* feat(promql): run native trees with more than two selectors
Lift the two-selector cap of the native PromQL route. A tree of one-to-one
vector matches and set operators, nested under point-wise wrappers, is now
admitted for any number of selectors: on one table the node runs in the
region over every scan, across tables it runs on the frontend over one
RawSeriesSource per selector. Partitioned tables, aggregates, whole-vector
operations and many-to-one matches still keep the existing plan.
The dist analyzer and commutativity checks accept a same-table node with
more than two scans; without that such a plan fails in the analyzer.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): cover nested set operators and metric-engine multi-selector decline
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): wait for aborted partition tasks to release their inputs
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 3f8699782c3116c4063ec70a3522d1f5f3bff897)
* docs(promql): say spawned partitions are released after an error, not before
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A one-to-one comparison between two selectors, such as `a > on(host) b`
or `a == bool b`, kept the existing plan: the pair admission accepted
only arithmetic operators. The kernel already pairs comparisons by the
same matching rules and only differs in the value a pair keeps (the
left sample when the comparison holds, or 1/0 with `bool`), so the
operator check is dropped. This covers one table and two, and the
point-wise wrappers around a pair, such as `(a > b) * 100`.
Adds an end-to-end case over two tables comparing explicit rows, the
legacy answer and the native placement for a filter and a `bool`
comparison.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Reuse pack_evaluated_series for fused Frame output, preserving the single native driver and PlainRows/partial timing. Reject mismatched logical frame schemas and use the reset scan plan in integration tests. Add evaluated-packer lifecycle and invalid-sequence regressions, and address the review follow-ups.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## What's changed and what's your intention?
Enable vmbench-lite 05/06/08 cross-metric arithmetic using two
independently planned native operands and the existing DataFusion binary
join, PromQL matching and arithmetic projection.
Physical shape: `SeriesScan A(N) -> Fused A(N)` and `SeriesScan B(M) ->
Fused B(M)` execute region-side; separate MergeScan branches feed the
existing frontend Join. No native Join, second matching kernel or
Exchange is added; same-table fused admission is preserved.
A bounded analyzer pass locates each direct Fused(TableScan) through
Alias/Projection/Filter/Window wrappers, rewrites only the fused
subtree, and leaves all wrappers frontend-side. This preserves global
uniqueness checks. Subquery bodies disable native admission and restore
the prior flag before returning either success or error, avoiding use of
outer evaluation bounds.
### Verification
- `make fmt-check`
- `cargo check --locked -p query -p tests-integration --all-targets`
- Focused query admission/analyzer tests: 2 passed
- `cargo nextest run --locked -p tests-integration --lib -E
'test(native_hybrid_binary_05_06_08)'`: passed, independently rerun on
final source
- `cargo clippy --locked -p query --all-targets --all-features -- -D
warnings`
- Independent source review approved.
Standalone integration asserts two independent region-local
Fused->SeriesScan branches, four nonempty partitions per input, frontend
join placement, native/legacy complete-row parity, unmatched-series
removal (05), outer scalar projection (06), and independent rate inputs
(08). Additional cases cover subquery legacy fallback, valid
reduced-label `on(job)` matching, frontend uniqueness Window plus
`prom_assert_unique_match_group` Filter above MergeScan, and meaningful
duplicate-match execution errors under both routes.
### Scope and limitations
Standalone single-region per table, multiple scan partitions. Arithmetic
and admitted one-to-one matching only. No distributed Join/Exchange,
cross-region routing, group_left/right, set operators, native subqueries
or multi-field expansion. No full workspace suite or performance
benchmark is claimed. Based directly on narthex, independent of the
global-TopK PR.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the matching labels.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## What's changed and what's your intention?
Enable vmbench-lite 09, `topk(10, sum by(hostname)(rate(...[5m])))`,
through the existing native aggregate-finalize path. Split TopK/BottomK
into the global suffix above fully merged aggregate values. Other vector
operations retain their existing admission behavior. No new physical
operator or persisted/wire format is introduced.
Physical shape: `SeriesScan(N) -> FusedSeries(N, partial sum) ->
merge/coalesce -> AggregateFinalize(global TopK) -> rows`. TopK is never
partition-local.
The standalone regression uses one region, four nonempty scan
partitions, 28 series and two evaluation timestamps with different
winning hosts. It compares complete native/legacy results and
independently expected values, checks producer/finalizer plans, and
records original partial-state streams to prove the same hostname
contributes from distinct partitions at both timestamps. Partition
output counts were [14,14,14,14].
### Verification
- `make fmt-check`
- `cargo check --locked -p promql -p tests-integration --all-targets`
- `cargo nextest run --locked -p promql -E 'test(aggregate_finalize)'`:
28 passed
- `cargo nextest run --locked -p tests-integration --lib -E
'test(native_aggregate_test)'`: 2 passed
- `cargo clippy --locked -p promql --all-targets --all-features -- -D
warnings`
- Independent source review approved; main-agent targeted execution
checks passed.
### Scope and limitations
Standalone single-region, multiple scan partitions only. No distributed
Exchange, cross-region merge/routing, or new matching/subquery support.
No full workspace suite or performance benchmark is claimed. This PR is
independent of the hybrid-binary PR and is based directly on narthex.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the matching labels.
Under `or` the right table's tags join the left side's columns, and one of
them can share a name with the left side's field or time index; building
the output schema then failed instead of leaving the query to the existing
plan. Such a union is now declined. The output-label walk also declines an
aggregate or whole-vector node instead of passing its input's labels
through, which would describe the wrong schema if one were ever reached.
Adds an end-to-end case over two tables whose tags, field and time index
are all named differently, partly flushed and partly in the memtable,
under both series scan modes.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A binary or set operation between selectors of two tables, such as
`rate(a[5m]) / rate(b[5m])`, fell back to the existing plan: the native
route required every input to share one metric, time index and tag list,
and a fused node is placed inside the one region it reads.
Two tables are two regions, so such a pair now runs on the frontend. Each
input becomes a `RawSeriesSource` that rides in its own merge scan: the
region binds its scan to that table's metric, tags and fields, decodes the
labels and answers one explicit-label raw frame stream, reading its scan
partitions one after another so each series stays contiguous. The fused
node consumes those frames directly. Pairs on one table keep running in
the region; partitioned tables keep the existing plan.
A point-wise wrapper around an admitted pair, such as
`100 * (1 - a / b)`, is admitted too, on one table and across two. The
output labels are now derived from the whole tree with each input's own
tags, and the physical node takes them from the output schema instead of
narrowing by the root alone.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A range query `a or on(host) b` where one host's left series covers only
the middle steps: native must keep the left sample where it exists and
take the right sample with its own labels at the other steps, matching
the explicit rows and the legacy plan. The set test's row reader moves
to module level so both cases share it.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The legacy planner refuses `and`, `or` and `unless` over several fields
(MultiFieldsNotSupported), and Prometheus has one value per sample, so
per-field presence has no reference semantics. The compiler now declines
a set operation unless each side has exactly one field, and the query
keeps the legacy plan and its error.
Errors raised by the shared two-operand runtime now name the set node
instead of calling every such node "the binary node".
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
A two-scan FusedSeries over a table split into several regions has no
placement: DistPlannerAnalyzer rejects it with "only accept plan with at
most one child", so the query failed instead of falling back. This hit
the existing arithmetic pair and, since set operators now compile, set
operators too. Decline the native route for two selectors when the table
has partition columns.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Bind `and`, `or` and `unless` (with `on`/`ignoring`) to a new
`NativeExpr::Set` node instead of refusing them. The node runs the
existing MATCH-SET kernel through the binary runtime, which retains both
operands until the input ends, so a set operation reads every series of
both sides in one execution.
The set kernel now reports its workspace like the binary kernel does:
kept rows, the filter side's match groups, the sort and the merge are
charged to the node's reservation.
The compiler refuses a fill value on a set operator, which Prometheus
rejects and the parser accepts, and sides with different field counts.
Same-table set operations over two scans take the existing two-scan
native route.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The workspace denies the default Clippy warnings, and the new tests tripped
three of them without changing what any of them proves:
- an expected value was built through `scale as f64` even though the tuple
already carries an `f64`, so the conversion is spelled in the literal and the
cast is gone;
- the standalone replay printed the executed plan and the per-partition counts
even though the surrounding assertions already require four nonempty
partitions, 24 rows in total and eight closing frames, so the diagnostics
are gone rather than exempted from the workspace lint.
No production code, assertion or fixture changes.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Remove plan-global label-set occupancy and the speculative split-series tree guard. Keep the native evaluator, finality, memory budgets, errors and ordinary RecordBatch output local to each scan partition.
Narrow native admission to selectors and the small unary POC subset. Document that cross-partition final-label conflicts and semantic merging are outside this POC, and cover real rate/abs(rate) execution across dense and sparse storage spanning SST and memtable data.
Signed-off-by: discord9 <discord9@163.com>
Keep native queries available for unflushed memtables and SSTs with incomplete columnar series or samples coverage by selecting the ordinary raw-series reader. Preserve structural validation and propagate artifact read errors instead of retrying them through legacy execution.
Cover all four incomplete-readiness states in public PromQL queries, both storage scan modes and ordering, complete prepared-source execution, and artifact error propagation.
Signed-off-by: discord9 <discord9@163.com>
discord9
dennis zhuang
discord9