Commit Graph
1019 Commits
Author SHA1 Message Date
Dennis Zhuang 02eb3aa60c chore: merge narthex into feat/narthex-cross-table
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 18:20:21 +08:00
discord9 b29a471575 fix(promql): repair native execution contracts and remove obsolete paths (#9980)
* fix(promql): restore native scan statistics and remove unused raw merger

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): repair native execution contracts and remove obsolete paths

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): leave fused frame output repairs to separate PR

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs: restore native contracts and regenerate config reference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: slice nullable sparse frame fields after span selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): intersect anchored timestamp selection windows

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): name anchored timestamp boundary case type

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): align partitioned pair checks with hybrid operands

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
2026-10-27 17:41:38 +08:00
Dennis Zhuang ba8156be47 fix(promql): keep the right side's labels for a filled left side across tables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 16:21:31 +08:00
Dennis Zhuang 3d5f99596c chore: merge narthex into feat/narthex-cross-table
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 15:47:23 +08:00
discord9 70638a769f feat(promql): add explicit local canonical merge operator (#9969)
* feat(promql): add explicit local canonical merge operator

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): tighten canonical merge accounting and validation

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): remove redundant merge tree whitelist

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): retain canonical merge output label reservations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): finish canonical merge review simplifications

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): pin the parent barrier over canonical merge

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): assert the two-selector binary stays unfused

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 15:15:20 +08:00
Dennis Zhuang 7bcdec1a83 chore: merge narthex into feat/narthex-cross-table
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

# Conflicts:
#	src/promql/src/extension_plan/fused_series.rs
#	src/promql/src/extension_plan/fused_series/tests.rs
2026-10-27 14:39:04 +08:00
Dennis Zhuang ed562fc905 feat(promql): run vector-vector comparisons natively
A one-to-one comparison between two selectors, such as `a > on(host) b`
or `a == bool b`, kept the existing plan: the pair admission accepted
only arithmetic operators. The kernel already pairs comparisons by the
same matching rules and only differs in the value a pair keeps (the
left sample when the comparison holds, or 1/0 with `bool`), so the
operator check is dropped. This covers one table and two, and the
point-wise wrappers around a pair, such as `(a > b) * 100`.

Adds an end-to-end case over two tables comparing explicit rows, the
legacy answer and the native placement for a filter and a `bool`
comparison.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 12:36:20 +08:00
discord9 48797ab132 fix(promql): delegate fused frame output to the shared packer (#9968)
Reuse pack_evaluated_series for fused Frame output, preserving the single native driver and PlainRows/partial timing. Reject mismatched logical frame schemas and use the reset scan plan in integration tests. Add evaluated-packer lifecycle and invalid-sequence regressions, and address the review follow-ups.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 11:27:04 +08:00
discord9 a9266b2da9 feat(promql): fuse independent operands of hybrid binary queries (#9973)
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).

## What's changed and what's your intention?

Enable vmbench-lite 05/06/08 cross-metric arithmetic using two
independently planned native operands and the existing DataFusion binary
join, PromQL matching and arithmetic projection.

Physical shape: `SeriesScan A(N) -> Fused A(N)` and `SeriesScan B(M) ->
Fused B(M)` execute region-side; separate MergeScan branches feed the
existing frontend Join. No native Join, second matching kernel or
Exchange is added; same-table fused admission is preserved.

A bounded analyzer pass locates each direct Fused(TableScan) through
Alias/Projection/Filter/Window wrappers, rewrites only the fused
subtree, and leaves all wrappers frontend-side. This preserves global
uniqueness checks. Subquery bodies disable native admission and restore
the prior flag before returning either success or error, avoiding use of
outer evaluation bounds.

### Verification
- `make fmt-check`
- `cargo check --locked -p query -p tests-integration --all-targets`
- Focused query admission/analyzer tests: 2 passed
- `cargo nextest run --locked -p tests-integration --lib -E
'test(native_hybrid_binary_05_06_08)'`: passed, independently rerun on
final source
- `cargo clippy --locked -p query --all-targets --all-features -- -D
warnings`
- Independent source review approved.

Standalone integration asserts two independent region-local
Fused->SeriesScan branches, four nonempty partitions per input, frontend
join placement, native/legacy complete-row parity, unmatched-series
removal (05), outer scalar projection (06), and independent rate inputs
(08). Additional cases cover subquery legacy fallback, valid
reduced-label `on(job)` matching, frontend uniqueness Window plus
`prom_assert_unique_match_group` Filter above MergeScan, and meaningful
duplicate-match execution errors under both routes.

### Scope and limitations
Standalone single-region per table, multiple scan partitions. Arithmetic
and admitted one-to-one matching only. No distributed Join/Exchange,
cross-region routing, group_left/right, set operators, native subqueries
or multi-field expansion. No full workspace suite or performance
benchmark is claimed. Based directly on narthex, independent of the
global-TopK PR.

## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the matching labels.
2026-10-27 11:26:20 +08:00
discord9 6ca3d32090 feat(promql): finalize topk over globally merged aggregates (#9972)
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).

## What's changed and what's your intention?

Enable vmbench-lite 09, `topk(10, sum by(hostname)(rate(...[5m])))`,
through the existing native aggregate-finalize path. Split TopK/BottomK
into the global suffix above fully merged aggregate values. Other vector
operations retain their existing admission behavior. No new physical
operator or persisted/wire format is introduced.

Physical shape: `SeriesScan(N) -> FusedSeries(N, partial sum) ->
merge/coalesce -> AggregateFinalize(global TopK) -> rows`. TopK is never
partition-local.

The standalone regression uses one region, four nonempty scan
partitions, 28 series and two evaluation timestamps with different
winning hosts. It compares complete native/legacy results and
independently expected values, checks producer/finalizer plans, and
records original partial-state streams to prove the same hostname
contributes from distinct partitions at both timestamps. Partition
output counts were [14,14,14,14].

### Verification
- `make fmt-check`
- `cargo check --locked -p promql -p tests-integration --all-targets`
- `cargo nextest run --locked -p promql -E 'test(aggregate_finalize)'`:
28 passed
- `cargo nextest run --locked -p tests-integration --lib -E
'test(native_aggregate_test)'`: 2 passed
- `cargo clippy --locked -p promql --all-targets --all-features -- -D
warnings`
- Independent source review approved; main-agent targeted execution
checks passed.

### Scope and limitations
Standalone single-region, multiple scan partitions only. No distributed
Exchange, cross-region merge/routing, or new matching/subquery support.
No full workspace suite or performance benchmark is claimed. This PR is
independent of the hybrid-binary PR and is based directly on narthex.

## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the matching labels.
2026-10-27 11:26:14 +08:00
Dennis Zhuang c9423f0193 fix(promql): decline cross-table outputs the native node cannot name
Under `or` the right table's tags join the left side's columns, and one of
them can share a name with the left side's field or time index; building
the output schema then failed instead of leaving the query to the existing
plan. Such a union is now declined. The output-label walk also declines an
aggregate or whole-vector node instead of passing its input's labels
through, which would describe the wrong schema if one were ever reached.

Adds an end-to-end case over two tables whose tags, field and time index
are all named differently, partly flushed and partly in the memtable,
under both series scan modes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 02:38:21 +08:00
Dennis Zhuang 4ce757b611 feat(promql): run cross-table selector pairs natively
A binary or set operation between selectors of two tables, such as
`rate(a[5m]) / rate(b[5m])`, fell back to the existing plan: the native
route required every input to share one metric, time index and tag list,
and a fused node is placed inside the one region it reads.

Two tables are two regions, so such a pair now runs on the frontend. Each
input becomes a `RawSeriesSource` that rides in its own merge scan: the
region binds its scan to that table's metric, tags and fields, decodes the
labels and answers one explicit-label raw frame stream, reading its scan
partitions one after another so each series stays contiguous. The fused
node consumes those frames directly. Pairs on one table keep running in
the region; partitioned tables keep the existing plan.

A point-wise wrapper around an admitted pair, such as
`100 * (1 - a / b)`, is admitted too, on one table and across two. The
output labels are now derived from the whole tree with each input's own
tags, and the physical node takes them from the output schema instead of
narrowing by the root alone.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 01:53:07 +08:00
discord9 6b7d1bca4c test(promql): strengthen hybrid binary review regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 01:03:31 +08:00
discord9 1c6c288936 test(promql): address global topk review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 00:54:51 +08:00
discord9 9270b1ef7c test(promql): narrow global topk partition recording
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 18:53:54 +08:00
Dennis Zhuang 78dadff86d test(promql): pin per-step or matching through the native entry
A range query `a or on(host) b` where one host's left series covers only
the middle steps: native must keep the left sample where it exists and
take the right sample with its own labels at the other steps, matching
the explicit rows and the legacy plan. The set test's row reader moves
to module level so both cases share it.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-26 17:44:31 +08:00
discord9 c25b6a9fbd feat(promql): fuse independent operands of hybrid binary queries
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 17:05:42 +08:00
Dennis Zhuang c24997e543 fix(promql): leave multi-field set operations to the legacy plan
The legacy planner refuses `and`, `or` and `unless` over several fields
(MultiFieldsNotSupported), and Prometheus has one value per sample, so
per-field presence has no reference semantics. The compiler now declines
a set operation unless each side has exactly one field, and the query
keeps the legacy plan and its error.

Errors raised by the shared two-operand runtime now name the set node
instead of calling every such node "the binary node".

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-26 16:55:15 +08:00
Dennis Zhuang cef469b450 fix(query): keep two-selector native plans off partitioned tables
A two-scan FusedSeries over a table split into several regions has no
placement: DistPlannerAnalyzer rejects it with "only accept plan with at
most one child", so the query failed instead of falling back. This hit
the existing arithmetic pair and, since set operators now compile, set
operators too. Decline the native route for two selectors when the table
has partition columns.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-26 16:28:53 +08:00
Dennis Zhuang 1be0df046c feat(promql): compile and run set operators natively
Bind `and`, `or` and `unless` (with `on`/`ignoring`) to a new
`NativeExpr::Set` node instead of refusing them. The node runs the
existing MATCH-SET kernel through the binary runtime, which retains both
operands until the input ends, so a set operation reads every series of
both sides in one execution.

The set kernel now reports its workspace like the binary kernel does:
kept rows, the filter side's match groups, the sort and the merge are
charged to the node's reservation.

The compiler refuses a fill value on a set operator, which Prometheus
rejects and the parser accepts, and sides with different field counts.
Same-table set operations over two scans take the existing two-scan
native route.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-26 16:23:05 +08:00
discord9 db5e31a005 feat(promql): finalize topk over globally merged aggregates
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 15:51:51 +08:00
discord9 f2041d8d9d test(promql): keep the frame output tests lint-clean
The workspace denies the default Clippy warnings, and the new tests tripped
three of them without changing what any of them proves:

- an expected value was built through `scale as f64` even though the tuple
  already carries an `f64`, so the conversion is spelled in the literal and the
  cast is gone;
- the standalone replay printed the executed plan and the per-partition counts
  even though the surrounding assertions already require four nonempty
  partitions, 24 rows in total and eight closing frames, so the diagnostics
  are gone rather than exempted from the workspace lint.

No production code, assertion or fixture changes.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 14:51:19 +08:00
discord9 949eeeff38 fix(promql): stream evaluated frames with batch-level backpressure
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 14:51:19 +08:00
discord9 24e214b7d4 feat(promql): bind instant anchors and execute native timestamps
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 11:25:43 +08:00
discord9 ab721b912b test(promql): quote region label in Mito fixture SQL
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:15:22 +08:00
discord9 9e0f74091b test(promql): scope partition proof to unfiltered scans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:15:22 +08:00
discord9 11287b8c71 test(promql): cover scalar aggregate composition and fallback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:15:22 +08:00
discord9 503c3ec48d test(promql): retain and verify empty vector result schemas
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 4da340262e test(promql): let query encoding use auto-deref
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 00338b5121 test(promql): import partition properties for output assertions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 12c7d0ad4f test(promql): prove each scalar-route partition produces output
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 04e8cfd2bb test(promql): disambiguate the frontend SQL test helper
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 6eff9f5804 test(promql): verify scalar routes and four-partition broadcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 73b9a77932 feat(promql): adapt complete aggregate wiring to recursive native runtime
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 21:25:05 +08:00
discord9 8dbf08d252 feat(promql): execute independent selector scans in fused binary trees
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 20:10:32 +08:00
discord9 222b1916ec test(promql): read bounded scan metrics through cooperative wrapper
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 14:46:45 +08:00
discord9 7e70b220cb test(promql): account for counter-zero range extrapolation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 14:12:58 +08:00
discord9 4748081579 feat(promql): fan out a shared raw scan to bound binary inputs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 13:47:42 +08:00
discord9 aa4a381f71 refactor(promql): keep fused POC execution partition-local
Remove plan-global label-set occupancy and the speculative split-series tree guard. Keep the native evaluator, finality, memory budgets, errors and ordinary RecordBatch output local to each scan partition.

Narrow native admission to selectors and the small unary POC subset. Document that cross-partition final-label conflicts and semantic merging are outside this POC, and cover real rate/abs(rate) execution across dense and sparse storage spanning SST and memtable data.

Signed-off-by: discord9 <discord9@163.com>
2026-10-25 03:33:49 +08:00
discord9 3b0587aa8d fix(mito2): read raw series when columnar indexes are not ready
Keep native queries available for unflushed memtables and SSTs with incomplete columnar series or samples coverage by selecting the ordinary raw-series reader. Preserve structural validation and propagate artifact read errors instead of retrying them through legacy execution.

Cover all four incomplete-readiness states in public PromQL queries, both storage scan modes and ordering, complete prepared-source execution, and artifact error propagation.

Signed-off-by: discord9 <discord9@163.com>
2026-10-24 23:01:49 +08:00
discord9 5ba2d76f5a test(query): cover fused Columnar Float64 execution
Signed-off-by: discord9 <discord9@163.com>
2026-10-24 21:05:18 +08:00
discord9 51f754356b test(query): iterate captured delta result batches
Signed-off-by: discord9 <discord9@163.com>
2026-10-24 19:05:28 +08:00
discord9 ebfbd2ae5d fix(query): preserve legacy semantics at native admission
Signed-off-by: discord9 <discord9@163.com>
2026-10-24 18:58:51 +08:00
discord9 bb5a7ee540 test(promql): clarify partition fixture and allow richer explain
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit ee5566d201774983e235716a67f2600b267c7c99)
2026-10-23 19:57:39 +08:00
discord9 ea060286cc test(promql): require real partition output and strict native results
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit a13b1a723e7839f5b4e799a4493a507369de8893)
2026-10-23 19:57:39 +08:00
discord9 980edef08d test(promql): observe actual fused region execution
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 313c8c181f3331bbaa67539a2739e26d6d0e3dec)
2026-10-23 19:57:39 +08:00
discord9 c1dcac0f2b test(promql): verify actual frontend results and view definition
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 64a52a2a87245f16c4f2c7be20ba275302e586df)
2026-10-23 19:57:39 +08:00
discord9 c48c5ec5e1 test(promql): execute native standalone queries and compare results
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 12ea89c6ca329a91b6b3ee50ff73c93ab6626dc1)
2026-10-23 19:57:39 +08:00
Yingwen ce4fc4529e feat(mito): read columnar series through prepared sources (#9892)
* feat(mito): read columnar series through prepared sources

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): rebuild schema-incompatible columnar sidecars

Signed-off-by: evenyag <realevenyag@gmail.com>

* docs(mito): regenerate series format configuration

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito): batch prepared series reads and preflight inputs

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-23 15:11:29 +08:00
discord9 0b412679c8 feat(query): add experimental_native_promql config option
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-17 23:20:32 +08:00