I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Spec: Narthex Unified SeriesFrame refactor, PR A of two.
- #9914, #9868, #9937, #9896, #9933, #9936, #9882 and #9886 are all
merged, so this is a plain increment on
`narthex` with no stacked dependency.
## What's changed and what's your intention?
SeriesFrame is the internal transport of the Narthex native PromQL path,
and it could only carry
raw samples: an evaluated result had to fall back to a plain row table.
This makes **one
self-describing Arrow format carry both**, so a receiver decodes a frame
from the frame alone,
without the scanner, region, or planning context its producer had.
What is unified is the **physical representation and its codec**, not
the runtime data structures.
`RawSeriesChunk`, `Series` and `EvaluatedChunk` are unchanged; only the
adapter boundaries branch
on the sample shape.
### Base
- base: `narthex` @ `771347df5d9`
- HEAD: `8bc53c48714`
- **no stacked dependency**: everything this PR built on has landed
This is a plain increment on the current `narthex` tip, with focused
follow-up fixes for review
and CI. Nothing from the earlier stack is carried: #9914 is merged, so
no `MetricName` rename appears here. The codec itself, the
adapters, the tests and the two frozen wire fixtures are all that moved,
plus the call sites that
had to follow the renamed API.
### The format (`layout = v2`)
One frame family, three legal combinations. Every other combination is
refused on both the writing
and the reading side:
| content | identity | time | producer |
| --- | --- | --- | --- |
| `raw` | `key` | `timestamps` | a storage raw-series scan |
| `raw` | `explicit` | `timestamps` | a raw input whose labels are
already decoded |
| `evaluated` | `explicit` | `grid_spans` | a compute node's output |
`evaluated` + `key` is refused because a computation may rewrite its
labels, so it cannot claim a
storage key as its identity. `raw` + `grid_spans` and `evaluated` +
`timestamps` are refused because
raw samples have no grid and computed values have no timestamps of their
own.
Column layout (`series_frame_schema` in
`src/datatypes/src/series/frame.rs`):
```text
read_key: Binary (NOT NULL for key mode; NULLABLE for explicit mode)
is_last: Boolean NOT NULL
time: Struct<timestamps: List<Int64>> // raw
Struct<spans: List<Struct<start: u64, len: u64>>> // evaluated
fields: Struct<<field_name>: SamplePayload, ...>
name: Utf8 NULLABLE // explicit only, __name__
labels: List<Struct<name, value>> // explicit only, no __name__, no duplicates
```
`read_key` must be populated for every raw row, including raw/explicit,
and must be NULL for
evaluated rows. `SamplePayload` (float / native histogram / mixed) keeps
its existing representation
and is not redesigned.
Protocol metadata lives under `greptime.series.*`.
`SeriesFrameDeclaration::parse` reads the
layout, identity, time and ordering declaration. Schemas explicitly
declare `contiguous` by default;
raw streams may declare `read_key_then_time`, while evaluated streams
accept only `contiguous`.
Missing ordering, unknown values and unknown reserved keys are refused
by declaration parsing,
direct decoding (including empty batches), and stream admission.
| key | value |
| --- | --- |
| `layout` | `v2` |
| `content` | `raw` / `evaluated` |
| `labels` | `key` / `explicit` |
| `time` | `timestamps` / `grid_spans` |
| `grid` | required for evaluated, forbidden for raw; canonical
`GridSpec` JSON |
| `label_decode` | required for key, forbidden for explicit |
| `ordering` | `contiguous` / `read_key_then_time` |
### Time axis
- raw: every row carries its own strictly increasing `i64` millisecond
timestamps.
- evaluated: the `GridSpec` is declared once in schema metadata
(`{start_ms, step_ms, len}`), and a row carries only the **absolute grid
positions** as spans.
A sparse grid is never expanded and its holes are never filled; `start +
len` is checked against
the grid with checked arithmetic.
- A zero-sample closing chunk is legal for evaluated and **is kept** as
one frame row with zero
positions and `is_last = true`, because dropping it would drop the
finality it exists to carry.
A zero-sample chunk that closes nothing is refused, and a raw frame
still refuses an empty chunk.
### Series chains and `is_last`
`is_last` closes **one chunk chain**, not the end of a PromQL label set.
After a native label
rewrite, two different input series can produce **the same complete
labels**, and each closes
itself. The writer allows that and the reader does not treat "same
labels" as a re-opened series.
A chain that changes identity before closing, and EOF with an open
chain, are both errors.
### Key code locations
- `src/datatypes/src/series/frame.rs`: shared schema, declaration
parser, bounded builder,
cursor and raw/evaluated decoder.
- `src/common/query/src/series_frame.rs`: typed pack/unpack adapters and
chain validators.
- `src/datatypes/src/series/time.rs`: canonical `GridSpec`
serialization.
### Review and CI fixes
- Repaired the two table PK-binding fixtures with real `host` and `rack`
Tag columns and matching
primary-key metadata. Production `LabelDecodeSpec::validate()` stays
strict. The mismatch test
retains all six cases, including changed tag order, and checks rejection
before polling.
- Removed the missing-ordering fallback and added one private ordering
validator shared by the
declaration parser and direct decoder. Invalid evaluated ordering cannot
bypass stream checks.
- Compatibility metadata is tested exactly as written, without merging
canonical metadata into
malformed inputs. The direct-decoder matrix covers raw/evaluated,
empty/nonempty, missing,
unknown and interleaved ordering, and valid raw versus invalid evaluated
`read_key_then_time`.
- Independent source and anti-over-design review: **APPROVE** for
`771347df5d9..8bc53c48714`. Review approval is separate from the CI
status of the new head.
### Limitations
- Not reachable from a query yet: `FusedSeriesExec` still answers
`PlainRows` by default. The
second output format is PR B.
- Not verified: Windows; the results below are Linux.
- Deliberately **not** done, per the spec: evaluated **input** wiring in
`FusedSeriesExec`,
Complete Aggregate's SeriesFrame input, Partial/Final physical plans,
cross-region exchange,
remote NativeTree serialization, NodePath rework, #9892
storage-interface changes, #9844 remote
read wiring.
- The unreleased v1 frame format is **not** read. In-tree producers,
consumers, tests and the two
frozen wire fixtures all move to v2 together.
### Does this break API or data compatibility?
No published format changes. SST, WAL, manifest and the generic
Flight/Protobuf formats are
untouched. The only versioned thing here is the unreleased internal
SeriesFrame layout, which goes
from `v1` to `v2` in one step and is not read in both directions.
### Tests
The existing tests include real pack → batch stream → unpack round
trips, not only schema
snapshots:
- raw/key with its decoded PK, raw/explicit with complete labels, both
across batches and through
Arrow IPC
- evaluated dense, sparse (multiple spans, holes not filled),
multi-field (a position can hold a
value in one field and nothing in another; NULL ≠ NaN), native histogram
and mixed payloads
(payload kind and values survive, no silent float-ification)
- evaluated empty closing chunk kept as one row; multi-batch streams
keeping order and finality;
two consecutive chains with equal labels each closing independently; EOF
with an open chain
- malformed frames refused: non-increasing timestamps,
overlapping/out-of-order/out-of-bounds/zero
length spans, payload length mismatch, span-count overflow, missing
grid, malformed `GridSpec`,
missing label decode, contradictory metadata, wrong layout version, an
evaluated row carrying a
storage key, an empty row that closes nothing
<!-- CURRENT_VERIFICATION_START -->
Verified on Linux at `8bc53c48714`; every command below exited 0:
```text
make fmt-check
-> clean (rustfmt and repository source checks)
cargo nextest run -p datatypes -p common-query -p table --retries 0
-> 690 passed, 0 skipped; includes both reported CI failures
cargo nextest run -p mito-codec -p promql -p query --retries 0
-> 2055 passed, 2 skipped
cargo nextest run -p mito2 -E 'test(engine::scan_test::test_mito_handle_query_raw_series) | test(engine::scan_test::test_two_phase_native_raw_stream) | test(engine::scan_test::test_two_phase_raw_) | test(engine::scan_test::test_mito_raw_series_)' --retries 0
-> 8 passed, 1733 skipped (focused raw-scan selection)
cargo check --workspace --all-targets
-> passed
cargo clippy -p datatypes -p common-query -p table -p mito-codec -p promql -p query --all-targets -- -D warnings
-> passed (scoped crates, default features)
```
License headers, TOML formatting and `git diff --check` also pass. These
are local Linux
results, not a claim that the new-head GitHub checks are green. Full
`make test`, workspace
all-features Clippy and Windows execution were not run locally in this
round.
<!-- CURRENT_VERIFICATION_END -->
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 176f6f6479)
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Test-only follow-up to #9882. Merge #9882 into `narthex` first, then
merge this PR. The PR base remains `feat/narthex-instant-at` until its
parent lands.
## What's changed and what's your intention?
Add regression coverage for the existing native multi-field
`timestamp()` refusal. The net change against the refreshed #9882 is
exactly **one file, `src/query/src/promql/native_compile/tests.rs`, with
43 added lines and no production changes**.
- `timestamp(x)` over two fields reaches the raw-selector guard;
`timestamp(abs(x))` reaches the evaluated-vector guard. Both must return
`NotImplemented` naming `timestamp() over a multi-value input` as the
reason.
- The same two expressions over one field must compile. These positive
controls establish that the refusal is about field count rather than
blanket rejection of timestamp expressions.
- A small local multi-field schema fixture leaves the existing
single-field fixture unchanged.
The old branch history was not mechanically replayed: only this final
test delta is retained atop the new parent. Production refusal, public
single-field native timestamp admission and the paired public-planner
multi-field fallback test belong to #9882. This PR does not implement
multi-field timestamp results or add fallback machinery.
### Verification
Verified clean final HEAD
**`9a702209747646a68a67fc09a3d44a3c30cfc5cf`**, directly atop #9882
**`24e214b7d4b3d277ce04a7e40d3aaf3d7673c70a`** (rebased onto
`narthex=aa30968d43b3`, including #9936).
- `make fmt-check` and the exact-parent `git diff --check`: passed.
- `cargo check --locked -p query -p promql --all-targets`: passed.
- `cargo nextest run --locked -p query -p promql --retries 0
--no-fail-fast`: **1990 passed, 2 skipped**.
- `cargo clippy --locked -p query -p promql --all-targets --all-features
-- -D warnings`: passed.
Old `06695dcf4a5` CI results are not attributed to this rewritten HEAD.
The refreshed Checks, Rust CI and Integration CI results for this exact
immutable HEAD are required before merge; this PR still follows #9882.
The 43 added test lines and their context were compared with the
original `3e80f822595..93ad42b76d5` test-only delta. They are identical;
only Git blob IDs and hunk line numbers move because #9936 changed the
surrounding baseline. Parent standalone Native/Complete Aggregate
verification is reported in #9882 rather than attributed to this
test-only change.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
This test-only change alters no API, schema, stored data or wire format,
and requires no release backport.
Refreshed CI runs for this published immutable HEAD are pending:
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266409294),
[Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266409205),
[Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266409248).
No merge is performed until these new-HEAD checks pass.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Follows the merged multi-input wiring in #9896, Complete Aggregate in
#9933, and zero-input Scalar/literal broadcasts in #9936. Merge this PR
before its test-only follow-up #9886.
## What's changed and what's your intention?
Connect instant-selector `@` and `timestamp()` to the standalone native
PromQL planner and recursive executor on current `narthex`.
- Supports fixed `@ <timestamp>`, `@ start()`, `@ end()` and
positive/negative offsets. The selector reads at `anchor - offset`, with
the existing lookback and staleness rules, and replays its answer on the
statement's evaluation grid. Its scan bounds follow the anchor even when
the selected data predates the statement.
- `timestamp(raw_selector)` reports the selected source sample's
original timestamp in seconds. `timestamp(abs(raw_selector))` and other
supported evaluated-vector arguments report evaluation-grid seconds only
where the argument has a value. These remain distinct
compilation/runtime paths; an offset changes which source sample is
chosen, not the timestamp projected from that sample.
- Adds `timestamp` to public native planner admission for supported
single-field expressions. This is no longer only a compiler/executor API
capability.
- Retains recursive RuntimeNode ownership, InputId routing, per-node
metrics, SeriesLabels and the current frame decoder. The evaluated
Timestamp node reuses existing NodeState accounting and lifecycle, with
projection workspace reserved before allocation.
### Integration with Complete Aggregate
The migration combines Timestamp recursion with the latest scalar-aware
`reads_across_series` handling. AggregateFinalize receives only the
necessary Timestamp traversal arms and test-fixture updates. Existing
`partial_fold`, Complete Aggregate admission, scalar-binary
prefixes/suffixes, `FusedSeries.children: Vec<LogicalPlan>` and
`BoundInput.source_index` are preserved; no old planner/compiler files
or flat NodePath runtime are restored.
This remains a single-region standalone experimental Native PoC.
Multi-field `timestamp()` still declines native compilation and keeps
the existing planner. Timestamp/aggregate shapes outside the current
split grammar likewise retain the existing plan. This PR does not add
general fallback machinery, cross-table routing, Join, Exchange or
distributed execution. `timestamp(x @ t offset d)` preserves
GreptimeDB's existing selection semantics rather than introducing
Prometheus' special-case offset behavior.
### Synchronization with merged Scalar roots
Rebased the reviewed B-only change onto `narthex` at
`aa30968d43b3bf9b5c335c6ac3a9f38be58d653b` (#9936 merged). This is a
single B implementation commit directly on that baseline, followed
separately by #9886's test-only commit. The only textual conflict was
adjacent added planner tests; both timestamp tests and the
constant-root/broadcast test remain intact. No production merge fix or
fallback expansion was needed. The zero-child Scalar route, literal
broadcasting, compiler/admission guards, recursive runtime accounting
and Complete Aggregate remain present.
### Verification
Verified clean final HEAD
**`24e214b7d4b3d277ce04a7e40d3aaf3d7673c70a`**, rebased directly onto
`narthex` **`aa30968d43b3bf9b5c335c6ac3a9f38be58d653b`**.
- `make fmt-check` and `git diff aa30968d43b3..HEAD --check`: passed.
- `cargo check --locked -p query -p promql --all-targets` and the
separate `-p tests-integration --all-targets` check: passed.
- `cargo nextest run --locked -p query -p promql --retries 0
--no-fail-fast`: **1989 passed, 2 skipped**.
- `cargo nextest run --locked -p tests-integration --lib --retries 0
--no-fail-fast -E 'test(native_promql_) |
test(native_aggregate_complete_over_four_nonempty_mito_partitions)'`:
**17 passed**. This includes the instant/timestamp storage query,
retained native routes and Complete Aggregate over four nonempty Mito
partitions, including #9936's scalar broadcasts.
- `cargo nextest run --locked -p tests-integration --test main --retries
0 --no-fail-fast -E
'test(integration_http_file_test::test_prometheus_native_promql_route) |
test(integration_http_file_test::test_native_promql_route_plans_the_query)'`:
**2 passed**, preserving #9936's File-backed HTTP native and planning
routes.
- `cargo clippy --locked -p query -p promql --all-targets --all-features
-- -D warnings` and the separate `-p tests-integration` invocation:
passed.
- The added `native_instant_at.rs` was copied byte-for-byte for focused
`hawkeye format` followed by `hawkeye check`, using repository inception
year 2023; both passed and the copy remained identical.
Built fresh `greptime` and `sqlness-runner` binaries from this exact
HEAD, then ran isolated standalone SQLness compatibility cases
`at_modifier`, `fused_multi_standalone` and `scalar_entry`: **all 3
passed**. Expected `.result` files remained byte-identical; the runner
stopped the temporary instance and removed its state. These are
compatibility checks, not substitutes for the Native execution tests
above. GitHub CI results for the old `d09e4fdc057` HEAD are not reused:
refreshed Checks, Rust CI and Integration CI must complete successfully
on the new immutable HEAD before merge.
The new actual standalone test checks independent expected timestamps,
values and host labels, then compares with the legacy route. Its cases
cover fixed/start/end anchors, pre-statement anchors, signed offsets,
the exclusive lookback boundary and exact zero-lookback selection,
missing values, exact-bit stale markers, and raw-sample versus
evaluation-grid timestamps. For a bare selector whose newest source
value is NULL, the existing Arrow representations differ: Native emits
no row, whereas legacy keeps a selected NULL-valued row; the test
asserts both representations explicitly, including the legacy
timestamp/label, without filtering results or changing production
semantics. It collects the output, observes `FusedSeriesExec` and checks
native output metrics, rather than treating a logical plan or old CI run
as execution evidence.
The existing ordinary single-input/two-independent-scan tests and
four-nonempty-partition Complete Aggregate test are retained as separate
regressions. The two scans are still the #9896 same-table capability;
this PR does not claim true cross-table dual input.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
The native tree additions change an internal experimental Rust
interface; no persisted metadata or wire format is changed. No release
backport is requested.
Refreshed CI runs for this published immutable HEAD are pending:
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266408649),
[Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266408660),
[Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266408658).
No merge is performed until these new-HEAD checks pass.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Stacked on #9868 (`feat/narthex-fused-wiring`, currently at
`990d932198ab0c5e41b724a54d990e5a8ffc8acb`). This change needs that PR's
unmerged `FusedSeries`/`SeriesScan`/in-process native route, so the base
is its branch; GitHub will retarget to `narthex` once #9868 merges.
## What's changed and what's your intention?
Enable the existing opt-in native PromQL route for literal vector/scalar
arithmetic such as `x * 100`, unary negation such as `-x`, and constant
scalar statements such as `1 + 2`. These public entries return ordinary
nonempty query RecordBatches with the same values and output schema as
the existing planner.
- Traverse unary and binary expression operands when locating the one
selector and its range requirements. Constant-only statements reuse the
existing constant evaluator and statement grid; unsupported shapes still
use the existing planner.
- Execute constant Scalar roots as one unlabelled evaluated chunk on the
full statement grid, with existing memory admission, output metrics and
error/cleanup paths. A no-input tree has genuinely zero logical/physical
children rather than a discarded placeholder scan.
- Keep each raw input partition attached to its own `FusedSeriesExec`
evaluator. This change adds neither an input-side Coalesce nor a
SeriesFrame exchange. Ordinary output collection continues to use
DataFusion.
- Cover scalar NaN/negative-zero/EOF/memory boundaries, child arity and
output schemas, independent lazy input progress, and the actual public
HTTP/query-handler route. The standalone single-region fixture persists
four disjoint input batches: for both broadcast and unary queries, all
four input/native partitions perform work and the complete result is
exactly 24 labelled timestamp/value rows. A four-stream barrier test
collected by DataFusion verifies concurrent progress; this is not a
CPU-overlap or speedup benchmark.
- Add a generated SQLness compatibility case for scalar arithmetic,
broadcast and unary results, rerun successfully without result changes.
Scope: constant scalar roots and literal broadcasts only.
`scalar(vector)` adaptation, dynamic per-grid scalar
arithmetic/broadcast, global Vector/Select and Complete Aggregate wiring
are separate work and are not enabled here. The experimental
configuration/request gates and current unsupported-shape fallback
remain in place. No persisted or remote wire format changes are
introduced; internal Fused constructor callers are migrated together.
Verification on this exact HEAD
(`d9eb9778f113dd9bd6125305a9ea447b4aa6ab1d`, restacked onto the current
dependency
`990d932198ab0c5e41b724a54d990e5a8ffc8acb`), exit 0:
```text
cargo fmt --all -- --check
git diff --check
cargo check --locked -p promql -p query --tests
cargo check --locked -p tests-integration --lib --test main
cargo nextest run --locked -p promql -p query --retries 0
1870 passed, 2 skipped
cargo nextest run --locked -p tests-integration --test main --retries 0 \
-E 'test(=integration_http_file_test::test_native_promql_route_plans_the_query) | test(=integration_http_file_test::test_prometheus_native_promql_route)'
2 passed, 770 filtered out (four real input partitions, exact 24 values)
cargo nextest run --locked -p tests-integration --lib --retries 0 -E 'test(native_promql_)'
4 passed, 251 filtered out (upstream VIEW/metric/ANALYZE compatibility)
cargo clippy --locked -p promql -p query --all-targets -- -D warnings
cargo clippy --locked -p tests-integration --lib --test main -- -D warnings
```
SQLness (generated, then rerun with an unchanged result):
```text
/mnt/nvme_rust/rust-targets-2/d-scalar-entry/debug/sqlness-runner bare \
--bins-dir .../debug --case-dir <isolated root> --test-filter '^standalone:scalar_entry$' --jobs 1
```
This SQLness case uses the default public compatibility route, not the
native request flag; it does not replace the native execution assertions
above. Full workspace `make test`/`make clippy`/udeps and external CI
have not been run for this change.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Rebased onto `narthex` at `bd6be2cf8f83b04280d3113467563ae81005e2f3`,
including merged #9868, #9937, #9914, #9870, #9871 and #9896. This PR no
longer depends on the old #9896 feature branch.
- Builds on the existing native Complete aggregate runtime from #9873.
#9936 can be restacked after this PR merges.
## What's changed and what's your intention?
Connect Complete aggregation to the current experimental native PromQL
execution path while preserving actual partition-local evaluation:
```text
SeriesScanExec N → FusedSeriesExec N → complete partial accumulator states N
→ CoalescePartitionsExec → FloatMerger → AggregateFinalizeExec 1 → ordinary RecordBatch
```
### Execution and current API adaptations
- Keep #9896's `FusedSeries.children`, physical children,
`BoundInput.source_index`, and existing ordinary multi-input execution.
An optional `partial_fold` changes only the output mode: ordinary
executions still emit query rows; aggregate producers fold evaluated
chunks into complete accumulator state and export it only after
successful input EOF. The aggregate path currently has one actual scan
child, with a separate evaluator for each of its N partitions; no gather
is inserted before evaluation.
- Split at the innermost supported Complete aggregate. The existing
`FloatMerger` combines full state, not finalized partition answers:
average retains count and compensated sum, and variance retains count,
mean and M2. The finalizer waits for all producer input to end
successfully before running the global suffix or returning rows. A late
source error fails the query rather than returning an incomplete
aggregate.
- Build the suffix as a separate `NativeTree`. Its `source_index = 0`
names its own evaluated input, not the original scan. Obtain the actual
input from `NativeExecutionState::input_ids(0)` and call
`NativeExecutor::push_evaluated(InputId, chunk)`; do not equate a source
index with a runtime input ID. No `NodePath` or flat executor is
restored.
- Use current `SeriesLabels` and the raw field layout from Native Input
output samples, without restoring a separate scan-schema mirror. Reuse
the existing row converter for final output. State and output
reservations are execution-local and released on errors and stream drop,
including when a physical plan is retained for another execution.
- Preserve `sum`, `avg`, `count`, `min`, `max`, `group`, `stddev` and
`stdvar`, `by` / `without`, maps and nested aggregate suffixes. Literal
scalar broadcasts work below the cut (`sum(8 * rate(...))`, `max(10 -
rate(...))`) and above it (`100 - avg(...)`, `sum(...) * 2`, `100 /
sum(...)`, `-sum(...)`), keeping subtraction and division operand order.
- Compose the current selector matcher/time-bound filters with aggregate
planning. Add `sum(rate(counter{job="a"}[5m]))` to the existing
standalone fixture; the independently expected answer is 9.
**The private state-row `RecordBatch` codec remains a temporary
bridge.** Producer, codec and finalizer retain the explanation that
future native final aggregation should consume series frames directly,
eliminating the intermediate state-row encoding, decoding and
reconstruction. This PR does not implement that replacement.
### Scope and compatibility
The supported target is standalone, one Region, multiple nonempty scan
partitions, with in-process region transport and the existing
engine/per-query native opt-ins. Ordinary Fused input traversal is not
redesigned. There is no cross-Region series merge, generic Exchange,
multi-input Aggregate Join, distributed Partial/Final architecture, new
public configuration, persisted format or wire format. No performance or
CPU-overlap improvement is claimed. Unsupported aggregate topologies
remain outside this PoC.
### Verification
All commands below passed on exact head
`9720e1bf88a776bd017b304152a055e1735d4ba1`, over the narthex base above,
using `RUST_MIN_STACK=8388608`, a dedicated external Cargo target, and
no test retries:
- `cargo fmt -p promql -p query -p tests-integration -- --check`
- `cargo check --locked -p query -p promql --all-targets`
- `cargo check --locked -p tests-integration --all-targets`
- `cargo nextest run --locked -p query -p promql --no-fail-fast
--retries 0` — **1967 passed, 2 skipped**.
- `cargo nextest run --locked -p tests-integration --lib --no-fail-fast
--retries 0 -E
'test(native_aggregate_complete_over_four_nonempty_mito_partitions) |
test(native_promql_)'` — **16 passed, 251 skipped**.
- `cargo clippy --locked -p query -p promql --all-targets --all-features
-- -D warnings`
- `cargo clippy --locked -p tests-integration --all-targets
--all-features -- -D warnings`
- `git diff --check bd6be2cf8f83b04280d3113467563ae81005e2f3..HEAD`
- `hawkeye format --dry-run` and `hawkeye check`
The standalone aggregate case checks typed physical-plan structure as
well as results: `SeriesScanExec` and partial Fused each retain four
partitions, with no intervening exchange; the finalizer consumes the
producer's state schema and answers one partition. For unfiltered
counter cases all four evaluators must report positive contributions,
totaling five and at most two per partition, forcing the three-member
group across partitions. Results are checked for labels, timestamps,
columns, values and multiplicity against independent expectations, in
addition to legacy execution.
The five nonzero counter rates are 1/3/5/2/4: total sum 15; per-job sums
9/6, means 3/3, counts 3/2 and standard deviations sqrt(8/3)/1. Scalar
cases include `sum(8 * rate(...)) = 120`, `max(10 - rate(...)) = 9`,
`100 - avg(...) = 97`, `100 / sum(...) = 100/15` and unary minus = -15.
The subtraction prefix distinguishes an incorrectly moved suffix (5) or
reversed operands (-5). The matcher selects three series, so only that
case does not require four nonempty partitions; all unfiltered
assertions remain intact.
Unit coverage includes all eight accumulator kinds, unequal producer
counts (one value versus two, so the merged mean is 6 rather than the
mean-of-means 5), nested/map/scalar suffixes, empty input, producer read
barriers, late errors, and reservation/drop boundaries. Local
full-workspace tests, an HTTP smoke run and native SQLness have not been
run; the SQLness runner does not carry the per-query native hint.
Final-head CI for `9720e1bf88a776bd017b304152a055e1735d4ba1` has
completed: [Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361500),
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361419),
[Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361402)
all succeeded. The final check rollup contains 40 successful checks and
2 skipped checks, with no pending or failed checks. An earlier Semantic
Pull Request run was cancelled and superseded by a successful run; the
current Semantic PR title check passed. No workflow reruns were
requested.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Uses the selector requirements and independent binary compilation
already merged in #9870 and #9871. This PR now targets `narthex`
directly.
## What's changed and what's your intention?
Run a compiled vector–vector expression from the real PromQL entry point
with **two independent scans**, including different matchers and time
bounds, instead of folding both operands onto one scan.
- `FusedSeries` keeps `Vec<LogicalPlan>` children in source-index order.
Each `BoundInput` names its child through `source_index`; its raw schema
remains in the compiled tree rather than being duplicated in the
binding. Logical identity includes source indices and ordered fields.
- The planner prepares each selector in its own context, preserves its
matcher filters, and adds its own compiler-derived scan bounds,
including range lower-edge semantics, `@` anchors and offsets. The two
scan children and source indices 0/1 remain distinct even for repeated
selectors.
- The physical adapter keeps the existing vector of `(physical plan,
label source, ordering)` tuples. Each child's frames are decoded and
labels materialized independently against that source's schema. A chunk
goes only to the runtime input IDs allocated for its source. Input ends
globally only after every source and partition has reached EOF;
errors/drop release queued inputs and evaluator reservations.
- Full raw tags remain available to bind scanners; one-to-one binary
output projects labels according to its matching rule, so `on(host)`
emits `host`, not an unrelated raw `kind` tag.
- A narrow same-table pair can use the existing in-process region-local
placement exception. Only that admitted root pair bypasses the unary
column-requirement rewriter; the general multi-input and wire-encoding
restrictions remain. This does not establish cross-region matching.
### Partitions and scope
A single physical input retains **Scan N → Fused N**. With two inputs,
**each scan retains N partitions**, while Fused reports one output
partition and consumes each source's partitions sequentially with a
separate decoder per partition. It does not pair left partition `i` with
right partition `i`, and the gathered stream is treated as `Contiguous`,
not globally `KeyThenTime`.
This is an experimental, opt-in **standalone, single-region PoC** for
ordinary Mito tables, millisecond timestamps, `Float64` fields, and
same-table one-to-one arithmetic pairs with default matching or nonempty
`on(...)`. It does not add cross-region routing, co-location, semantic
series merging, Exchange, global label sets, group matching, or a
general fallback-proof system. Metric-table pairs, comparisons, group
matching, empty `on()`, aggregates and subqueries are outside this pair
admission; existing single-selector admission is preserved.
The experimental internal Rust plan interface changes from a single
child to vectors; persisted schemas and wire formats do not change. No
release backport or official documentation-site change is required for
this default-off experimental extension.
### Verification
Completed local checks:
- `make fmt-check` and source-only `git diff --check`: passed. The
sqlness runner-owned `.result` EOF blank line is excluded from the
source whitespace check, not manually trimmed.
- `cargo check -p promql -p query --all-targets`: passed.
- `cargo nextest run -p promql -p query --retries 0 --no-fail-fast`:
**1,897 passed, 2 skipped**.
- `cargo clippy -p promql -p query --all-targets --all-features -- -D
warnings`: passed.
- `cargo check -p tests-integration --all-targets`: passed.
- `cargo nextest run -p tests-integration --lib --retries 0
--no-fail-fast -E 'test(native_promql)'`: **15 passed, 251 filtered
out**.
- `cargo clippy -p tests-integration --all-targets --all-features -- -D
warnings`: passed.
The direct integration regression reads 4,608 hosts from two distinct
`kind` matchers. It verifies the entire instant subtraction result
(`-915` per host) and independent anchored/offset rate result (`-3` per
host) against explicit numeric expectations and the legacy planner. Both
real scan children have four partitions and each produces rows on
multiple partitions; Fused has one output partition. Source indices,
host-only output labels, and each window's raw input-sample count are
asserted. Separate adapter tests cover distinct raw schemas, unequal
partition counts, later keys in earlier partitions, empty inputs,
ordering refusal and failure/drop cleanup. Single-child shared-input
tests retain Scan N → Fused N and once-per-partition scan execution.
- `cargo build -p cmd --bin greptime -p sqlness-runner`: passed; the SQL
run used that freshly built binary.
- `sqlness-runner bare -c "$CASE_DIR" -t fused_multi_standalone
--bins-dir "$CARGO_TARGET_DIR/debug" -j1` (`CASE_DIR` selects only the
checked-in standalone fixture): regenerated `.result` through the
runner, then reran successfully with an unchanged artifact hash. All
**13 TQL queries / 35 result rows** were checked independently for
values, timestamps and labels, including matcher subtraction `-45/-90`
and independently anchored rate subtraction `-9/-18`. This SQL case is a
public-semantics regression, not evidence of native routing or partition
activity; the direct integration test provides that evidence.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Depends on the selector-binding PR #9870 (base branch
`feat/narthex-selector-bindings`); the
multi-scan execution wiring follows in #9896, based on this branch.
## What's changed and what's your intention?
Compile vector–vector arithmetic and comparison expressions into the
existing native `Binary`
runtime, with one independently bound input per selector occurrence.
- Replace the base's vector–vector refusal with the branch that compiles
both operands and binds the
matching: `on`/`ignoring`, `group_left`/`group_right`, `bool`, and the
`fill`/`fill_left`/
`fill_right` values, all through the existing matching types and
`binary_output_schema`.
- The base's scalar–vector path is untouched: `broadcast` and
`scalar_matching` still handle an
operation with one constant side, and a modifier written there is still
refused rather than
dropped.
- Set operators stay refused, as do vector–vector operations in the
tree-only `compile` entry, so a
caller cannot silently lose which scan feeds which selector;
`compile_selectors` is the entry for
multi-input expressions.
- Add ten real compiler-to-executor tests
(`src/query/tests/native_multi_selector.rs`) for arithmetic,
`on`/`ignoring`, grouping and fill, comparisons including output labels
and positions, distinct
fields/windows over the same metric, nested inputs, and a historical
range not clipped to the
statement start.
The tests supply fixture scans from the compiled requirements and feed
the returned executor paths.
Production Fused/scan-planner wiring and distributed placement are not
in this PR.
Validation on `f71f3bb4a94`:
- `cargo nextest run -p promql -p query`: 1814 passed, 2 skipped.
- `cargo nextest run -p query --test native_multi_selector`: 10 passed.
- `cargo nextest run -p query --lib promql::native_compile`: 97 passed.
- `cargo clippy -p query --all-targets --all-features -- -D warnings`:
passed.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
Signed-off-by: discord9 <discord9@163.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
First of three; vector–vector lowering (#9871) follows on this input
contract, and the multi-scan
execution wiring (#9896) follows that. Base is the fused wiring branch,
so the three form one
reviewable stack.
## What's changed and what's your intention?
Expose selector-specific scan requirements and executor input bindings
from the native PromQL
compiler without expanding its supported expression set.
- Add `compile_selectors`, whose callback receives metric, matchers,
resolved offset/anchor, actual
evaluation endpoints, and instant lookback or range-window duration, and
returns the selected raw
schema.
- Return each requirement and schema with its ordering and executor
`NodePath`, obtained through the
existing tree walker.
- Provide checked bounding scan intervals. Window and positive-lookback
lower edges are excluded;
zero-lookback instant selection includes the timestamp. The interval may
contain gaps not used by
individual evaluation windows.
- An anchored selector takes **both** edges from its anchor, not just
the upper one. Taking the lower
edge from the grid would scan from the statement's start, which is the
wrong side of the data for
an anchor before the statement and skips the selected sample for one
after it. Covered by
`an_anchored_instant_selector_scans_from_its_anchor`, which pins all
three cases: anchored inside
the statement, anchored before it, and a zero lookback.
- Preserve the existing `compile` API and selector/range/math behavior,
including the base's
scalar–vector broadcasting (`broadcast` / `scalar_matching`). A
vector–vector operation stays
refused here, naming the missing per-input identity.
Tests cover the callback requirement, schema identity in the actual
input node, nested input paths,
callback errors, offset/anchor bounds, anchored instant bounds, an
unaligned query end, lookback
boundaries, and overflow.
Validation on `d984231b6cb`:
- `cargo nextest run -p promql -p query`: 1800 passed, 2 skipped.
- `cargo nextest run -p query --lib promql::native_compile`: 93 passed.
- `cargo clippy -p query --all-targets --all-features -- -D warnings`:
passed.
No persisted or wire-format changes.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
Signed-off-by: discord9 <discord9@163.com>
discord9
discord9