mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
dd79052c7fd502d9e143fbaa264bb030d6c3ea6f
6881
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dennis zhuang
|
dd79052c7f |
fix(mito2): stop the series distributor from stalling on unread partitions (#10003)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Cherry-pick of #9991 (merged into `main`) onto `narthex`. ## What's changed and what's your intention? `FusedSeriesExec` with two inputs reads every scan partition one after another. On `narthex` this hits the series distributor's round-robin `send_timeout(100us)`. Tokio timers tick at 1 ms, so after a merge (one batch per series) each series batch costs about 1 ms per unpolled partition. With 16 partitions, two-selector native queries over regions with several SSTs spent ~470 ms to 12.7 s in `SeriesScanExec`, while legacy took ~10 ms. #9991 waits on all senders at once and sends to the first free slot. The cherry-pick applies cleanly and the diff is identical to #9991. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 discord9
|
70638a769f |
feat(promql): add explicit local canonical merge operator (#9969)
* feat(promql): add explicit local canonical merge operator Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): tighten canonical merge accounting and validation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): remove redundant merge tree whitelist Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): retain canonical merge output label reservations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): finish canonical merge review simplifications Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the parent barrier over canonical merge Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): assert the two-selector binary stays unfused Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
1dbc2c15ce |
fix(mito2): stop the series distributor from stalling on unread partitions (#9991)
When every partition channel is full, the distributor walked the senders
round-robin with a 100us send timeout each. Tokio timers fire on a 1ms
tick, so a consumer that leaves some partitions unpolled costs about one
millisecond per unpolled partition for every series batch.
SeriesScan::build_stream and scan_all_partitions drain partitions one
after another, and SortPreservingMergeExec above a multi-partition
SeriesScan only pulls the partition holding the current minimum key.
With 16 partitions and one batch per series after a merge, a 40-series
scan read partition by partition took ~480ms (measured on the native
PromQL two-input route).
Wait on all senders at once and send to the first one that frees a slot.
The send-timeout metric goes away with the timeout.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
discord9
|
48797ab132 |
fix(promql): delegate fused frame output to the shared packer (#9968)
Reuse pack_evaluated_series for fused Frame output, preserving the single native driver and PlainRows/partial timing. Reject mismatched logical frame schemas and use the reset scan plan in integration tests. Add evaluated-packer lifecycle and invalid-sequence regressions, and address the review follow-ups. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
a9266b2da9 |
feat(promql): fuse independent operands of hybrid binary queries (#9973)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## What's changed and what's your intention? Enable vmbench-lite 05/06/08 cross-metric arithmetic using two independently planned native operands and the existing DataFusion binary join, PromQL matching and arithmetic projection. Physical shape: `SeriesScan A(N) -> Fused A(N)` and `SeriesScan B(M) -> Fused B(M)` execute region-side; separate MergeScan branches feed the existing frontend Join. No native Join, second matching kernel or Exchange is added; same-table fused admission is preserved. A bounded analyzer pass locates each direct Fused(TableScan) through Alias/Projection/Filter/Window wrappers, rewrites only the fused subtree, and leaves all wrappers frontend-side. This preserves global uniqueness checks. Subquery bodies disable native admission and restore the prior flag before returning either success or error, avoiding use of outer evaluation bounds. ### Verification - `make fmt-check` - `cargo check --locked -p query -p tests-integration --all-targets` - Focused query admission/analyzer tests: 2 passed - `cargo nextest run --locked -p tests-integration --lib -E 'test(native_hybrid_binary_05_06_08)'`: passed, independently rerun on final source - `cargo clippy --locked -p query --all-targets --all-features -- -D warnings` - Independent source review approved. Standalone integration asserts two independent region-local Fused->SeriesScan branches, four nonempty partitions per input, frontend join placement, native/legacy complete-row parity, unmatched-series removal (05), outer scalar projection (06), and independent rate inputs (08). Additional cases cover subquery legacy fallback, valid reduced-label `on(job)` matching, frontend uniqueness Window plus `prom_assert_unique_match_group` Filter above MergeScan, and meaningful duplicate-match execution errors under both routes. ### Scope and limitations Standalone single-region per table, multiple scan partitions. Arithmetic and admitted one-to-one matching only. No distributed Join/Exchange, cross-region routing, group_left/right, set operators, native subqueries or multi-field expansion. No full workspace suite or performance benchmark is claimed. Based directly on narthex, independent of the global-TopK PR. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the matching labels. |
||
|
discord9
|
6ca3d32090 |
feat(promql): finalize topk over globally merged aggregates (#9972)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## What's changed and what's your intention? Enable vmbench-lite 09, `topk(10, sum by(hostname)(rate(...[5m])))`, through the existing native aggregate-finalize path. Split TopK/BottomK into the global suffix above fully merged aggregate values. Other vector operations retain their existing admission behavior. No new physical operator or persisted/wire format is introduced. Physical shape: `SeriesScan(N) -> FusedSeries(N, partial sum) -> merge/coalesce -> AggregateFinalize(global TopK) -> rows`. TopK is never partition-local. The standalone regression uses one region, four nonempty scan partitions, 28 series and two evaluation timestamps with different winning hosts. It compares complete native/legacy results and independently expected values, checks producer/finalizer plans, and records original partial-state streams to prove the same hostname contributes from distinct partitions at both timestamps. Partition output counts were [14,14,14,14]. ### Verification - `make fmt-check` - `cargo check --locked -p promql -p tests-integration --all-targets` - `cargo nextest run --locked -p promql -E 'test(aggregate_finalize)'`: 28 passed - `cargo nextest run --locked -p tests-integration --lib -E 'test(native_aggregate_test)'`: 2 passed - `cargo clippy --locked -p promql --all-targets --all-features -- -D warnings` - Independent source review approved; main-agent targeted execution checks passed. ### Scope and limitations Standalone single-region, multiple scan partitions only. No distributed Exchange, cross-region merge/routing, or new matching/subquery support. No full workspace suite or performance benchmark is claimed. This PR is independent of the hybrid-binary PR and is based directly on narthex. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the matching labels. |
||
 discord9
|
6b7d1bca4c |
test(promql): strengthen hybrid binary review regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1c6c288936 |
test(promql): address global topk review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
dennis zhuang
|
45852dbf67 |
feat(promql): compile and run set operators natively (#9970)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring (#25): set operators. The kernel came in #9679; the two-input route comes from #9870 / #9871 / #9896. ## What's changed and what's your intention? `and`, `or` and `unless`, with `on(...)` / `ignoring(...)`, now compile into the native tree and run natively. Before, the compiler refused them and the query fell back to the legacy plan. - **Plan.** New `NativeExpr::Set { left, right, op: BoundSetOp { op, labels } }`. There is no cardinality, `bool` or fill field, because Prometheus rejects those modifiers on set operators. Validation requires two evaluated vectors on the same grid; the output takes the left side's fields. - **Runtime.** `BinaryRuntime` runs either kernel (`PairKernel::{Binary, Set}`), so a set node gets the same operand retention, assembly, reservation, queueing and contiguity checks as vector–vector binary. Errors name the set node. - **Kernel.** `set_vectors` charged no memory. `set_vectors_with_reservation` now charges it the way the binary kernel does: output schema, filter-side group table, kept rows, the sort phase, error text and the merge. Membership is decided from one presence bitmap per filter match group (one bit per grid point), so each kept sample is an O(1) lookup. Before, a coarse `on()` scanned every filter candidate at every kept point, O(K·P·F). Results are unchanged; the Prometheus golden set cases still pass. - **Compiler.** `set_operation` binds the operator and the matching labels, and nests on either side and above. It refuses a fill value: promql-parser accepts one, but Prometheus rejects "filling in missing series not allowed for set operators". It also refuses any side that is not a single field. Legacy refuses multi-field set operations (both sides for `or`, the left side for `and` / `unless`), and a per-field presence rule is not defined by Prometheus, so those queries keep the legacy plan. - **Planner.** Same-table set operations take the existing two-scan route, where `FusedSeriesExec` reads every partition of both scans into one execution, so a set operation sees every series of both sides. - **Partitioned tables.** A two-selector native plan on a table with partition columns failed in `DistPlannerAnalyzer` ("only accept plan with at most one child") instead of falling back. This already happened for arithmetic pairs such as `x{kind="a"} - on(host) x{kind="b"}`, and set operators would have hit it too. The planner now declines two-selector plans on such tables, so both keep the legacy plan. Not covered yet: set operations across two tables (`rate(x[5m]) and on(job) up`), and those with aggregate operands (`sum by (job)(a) or sum by (job)(b)`). Both still use the legacy plan. The presence bitmaps take `groups × grid points / 8` bytes. That can exceed the filter samples only when groups are very sparse (under about 1/64 occupancy); such a query is refused by the memory pool rather than running over budget. Allocating each bitmap over its group's position span would tighten this, and is left for later. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
f882c1182d |
refactor(promql): reuse prepared native operand context
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
9270b1ef7c |
test(promql): narrow global topk partition recording
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dennis Zhuang
|
78dadff86d |
test(promql): pin per-step or matching through the native entry
A range query `a or on(host) b` where one host's left series covers only the middle steps: native must keep the left sample where it exists and take the right sample with its own labels at the other steps, matching the explicit rows and the legacy plan. The set test's row reader moves to module level so both cases share it. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
7f545344c4 |
fix(promql): hold the labels of queued results until they are handed on
A node that reads its whole input (binary, set, vector) queues the series its kernel produced and settles its reservation on the queue. Each queued result was held for `evaluated_bytes`, which covers the chunk and its fields but not the label set the chunk owns, so once the operands were released the reservation dropped below what the queue actually held. A set over many long-labelled one-point series made the gap large. Hold each result that does not share the input for `series_bytes`, the complete owned estimate including labels. The figure stored with the entry is still the one released when it is handed on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
c25b6a9fbd |
feat(promql): fuse independent operands of hybrid binary queries
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dennis Zhuang
|
9d281b401f |
perf(promql): decide set membership from a per-group presence bitmap
The set kernel checked every filter series of a kept series' match group at every kept point, so a coarse `on()` / `on(...)` cost O(kept series x points x group size). Prometheus decides a step from the set of signatures present on the filter side. Build one bitmap per filter match group, one bit per grid point, from the filter side's rows, and look each kept sample up in O(1). Each bitmap is charged to the walk phase when its group is first met and is released with the group table before the sort phase. Results are unchanged. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
c24997e543 |
fix(promql): leave multi-field set operations to the legacy plan
The legacy planner refuses `and`, `or` and `unless` over several fields (MultiFieldsNotSupported), and Prometheus has one value per sample, so per-field presence has no reference semantics. The compiler now declines a set operation unless each side has exactly one field, and the query keeps the legacy plan and its error. Errors raised by the shared two-operand runtime now name the set node instead of calling every such node "the binary node". Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
cef469b450 |
fix(query): keep two-selector native plans off partitioned tables
A two-scan FusedSeries over a table split into several regions has no placement: DistPlannerAnalyzer rejects it with "only accept plan with at most one child", so the query failed instead of falling back. This hit the existing arithmetic pair and, since set operators now compile, set operators too. Decline the native route for two selectors when the table has partition columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
1be0df046c |
feat(promql): compile and run set operators natively
Bind `and`, `or` and `unless` (with `on`/`ignoring`) to a new `NativeExpr::Set` node instead of refusing them. The node runs the existing MATCH-SET kernel through the binary runtime, which retains both operands until the input ends, so a set operation reads every series of both sides in one execution. The set kernel now reports its workspace like the binary kernel does: kept rows, the filter side's match groups, the sort and the merge are charged to the node's reservation. The compiler refuses a fill value on a set operator, which Prometheus rejects and the parser accepts, and sides with different field counts. Same-table set operations over two scans take the existing two-scan native route. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
db5e31a005 |
feat(promql): finalize topk over globally merged aggregates
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1cab5f1624 |
feat(promql): add a series frame output to the fused native node (#9943)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - **Stacked on #9942** (PR A, the unified SeriesFrame codec). - GitHub cannot target a fork branch, so this PR's base reads `narthex`. **Review #9942 first.** - The commits unique to this PR are `b32573f9cbf`, `16b3e428951`, `d5547c55bdf` and `f2041d8d9de` (test-only lint cleanup). The chain is `narthex ( |
||
|
discord9
|
f2041d8d9d |
test(promql): keep the frame output tests lint-clean
The workspace denies the default Clippy warnings, and the new tests tripped three of them without changing what any of them proves: - an expected value was built through `scale as f64` even though the tuple already carries an `f64`, so the conversion is spelled in the literal and the cast is gone; - the standalone replay printed the executed plan and the per-partition counts even though the surrounding assertions already require four nonempty partitions, 24 rows in total and eight closing frames, so the diagnostics are gone rather than exempted from the workspace lint. No production code, assertion or fixture changes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
949eeeff38 |
fix(promql): stream evaluated frames with batch-level backpressure
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
e3c6193e57 |
test(promql): pin the frame output's boundaries and its pending hand-over
The frame output was covered for its happy path and for the plan-level decisions, but three things the spec asks about had no test of their own at this boundary, and one test did not prove what it claimed. The pending test is the one that mattered. It awaited the stream, and `StagedStream` wakes itself right after reporting pending, so the await ran straight through the boundary: every later assertion held whether or not the buffered rows were handed over at the pending poll. It now polls that single poll by hand and requires it to answer the rows it already encoded. Deleting the `flush_frames` call from the pending branch makes it fail, which it did not before. The three boundaries now have their own tests: a chain that closes with no position survives as one row where the row table answers nothing at all; a sparse series carries exactly the positions the row table answers and no filled hole; and two chains that close on one label set are both kept, with values that differ so a lost chain would show. Also drops the `as_str` helper, which only asked whether to show a format and then spelled the same string twice, and corrects two comments: `convert` produces three outputs rather than two, and a built plan is not a promise that execution cannot fail. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
2f173c58bd |
feat(promql): add a series frame output to the fused native node
The fused node answers one way: a row table with the grid timestamps, one column per series field and one per declared tag. That is what a query returns, but a consumer of the native path needs to tell series and positions apart, and the row shape flattens both away - which series a value belongs to, and which grid positions that series actually holds. This adds a second output format, fixed when the plan is built rather than chosen at run time: PlainRows the existing row table, unchanged and still the only one a query reaches SeriesFrame the same evaluated chunks handed over as the unified series frame The format is part of the plan because it decides the node's output schema, so it takes part in the node's equality, its EXPLAIN text and its rebuild over new children. A row-table plan prints exactly what it printed before; a frame plan says so on its own line. A frame is encoded straight from each evaluated chunk, never by flattening it into rows and packing it again, so a sparse series keeps its runs and a closing chunk with no position is still one row. The output keeps one partition per input partition: a frame is not a reason to coalesce, and no exchange is inserted in either format. The partial-state output that a complete aggregate reads is a third thing again, and the format does not reach it. When a node exports partition-local state, the format is `PlainRows` by construction, `convert` hands every chunk to the fold before any format is consulted, and the EXPLAIN line keeps its `partial state:` suffix. Nothing in the aggregate path is rewritten to make room for frames. A tree whose root is not evaluated samples is refused while the plan is built, because aggregate state is a contribution to a later merge and a scalar has no series to name. So is a frame output over more than one raw source, which this node does not run in parallel yet. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
eeca48bc43 |
refactor(series): unify raw and evaluated SeriesFrame codec (#9942)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Spec: Narthex Unified SeriesFrame refactor, PR A of two. - #9914, #9868, #9937, #9896, #9933, #9936, #9882 and #9886 are all merged, so this is a plain increment on `narthex` with no stacked dependency. ## What's changed and what's your intention? SeriesFrame is the internal transport of the Narthex native PromQL path, and it could only carry raw samples: an evaluated result had to fall back to a plain row table. This makes **one self-describing Arrow format carry both**, so a receiver decodes a frame from the frame alone, without the scanner, region, or planning context its producer had. What is unified is the **physical representation and its codec**, not the runtime data structures. `RawSeriesChunk`, `Series` and `EvaluatedChunk` are unchanged; only the adapter boundaries branch on the sample shape. ### Base - base: `narthex` @ `771347df5d9` - HEAD: `8bc53c48714` - **no stacked dependency**: everything this PR built on has landed This is a plain increment on the current `narthex` tip, with focused follow-up fixes for review and CI. Nothing from the earlier stack is carried: #9914 is merged, so no `MetricName` rename appears here. The codec itself, the adapters, the tests and the two frozen wire fixtures are all that moved, plus the call sites that had to follow the renamed API. ### The format (`layout = v2`) One frame family, three legal combinations. Every other combination is refused on both the writing and the reading side: | content | identity | time | producer | | --- | --- | --- | --- | | `raw` | `key` | `timestamps` | a storage raw-series scan | | `raw` | `explicit` | `timestamps` | a raw input whose labels are already decoded | | `evaluated` | `explicit` | `grid_spans` | a compute node's output | `evaluated` + `key` is refused because a computation may rewrite its labels, so it cannot claim a storage key as its identity. `raw` + `grid_spans` and `evaluated` + `timestamps` are refused because raw samples have no grid and computed values have no timestamps of their own. Column layout (`series_frame_schema` in `src/datatypes/src/series/frame.rs`): ```text read_key: Binary (NOT NULL for key mode; NULLABLE for explicit mode) is_last: Boolean NOT NULL time: Struct<timestamps: List<Int64>> // raw Struct<spans: List<Struct<start: u64, len: u64>>> // evaluated fields: Struct<<field_name>: SamplePayload, ...> name: Utf8 NULLABLE // explicit only, __name__ labels: List<Struct<name, value>> // explicit only, no __name__, no duplicates ``` `read_key` must be populated for every raw row, including raw/explicit, and must be NULL for evaluated rows. `SamplePayload` (float / native histogram / mixed) keeps its existing representation and is not redesigned. Protocol metadata lives under `greptime.series.*`. `SeriesFrameDeclaration::parse` reads the layout, identity, time and ordering declaration. Schemas explicitly declare `contiguous` by default; raw streams may declare `read_key_then_time`, while evaluated streams accept only `contiguous`. Missing ordering, unknown values and unknown reserved keys are refused by declaration parsing, direct decoding (including empty batches), and stream admission. | key | value | | --- | --- | | `layout` | `v2` | | `content` | `raw` / `evaluated` | | `labels` | `key` / `explicit` | | `time` | `timestamps` / `grid_spans` | | `grid` | required for evaluated, forbidden for raw; canonical `GridSpec` JSON | | `label_decode` | required for key, forbidden for explicit | | `ordering` | `contiguous` / `read_key_then_time` | ### Time axis - raw: every row carries its own strictly increasing `i64` millisecond timestamps. - evaluated: the `GridSpec` is declared once in schema metadata (`{start_ms, step_ms, len}`), and a row carries only the **absolute grid positions** as spans. A sparse grid is never expanded and its holes are never filled; `start + len` is checked against the grid with checked arithmetic. - A zero-sample closing chunk is legal for evaluated and **is kept** as one frame row with zero positions and `is_last = true`, because dropping it would drop the finality it exists to carry. A zero-sample chunk that closes nothing is refused, and a raw frame still refuses an empty chunk. ### Series chains and `is_last` `is_last` closes **one chunk chain**, not the end of a PromQL label set. After a native label rewrite, two different input series can produce **the same complete labels**, and each closes itself. The writer allows that and the reader does not treat "same labels" as a re-opened series. A chain that changes identity before closing, and EOF with an open chain, are both errors. ### Key code locations - `src/datatypes/src/series/frame.rs`: shared schema, declaration parser, bounded builder, cursor and raw/evaluated decoder. - `src/common/query/src/series_frame.rs`: typed pack/unpack adapters and chain validators. - `src/datatypes/src/series/time.rs`: canonical `GridSpec` serialization. ### Review and CI fixes - Repaired the two table PK-binding fixtures with real `host` and `rack` Tag columns and matching primary-key metadata. Production `LabelDecodeSpec::validate()` stays strict. The mismatch test retains all six cases, including changed tag order, and checks rejection before polling. - Removed the missing-ordering fallback and added one private ordering validator shared by the declaration parser and direct decoder. Invalid evaluated ordering cannot bypass stream checks. - Compatibility metadata is tested exactly as written, without merging canonical metadata into malformed inputs. The direct-decoder matrix covers raw/evaluated, empty/nonempty, missing, unknown and interleaved ordering, and valid raw versus invalid evaluated `read_key_then_time`. - Independent source and anti-over-design review: **APPROVE** for `771347df5d9..8bc53c48714`. Review approval is separate from the CI status of the new head. ### Limitations - Not reachable from a query yet: `FusedSeriesExec` still answers `PlainRows` by default. The second output format is PR B. - Not verified: Windows; the results below are Linux. - Deliberately **not** done, per the spec: evaluated **input** wiring in `FusedSeriesExec`, Complete Aggregate's SeriesFrame input, Partial/Final physical plans, cross-region exchange, remote NativeTree serialization, NodePath rework, #9892 storage-interface changes, #9844 remote read wiring. - The unreleased v1 frame format is **not** read. In-tree producers, consumers, tests and the two frozen wire fixtures all move to v2 together. ### Does this break API or data compatibility? No published format changes. SST, WAL, manifest and the generic Flight/Protobuf formats are untouched. The only versioned thing here is the unreleased internal SeriesFrame layout, which goes from `v1` to `v2` in one step and is not read in both directions. ### Tests The existing tests include real pack → batch stream → unpack round trips, not only schema snapshots: - raw/key with its decoded PK, raw/explicit with complete labels, both across batches and through Arrow IPC - evaluated dense, sparse (multiple spans, holes not filled), multi-field (a position can hold a value in one field and nothing in another; NULL ≠ NaN), native histogram and mixed payloads (payload kind and values survive, no silent float-ification) - evaluated empty closing chunk kept as one row; multi-batch streams keeping order and finality; two consecutive chains with equal labels each closing independently; EOF with an open chain - malformed frames refused: non-increasing timestamps, overlapping/out-of-order/out-of-bounds/zero length spans, payload length mismatch, span-count overflow, missing grid, malformed `GridSpec`, missing label decode, contradictory metadata, wrong layout version, an evaluated row carrying a storage key, an empty row that closes nothing <!-- CURRENT_VERIFICATION_START --> Verified on Linux at `8bc53c48714`; every command below exited 0: ```text make fmt-check -> clean (rustfmt and repository source checks) cargo nextest run -p datatypes -p common-query -p table --retries 0 -> 690 passed, 0 skipped; includes both reported CI failures cargo nextest run -p mito-codec -p promql -p query --retries 0 -> 2055 passed, 2 skipped cargo nextest run -p mito2 -E 'test(engine::scan_test::test_mito_handle_query_raw_series) | test(engine::scan_test::test_two_phase_native_raw_stream) | test(engine::scan_test::test_two_phase_raw_) | test(engine::scan_test::test_mito_raw_series_)' --retries 0 -> 8 passed, 1733 skipped (focused raw-scan selection) cargo check --workspace --all-targets -> passed cargo clippy -p datatypes -p common-query -p table -p mito-codec -p promql -p query --all-targets -- -D warnings -> passed (scoped crates, default features) ``` License headers, TOML formatting and `git diff --check` also pass. These are local Linux results, not a claim that the new-head GitHub checks are green. Full `make test`, workspace all-features Clippy and Windows execution were not run locally in this round. <!-- CURRENT_VERIFICATION_END --> ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 discord9andWeny Xu
|
23dc866868 |
fix(test): port MinIO conditional-write test fix to narthex (#9930)
fix(test): avoid MinIO connection reuse after conditional writes (#9877) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |
||
|
discord9
|
8bc53c4871 |
test(series): remove unused decoder fixture binding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
2b67d0d60c |
test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
62652ec62e |
fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
b20e22b426 |
refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
discord9
|
771347df5d |
test(promql): cover the multi-value timestamp refusal (#9886)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Test-only follow-up to #9882. Merge #9882 into `narthex` first, then merge this PR. The PR base remains `feat/narthex-instant-at` until its parent lands. ## What's changed and what's your intention? Add regression coverage for the existing native multi-field `timestamp()` refusal. The net change against the refreshed #9882 is exactly **one file, `src/query/src/promql/native_compile/tests.rs`, with 43 added lines and no production changes**. - `timestamp(x)` over two fields reaches the raw-selector guard; `timestamp(abs(x))` reaches the evaluated-vector guard. Both must return `NotImplemented` naming `timestamp() over a multi-value input` as the reason. - The same two expressions over one field must compile. These positive controls establish that the refusal is about field count rather than blanket rejection of timestamp expressions. - A small local multi-field schema fixture leaves the existing single-field fixture unchanged. The old branch history was not mechanically replayed: only this final test delta is retained atop the new parent. Production refusal, public single-field native timestamp admission and the paired public-planner multi-field fallback test belong to #9882. This PR does not implement multi-field timestamp results or add fallback machinery. ### Verification Verified clean final HEAD **`9a702209747646a68a67fc09a3d44a3c30cfc5cf`**, directly atop #9882 **`24e214b7d4b3d277ce04a7e40d3aaf3d7673c70a`** (rebased onto `narthex=aa30968d43b3`, including #9936). - `make fmt-check` and the exact-parent `git diff --check`: passed. - `cargo check --locked -p query -p promql --all-targets`: passed. - `cargo nextest run --locked -p query -p promql --retries 0 --no-fail-fast`: **1990 passed, 2 skipped**. - `cargo clippy --locked -p query -p promql --all-targets --all-features -- -D warnings`: passed. Old `06695dcf4a5` CI results are not attributed to this rewritten HEAD. The refreshed Checks, Rust CI and Integration CI results for this exact immutable HEAD are required before merge; this PR still follows #9882. The 43 added test lines and their context were compared with the original `3e80f822595..93ad42b76d5` test-only delta. They are identical; only Git blob IDs and hunk line numbers move because #9936 changed the surrounding baseline. Parent standalone Native/Complete Aggregate verification is reported in #9882 rather than attributed to this test-only change. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). This test-only change alters no API, schema, stored data or wire format, and requires no release backport. Refreshed CI runs for this published immutable HEAD are pending: [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266409294), [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266409205), [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266409248). No merge is performed until these new-HEAD checks pass. |
||
|
discord9
|
9663b6337c |
feat(promql): bind instant @ anchors and evaluate timestamp natively (#9882)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Follows the merged multi-input wiring in #9896, Complete Aggregate in #9933, and zero-input Scalar/literal broadcasts in #9936. Merge this PR before its test-only follow-up #9886. ## What's changed and what's your intention? Connect instant-selector `@` and `timestamp()` to the standalone native PromQL planner and recursive executor on current `narthex`. - Supports fixed `@ <timestamp>`, `@ start()`, `@ end()` and positive/negative offsets. The selector reads at `anchor - offset`, with the existing lookback and staleness rules, and replays its answer on the statement's evaluation grid. Its scan bounds follow the anchor even when the selected data predates the statement. - `timestamp(raw_selector)` reports the selected source sample's original timestamp in seconds. `timestamp(abs(raw_selector))` and other supported evaluated-vector arguments report evaluation-grid seconds only where the argument has a value. These remain distinct compilation/runtime paths; an offset changes which source sample is chosen, not the timestamp projected from that sample. - Adds `timestamp` to public native planner admission for supported single-field expressions. This is no longer only a compiler/executor API capability. - Retains recursive RuntimeNode ownership, InputId routing, per-node metrics, SeriesLabels and the current frame decoder. The evaluated Timestamp node reuses existing NodeState accounting and lifecycle, with projection workspace reserved before allocation. ### Integration with Complete Aggregate The migration combines Timestamp recursion with the latest scalar-aware `reads_across_series` handling. AggregateFinalize receives only the necessary Timestamp traversal arms and test-fixture updates. Existing `partial_fold`, Complete Aggregate admission, scalar-binary prefixes/suffixes, `FusedSeries.children: Vec<LogicalPlan>` and `BoundInput.source_index` are preserved; no old planner/compiler files or flat NodePath runtime are restored. This remains a single-region standalone experimental Native PoC. Multi-field `timestamp()` still declines native compilation and keeps the existing planner. Timestamp/aggregate shapes outside the current split grammar likewise retain the existing plan. This PR does not add general fallback machinery, cross-table routing, Join, Exchange or distributed execution. `timestamp(x @ t offset d)` preserves GreptimeDB's existing selection semantics rather than introducing Prometheus' special-case offset behavior. ### Synchronization with merged Scalar roots Rebased the reviewed B-only change onto `narthex` at `aa30968d43b3bf9b5c335c6ac3a9f38be58d653b` (#9936 merged). This is a single B implementation commit directly on that baseline, followed separately by #9886's test-only commit. The only textual conflict was adjacent added planner tests; both timestamp tests and the constant-root/broadcast test remain intact. No production merge fix or fallback expansion was needed. The zero-child Scalar route, literal broadcasting, compiler/admission guards, recursive runtime accounting and Complete Aggregate remain present. ### Verification Verified clean final HEAD **`24e214b7d4b3d277ce04a7e40d3aaf3d7673c70a`**, rebased directly onto `narthex` **`aa30968d43b3bf9b5c335c6ac3a9f38be58d653b`**. - `make fmt-check` and `git diff aa30968d43b3..HEAD --check`: passed. - `cargo check --locked -p query -p promql --all-targets` and the separate `-p tests-integration --all-targets` check: passed. - `cargo nextest run --locked -p query -p promql --retries 0 --no-fail-fast`: **1989 passed, 2 skipped**. - `cargo nextest run --locked -p tests-integration --lib --retries 0 --no-fail-fast -E 'test(native_promql_) | test(native_aggregate_complete_over_four_nonempty_mito_partitions)'`: **17 passed**. This includes the instant/timestamp storage query, retained native routes and Complete Aggregate over four nonempty Mito partitions, including #9936's scalar broadcasts. - `cargo nextest run --locked -p tests-integration --test main --retries 0 --no-fail-fast -E 'test(integration_http_file_test::test_prometheus_native_promql_route) | test(integration_http_file_test::test_native_promql_route_plans_the_query)'`: **2 passed**, preserving #9936's File-backed HTTP native and planning routes. - `cargo clippy --locked -p query -p promql --all-targets --all-features -- -D warnings` and the separate `-p tests-integration` invocation: passed. - The added `native_instant_at.rs` was copied byte-for-byte for focused `hawkeye format` followed by `hawkeye check`, using repository inception year 2023; both passed and the copy remained identical. Built fresh `greptime` and `sqlness-runner` binaries from this exact HEAD, then ran isolated standalone SQLness compatibility cases `at_modifier`, `fused_multi_standalone` and `scalar_entry`: **all 3 passed**. Expected `.result` files remained byte-identical; the runner stopped the temporary instance and removed its state. These are compatibility checks, not substitutes for the Native execution tests above. GitHub CI results for the old `d09e4fdc057` HEAD are not reused: refreshed Checks, Rust CI and Integration CI must complete successfully on the new immutable HEAD before merge. The new actual standalone test checks independent expected timestamps, values and host labels, then compares with the legacy route. Its cases cover fixed/start/end anchors, pre-statement anchors, signed offsets, the exclusive lookback boundary and exact zero-lookback selection, missing values, exact-bit stale markers, and raw-sample versus evaluation-grid timestamps. For a bare selector whose newest source value is NULL, the existing Arrow representations differ: Native emits no row, whereas legacy keeps a selected NULL-valued row; the test asserts both representations explicitly, including the legacy timestamp/label, without filtering results or changing production semantics. It collects the output, observes `FusedSeriesExec` and checks native output metrics, rather than treating a logical plan or old CI run as execution evidence. The existing ordinary single-input/two-independent-scan tests and four-nonempty-partition Complete Aggregate test are retained as separate regressions. The two scans are still the #9896 same-table capability; this PR does not claim true cross-table dual input. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [ ] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). The native tree additions change an internal experimental Rust interface; no persisted metadata or wire format is changed. No release backport is requested. Refreshed CI runs for this published immutable HEAD are pending: [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266408649), [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266408660), [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40266408658). No merge is performed until these new-HEAD checks pass. |
||
|
discord9
|
9a70220974 |
test(promql): cover the multi-value timestamp refusal
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
24e214b7d4 |
feat(promql): bind instant anchors and execute native timestamps
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
aa30968d43 |
feat(promql): execute native scalar roots and literal broadcasts (#9936)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Stacked on #9868 (`feat/narthex-fused-wiring`, currently at `990d932198ab0c5e41b724a54d990e5a8ffc8acb`). This change needs that PR's unmerged `FusedSeries`/`SeriesScan`/in-process native route, so the base is its branch; GitHub will retarget to `narthex` once #9868 merges. ## What's changed and what's your intention? Enable the existing opt-in native PromQL route for literal vector/scalar arithmetic such as `x * 100`, unary negation such as `-x`, and constant scalar statements such as `1 + 2`. These public entries return ordinary nonempty query RecordBatches with the same values and output schema as the existing planner. - Traverse unary and binary expression operands when locating the one selector and its range requirements. Constant-only statements reuse the existing constant evaluator and statement grid; unsupported shapes still use the existing planner. - Execute constant Scalar roots as one unlabelled evaluated chunk on the full statement grid, with existing memory admission, output metrics and error/cleanup paths. A no-input tree has genuinely zero logical/physical children rather than a discarded placeholder scan. - Keep each raw input partition attached to its own `FusedSeriesExec` evaluator. This change adds neither an input-side Coalesce nor a SeriesFrame exchange. Ordinary output collection continues to use DataFusion. - Cover scalar NaN/negative-zero/EOF/memory boundaries, child arity and output schemas, independent lazy input progress, and the actual public HTTP/query-handler route. The standalone single-region fixture persists four disjoint input batches: for both broadcast and unary queries, all four input/native partitions perform work and the complete result is exactly 24 labelled timestamp/value rows. A four-stream barrier test collected by DataFusion verifies concurrent progress; this is not a CPU-overlap or speedup benchmark. - Add a generated SQLness compatibility case for scalar arithmetic, broadcast and unary results, rerun successfully without result changes. Scope: constant scalar roots and literal broadcasts only. `scalar(vector)` adaptation, dynamic per-grid scalar arithmetic/broadcast, global Vector/Select and Complete Aggregate wiring are separate work and are not enabled here. The experimental configuration/request gates and current unsupported-shape fallback remain in place. No persisted or remote wire format changes are introduced; internal Fused constructor callers are migrated together. Verification on this exact HEAD (`d9eb9778f113dd9bd6125305a9ea447b4aa6ab1d`, restacked onto the current dependency `990d932198ab0c5e41b724a54d990e5a8ffc8acb`), exit 0: ```text cargo fmt --all -- --check git diff --check cargo check --locked -p promql -p query --tests cargo check --locked -p tests-integration --lib --test main cargo nextest run --locked -p promql -p query --retries 0 1870 passed, 2 skipped cargo nextest run --locked -p tests-integration --test main --retries 0 \ -E 'test(=integration_http_file_test::test_native_promql_route_plans_the_query) | test(=integration_http_file_test::test_prometheus_native_promql_route)' 2 passed, 770 filtered out (four real input partitions, exact 24 values) cargo nextest run --locked -p tests-integration --lib --retries 0 -E 'test(native_promql_)' 4 passed, 251 filtered out (upstream VIEW/metric/ANALYZE compatibility) cargo clippy --locked -p promql -p query --all-targets -- -D warnings cargo clippy --locked -p tests-integration --lib --test main -- -D warnings ``` SQLness (generated, then rerun with an unchanged result): ```text /mnt/nvme_rust/rust-targets-2/d-scalar-entry/debug/sqlness-runner bare \ --bins-dir .../debug --case-dir <isolated root> --test-filter '^standalone:scalar_entry$' --jobs 1 ``` This SQLness case uses the default public compatibility route, not the native request flag; it does not replace the native execution assertions above. Full workspace `make test`/`make clippy`/udeps and external CI have not been run for this change. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
05ef1a4326 |
test(promql): use fixed scalar batch fixture
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
3c409f91de |
test(promql): align admission expectations with literal entry
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
e1e92a28ea |
test(promql): fix scalar schema ownership and rebuild API
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
ab721b912b |
test(promql): quote region label in Mito fixture SQL
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
9e0f74091b |
test(promql): scope partition proof to unfiltered scans
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
11287b8c71 |
test(promql): cover scalar aggregate composition and fallback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
503c3ec48d |
test(promql): retain and verify empty vector result schemas
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4da340262e |
test(promql): let query encoding use auto-deref
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
00338b5121 |
test(promql): import partition properties for output assertions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
12c7d0ad4f |
test(promql): prove each scalar-route partition produces output
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
04e8cfd2bb |
test(promql): disambiguate the frontend SQL test helper
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
6eff9f5804 |
test(promql): verify scalar routes and four-partition broadcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
5bcd96cead |
feat(promql): execute zero-input scalar roots through fused children
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
34761265d5 |
feat(promql): wire complete aggregation over parallel fused partitions (#9933)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Rebased onto `narthex` at `bd6be2cf8f83b04280d3113467563ae81005e2f3`, including merged #9868, #9937, #9914, #9870, #9871 and #9896. This PR no longer depends on the old #9896 feature branch. - Builds on the existing native Complete aggregate runtime from #9873. #9936 can be restacked after this PR merges. ## What's changed and what's your intention? Connect Complete aggregation to the current experimental native PromQL execution path while preserving actual partition-local evaluation: ```text SeriesScanExec N → FusedSeriesExec N → complete partial accumulator states N → CoalescePartitionsExec → FloatMerger → AggregateFinalizeExec 1 → ordinary RecordBatch ``` ### Execution and current API adaptations - Keep #9896's `FusedSeries.children`, physical children, `BoundInput.source_index`, and existing ordinary multi-input execution. An optional `partial_fold` changes only the output mode: ordinary executions still emit query rows; aggregate producers fold evaluated chunks into complete accumulator state and export it only after successful input EOF. The aggregate path currently has one actual scan child, with a separate evaluator for each of its N partitions; no gather is inserted before evaluation. - Split at the innermost supported Complete aggregate. The existing `FloatMerger` combines full state, not finalized partition answers: average retains count and compensated sum, and variance retains count, mean and M2. The finalizer waits for all producer input to end successfully before running the global suffix or returning rows. A late source error fails the query rather than returning an incomplete aggregate. - Build the suffix as a separate `NativeTree`. Its `source_index = 0` names its own evaluated input, not the original scan. Obtain the actual input from `NativeExecutionState::input_ids(0)` and call `NativeExecutor::push_evaluated(InputId, chunk)`; do not equate a source index with a runtime input ID. No `NodePath` or flat executor is restored. - Use current `SeriesLabels` and the raw field layout from Native Input output samples, without restoring a separate scan-schema mirror. Reuse the existing row converter for final output. State and output reservations are execution-local and released on errors and stream drop, including when a physical plan is retained for another execution. - Preserve `sum`, `avg`, `count`, `min`, `max`, `group`, `stddev` and `stdvar`, `by` / `without`, maps and nested aggregate suffixes. Literal scalar broadcasts work below the cut (`sum(8 * rate(...))`, `max(10 - rate(...))`) and above it (`100 - avg(...)`, `sum(...) * 2`, `100 / sum(...)`, `-sum(...)`), keeping subtraction and division operand order. - Compose the current selector matcher/time-bound filters with aggregate planning. Add `sum(rate(counter{job="a"}[5m]))` to the existing standalone fixture; the independently expected answer is 9. **The private state-row `RecordBatch` codec remains a temporary bridge.** Producer, codec and finalizer retain the explanation that future native final aggregation should consume series frames directly, eliminating the intermediate state-row encoding, decoding and reconstruction. This PR does not implement that replacement. ### Scope and compatibility The supported target is standalone, one Region, multiple nonempty scan partitions, with in-process region transport and the existing engine/per-query native opt-ins. Ordinary Fused input traversal is not redesigned. There is no cross-Region series merge, generic Exchange, multi-input Aggregate Join, distributed Partial/Final architecture, new public configuration, persisted format or wire format. No performance or CPU-overlap improvement is claimed. Unsupported aggregate topologies remain outside this PoC. ### Verification All commands below passed on exact head `9720e1bf88a776bd017b304152a055e1735d4ba1`, over the narthex base above, using `RUST_MIN_STACK=8388608`, a dedicated external Cargo target, and no test retries: - `cargo fmt -p promql -p query -p tests-integration -- --check` - `cargo check --locked -p query -p promql --all-targets` - `cargo check --locked -p tests-integration --all-targets` - `cargo nextest run --locked -p query -p promql --no-fail-fast --retries 0` — **1967 passed, 2 skipped**. - `cargo nextest run --locked -p tests-integration --lib --no-fail-fast --retries 0 -E 'test(native_aggregate_complete_over_four_nonempty_mito_partitions) | test(native_promql_)'` — **16 passed, 251 skipped**. - `cargo clippy --locked -p query -p promql --all-targets --all-features -- -D warnings` - `cargo clippy --locked -p tests-integration --all-targets --all-features -- -D warnings` - `git diff --check bd6be2cf8f83b04280d3113467563ae81005e2f3..HEAD` - `hawkeye format --dry-run` and `hawkeye check` The standalone aggregate case checks typed physical-plan structure as well as results: `SeriesScanExec` and partial Fused each retain four partitions, with no intervening exchange; the finalizer consumes the producer's state schema and answers one partition. For unfiltered counter cases all four evaluators must report positive contributions, totaling five and at most two per partition, forcing the three-member group across partitions. Results are checked for labels, timestamps, columns, values and multiplicity against independent expectations, in addition to legacy execution. The five nonzero counter rates are 1/3/5/2/4: total sum 15; per-job sums 9/6, means 3/3, counts 3/2 and standard deviations sqrt(8/3)/1. Scalar cases include `sum(8 * rate(...)) = 120`, `max(10 - rate(...)) = 9`, `100 - avg(...) = 97`, `100 / sum(...) = 100/15` and unary minus = -15. The subtraction prefix distinguishes an incorrectly moved suffix (5) or reversed operands (-5). The matcher selects three series, so only that case does not require four nonempty partitions; all unfiltered assertions remain intact. Unit coverage includes all eight accumulator kinds, unequal producer counts (one value versus two, so the merged mean is 6 rather than the mean-of-means 5), nested/map/scalar suffixes, empty input, producer read barriers, late errors, and reservation/drop boundaries. Local full-workspace tests, an HTTP smoke run and native SQLness have not been run; the SQLness runner does not carry the per-query native hint. Final-head CI for `9720e1bf88a776bd017b304152a055e1735d4ba1` has completed: [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361500), [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361419), [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/40201361402) all succeeded. The final check rollup contains 40 successful checks and 2 skipped checks, with no pending or failed checks. An earlier Semantic Pull Request run was cancelled and superseded by a successful run; the current Semantic PR title check passed. No workflow reruns were requested. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
9720e1bf88 |
style(promql): attach native predicate rustdoc directly
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |