mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-17 16:24:07 +00:00
72bee4ae9cd4d8eebae2e8cb951046b5ab804e73
6905
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
72bee4ae9c |
feat(promql): answer whole-vector and multi-metric statements natively (#10025)
* feat(promql): name a fused tree's tags from the existing plan A fused node declares exactly the tags the planner gives it, and reads each one back off the series, so a label set this route cannot derive is a label set the answer would lose. The walk over the tree derives labels for a match, a set operation and the point-wise wrappers over them, and names none for an aggregate's groups or a whole-vector operation - which today declines every such statement. Take those columns from the existing planner instead, the way the aggregate split already does: its tag columns are what it projects for the same statement, and they are now shared through one helper rather than written out twice. Only the tag columns are copied, because the node names its field columns after its own tree while the existing planner can name a computed column after its expression. A label rewrite is a shape the walk cannot derive at all: label_replace adds its destination label and an empty replacement removes it, so passing the input's labels through is an inference the rewrite contradicts. Report no label set there. The rewrite itself keeps the existing plan, and this does not change that: the native runtime refuses two series a rewrite gives one label set, where that plan answers them, so admitting the shape would turn an answer into an error. The test that holds this now covers the two-occurrence shape as well. An aggregate or whole-vector root does not reach this branch today: both read across series, so they are planned by the aggregate split or fall back. It is the prerequisite for the placement that runs such a tree in one execution, and reaches the branch when that lands. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run a classic histogram function above the aggregate cut The split accepts a suffix that runs once over the fully merged series, which is why TopK/BottomK were already admitted: the merge has folded every partition's contribution to each group, so the series the suffix reads are the ones the statement's input would have produced in one execution. A classic histogram function is that kind of operation. It reads each histogram off the bucket series that share its other labels and drops le, and the merge has already folded each (..., le) group into one complete series on the evaluated grid, so the buckets it reads are exactly the ones the input would have had. Reading le off those merged series is the same computation, not an approximation. LimitK still declines: it depends on the storage input order, which the merge does not preserve. This is what the o11ybench query `histogram_quantile(0.99, sum(rate(...[5m])) by (instance, le))` needs: it was planned by the existing path only because the split refused a whole-vector operation above the cut. The acceptance test runs the shape over real data and compares the rows two executions report: the fixture spreads six bucket series over several files and a memtable range, so the partial sums genuinely have partitions to merge, and the native answer must equal the complete input's for quantiles at both ends of the interpolation and for histogram_fraction. It also asserts the native run really merged state, so a silent fall back cannot pass it. Signed-off-by: discord9 <discord9@163.com> * feat(promql): read two metric logical tables as two inputs A two-selector statement whose two sides are aggregates kept the existing plan, even where both sides read a metric-engine logical table. The o11ybench query `sum(max by (...) (a)) by (namespace) / sum(b) by (namespace)` is that shape, and it fell back for three separate reasons. First, the engine gate refused every multi-selector metric-engine statement. The clause after it already requires each such selector to be a recognized logical rewrite, so the removed clause refused only the case this change handles. Second, a metric logical table is rewritten to a scan of the physical table restricted by `__table_id = <id>`, so two logical tables over one physical table prepare two scans of the same table. The check deciding whether the node binds one metric or one input per occurrence compared table names only, so the two metrics read as one and both inputs would carry the first one's name - a wrong answer, not a fall back. The id the rewrite already requires is now part of that identity. Third, a tree over two aggregates reads across series, so it is not answered per partition, and the branch for such trees was the aggregate split, which takes one child. A node with more than one input needs no split: it declares one partition and reads all of every input's partitions in one execution, so every group is folded in one place. Such a tree now falls through to the node. The one-selector path is unchanged: it still requires `can_split` and still declines a tree the split cannot cut. So is the multi-selector case whose inputs stay direct scans of one table - such a node is not region-local, as `Categorizer::check_extension_plan` refuses a cross-series plan over more than one scan - which is why the condition is the metric-logical-table one. The acceptance test runs both shapes over two logical tables of one physical table, with values that make a swapped input visibly wrong, and compares the rows the two routes report. Signed-off-by: discord9 <discord9@163.com> * test(promql): log the metric pair's rows through telemetry, not stdout The acceptance test printed its evidence with println!, which the crate lints as an error under -D warnings, and echoed a Display value through to_string inside the format arguments. Every other test in the file logs through common_telemetry::info!, so this one does too. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep the existing plan for a tree the runtime cannot execute The multi-input route skipped the aggregate split, and the split was what refused a tree the runtime cannot execute: `check_cut` declines a `quantile` and a grouping by a value field. Skipping it turned queries the existing plan answers into errors, because the node is chosen while planning the logical plan and a refusal the physical layer raises later has nowhere to fall back to. `quantile(0.5, a) / sum(b)` over two metric logical tables reported `NotImplemented: the native runtime does not execute the Quantile { .. } aggregate yet` where the existing plan answers it, and `count(count by (val) (a)) / sum(b)` failed on the grouping by a value field, which the outer `count` had hidden from the column check. Ask the runtime before taking the route, with the same construction the physical layer validates by, and keep the existing plan for whatever it refuses. Asking it rather than listing the unsupported kinds is what makes the check cover refusals this planner does not know about. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep the existing plan for a grouping and a selection the node cannot answer A multi-input node skips the aggregate split, so the refusals the split makes for reasons that are not about what the runtime computes were skipped with it. Two of them matter here. A grouping by the metric name keeps a label no column of the existing plan can name. The native aggregate folds its groups by `__name__` as by any other label, but the label set the node declares comes from the existing planner's columns, and `__name__` is the name a raw frame carries rather than a column of any scan - so the existing planner's `by(__name__)` finds nothing and reports no label. The schema would then report one label set for two groups the tree keeps apart: `sum by (__name__) (a or b)` over two metrics would answer two rows a reader cannot tell apart, which is a series identity lost rather than an answer given. A selection that depends on the order its input arrives in is the other. `limitk` keeps the first `k` series of each group in input order, while a node with more than one input reads its inputs' partitions through a gather that hands them over in the order they became ready: the schedule would decide which series a query keeps. Prometheus defines the selection over a series set. The split declines `LimitK` for the same reason. Every other vector operation orders its input itself - `topk` by label, `sort` by value, `limit_ratio` by a label hash - which is why only this one is asked about. Both shapes keep the existing plan. `limitk` is not answerable by either route today, since the existing planner has no kernel for the operator; the point is that the native route is not the first to answer a selection the schedule would decide. Each guard is checked with the guard removed, so both tests are known to fail without it. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline unsupported count_values groupings before execution Signed-off-by: discord9 <discord9@163.com> * docs(promql): scope the native capability probe guarantee Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the same-table pair the multi-selector exclusion used to keep out State that a Mito input joins a multi-input tree only when a second metric logical id makes the tree multi-input, and that only LimitK's selection depends on arrival order. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decline ranking grouped by a value field before native execution topk, bottomk and limitk key their groups by series labels, while the existing plan also accepts a value field there and forms one group per sample value. Admitted multi-input trees could therefore keep one row where the existing plan keeps one per group. Validate the grouping when the vector runtime is built so the statement keeps the existing plan. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
bb20057985 |
feat(promql): route evaluated fragments to their canonical owner (#10038)
* feat(promql): route evaluated chains independently by canonical key Signed-off-by: discord9 <discord9@163.com> * test(promql): verify canonical routing key semantics Signed-off-by: discord9 <discord9@163.com> * fix(promql): release closed-output chains and yield routing producers Signed-off-by: discord9 <discord9@163.com> * test(promql): check closed-chain budget before releasing the child Signed-off-by: discord9 <discord9@163.com> * fix(promql): account histogram slices and verify routing completion Signed-off-by: discord9 <discord9@163.com> * fix(promql): finalize contiguous series at repartition owners Wait for successful producer EOF before publishing exact complete metric-name/tag identities. Reuse the existing sample merge for position ordering, complementary fields, and conflicting-cell errors; recompute one final close per owner identity. Keep N producer tasks and M outputs. Admit nonwaiting queue payloads through the query pool, reserve merge workspace, and retain cancellation, failure, memory-release and unread-owner coverage. Signed-off-by: discord9 <discord9@163.com> * feat(promql): publish completed repartition identities incrementally Signed-off-by: discord9 <discord9@163.com> * test(promql): expose histogram amplification in public repartition Signed-off-by: discord9 <discord9@163.com> * fix(datatypes): encode only referenced histogram table rows Signed-off-by: discord9 <discord9@163.com> * test(promql): wait for queued retention before draining a failed read Signed-off-by: discord9 <discord9@163.com> * perf(promql): bound identity rescans and share repeated histogram rows Signed-off-by: discord9 <discord9@163.com> * docs(query): scope the evaluated validator to one chain Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
daa903254d |
perf(mito): cache native reader metadata (#10015)
* perf(mito): cache native reader metadata in the decoded SST tier Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): account for partially occupied metadata tree nodes Signed-off-by: evenyag <realevenyag@gmail.com> * style(mito): use absolute cache imports Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito): trim native metadata cache lookup overhead Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito): bound wide metadata benchmark setup Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito): allow manual benchmark measurement output Signed-off-by: evenyag <realevenyag@gmail.com> * docs: keep series specification focused on the file format Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
fe8b64f0e4 |
chore(ci): slide the compatibility window to v1.3.0 (#10054)
Signed-off-by: discord9 <discord9@163.com> |
||
|
|
eb8e4316ec |
ci: backport scoped heavy CI to narthex (#10053)
* ci: backport scoped heavy CI to narthex Backport #9927 and the CI-only changes from #10043. Adapt script references to narthex, preserve manual platform checks, and document the default-branch schedule limitation. Upstream-Commit: |
||
|
|
9f906da091 |
feat(promql): run time(), vector(s) and data-free scalar broadcasts natively (#9988)
* feat(promql): run time(), scalar(v), vector(s) and dynamic scalar broadcasts natively Bind time() and zero-argument date functions as EvalTime, scalar(v) as ScalarOf and vector(s) as VectorOf, and run binary operations whose scalar side varies per grid point: the scalar is read whole, vector chunks that arrive before it are held and charged, and both operand orders and bool comparisons follow Prometheus. A tree holding scalar(v) runs as one execution over every scan partition, and the planner admits it only over an unpartitioned Mito table. Data-free statements plan without a scan. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): hold the labels of chunks waiting for a varying scalar Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * feat(promql): read round and clamp parameters from varying scalars natively Bind round, clamp, clamp_min and clamp_max whose parameters are not all constants as a ParameterizedMap with scalar parameter nodes, run through map_math_with_parameters. The parameters are read whole first; vector chunks that arrive before them are held and charged, through the same hold the varying-scalar broadcast now shares. Constant parameters keep the MathOp map. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): pin a parameterized map's empty closing chunk Also correct the held-chunk docs, use same-table examples for varying map parameters, and assert the native decline instead of the existing planner's failure. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): plan vector(s) natively whatever the table's field is named The fused route declined a root field that is not a raw field, which is every root over vector(s): it names its result `value`. Take the compiled root's fields as they are and decline only a name the output already gives a tag or the time index. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): reserve a constant map parameter before building it Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(promql): leave scalar(v) to the existing planner until it has its own execution boundary Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): leave data-free statements that generate labels to the existing plan Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(promql): reserve the output point-wise kernels build before running them Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7996bcb6fd |
perf(mito): batch samples reads and reuse part readers (#10016)
* perf(mito): batch samples reads and reuse part readers Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): validate boundary reads and report scan memory exhaustion Signed-off-by: evenyag <realevenyag@gmail.com> * docs: keep samples specification focused on the file format Signed-off-by: evenyag <realevenyag@gmail.com> * style: remove trailing blank line in samples format doc Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
3c21571387 |
perf(mito): push predicates into columnar series discovery (#10014)
* perf(mito): push predicates into columnar series discovery Signed-off-by: evenyag <realevenyag@gmail.com> * perf(mito): resolve series tags once per discovery batch Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
c25fd3100e |
perf(mito2): coalesce raw series records before residual predicates (#10017)
* perf(mito2): coalesce raw series records before residual predicates `raw::make_stream` evaluated every residual predicate against one record per `ScanBatch`, and a `ScanBatch` carries the rows of a single series. A predicate pass is dominated by per-call setup rather than by per-row comparison, so the cost was paid once per series: profiling the lite-02 aggregate query put `common_recordbatch::filter::batch_filter` at ~20% of CPU across 1280 calls for 640 series, and 70% of that was per-call overhead. Buffer records until 4096 rows, concatenate them with `concat_batches`, and evaluate the predicates once per group. The grouping loop, the `is_last` marking, the strictly-increasing checks and the error messages are unchanged. Three guards keep the behaviour identical: - A record whose first primary key continues the series the buffer ends with is held back, because concatenation would fuse two chunk boundaries of one series. Two existing tests pin those boundaries. - With no predicate there is nothing to amortize, so the threshold drops to one record and the stream keeps its per-record shape. - Buffered records are reported one at a time when an upstream error arrives. Merging them would put rows into the chunk withheld as `pending`, dropping a prefix the per-record stream emitted before the error. All 12 vmbench-lite queries return identical series, timestamps and values. Paired A/B against the previous head: CPU is lower in 30 of 30 rounds, median cut 13.8%-29.4% across five queries. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> * perf(mito2): release coalesced source records before filtering The concatenated arrays own their buffers through `Arc`, so the source `RecordBatch`es are dead once `concat_batches` returns. Releasing them keeps the original arrays from staying alive across the predicate pass and the per-chunk yields of the same round. Raised in review of #10017; no behaviour change. Signed-off-by: discord9 <discord9@163.com> * perf(mito2): start a group at a record that fills a batch, drop the error split Two review comments on #10017. `make_stream` processed the records buffered before an upstream error one at a time, so the chunks of the earlier records came out before the error. Nothing reads that prefix: the error fails the query, and the `RawSeriesStream` contract promises per-series ordering and finality, not a prefix on error. The predicate path already withholds its `pending` chunk on failure, so the split was also inconsistent with the rest of the stream. Removed, along with the half of `filtered_boundary_fuses_chunks_on_eof_but_an_upstream_error_still_withholds_one_record` that pinned it; the remaining fusing assertion keeps its own test. A record that reaches `coalesce_rows` rows by itself now heads its own group. Concatenating it with whatever is buffered copies it, and its predicate pass is paid once either way; on its own it takes the single-record path unchanged. The threshold's doc comment no longer claims a record larger than the bound is isolated, which was never true. Neutral for the measured workload: a vmbench-lite series is 61 rows, so the threshold is never met by one record. `cargo nextest -p mito2` series-scan selection: 53 passed. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> |
||
|
|
487a398116 |
fix(promql): keep a group's points together when decoding aggregate state (#10008)
* fix(promql): keep a group's points together when decoding aggregate state The state encoder writes a group's points in ascending order, one row per (point, field). The decoder opened a group whenever the row's point differed from the group's first point, so every point of every group became its own group: each row re-allocated the label set's `Arc<str>` strings, and the merge then re-hashed those labels and repeated a map lookup for each of them. Open a group only when the labels change. A later point of the same group is appended to it and a repeated point is that point's next field, which keeps the arrival order of every (point, field) and therefore the floating-point merge order unchanged. A point that goes backwards within a group is refused. Appended points and their state vectors are admitted before they allocate. Measured with paired alternating A/B on one prepared vmbench-lite database, 30 measured requests per run, opposite order each round, 6 rounds: Q02 avg without(cpu) 5m p50 A/B 1.60, CPU A/B 1.25, 6/6 rounds faster Q11 sum by 4 tags 5m p50 A/B 1.57, CPU A/B 1.49, 6/6 Q01 100-(avg(...)*100) p50 A/B 1.40, CPU A/B 1.25, 6/6 Q10 count by p50 A/B 1.04, CPU A/B 0.99, 4/6 (neutral) Q03 sum by region,az p50 A/B 0.92, CPU A/B 0.87, 2/6 (slightly slower) `cargo nextest run -p promql` 987 passed, 0 skipped. Signed-off-by: discord9 <discord9@163.com> * fix(promql): accept concatenated aggregate state instead of rejecting it Review of #10008 found three problems. `decode_state` refused a row whose point went backwards within its group, and refused a repeated `(point, field)` pair. Both are real inputs once state crosses Flight: `CoalescingBatcher` merges consecutive small batches (`servers/src/grpc/flight/stream.rs:313`), so two partitions' state for one group arrives back to back and the second restarts the group at its first point. Accept them instead: the merge addresses points by index (`aggregate/state.rs:396-423`) and folds groups by label key, so a second entry for the same point lands in the same slot, exactly as merging the two batches separately would. Out-of-grid point and field indices are still errors. The doc comment on `decode_state` still described the old rule (a change of label or point opens a group); only a label change does now. `decode_admits_the_points_it_appends` did not test what it claimed: the fixed layout charge already exceeded its `tallest / 2` threshold, so the first admission was refused whether or not appended points were charged, and the test passed with the append-time `budget(total)?` removed. Derive the threshold from a one-point batch of the same group so only appended points can exceed it. Both files' tests updated to pin the new behaviour: a concatenated batch of two encoders' output decodes like the two batches merged, a repeated field sums, and the budget test now fails if the append-time charge is dropped. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> Signed-off-by: discord9 <discord9@users.noreply.github.com> |
||
|
|
5b17bbd584 |
feat(promql): run info() natively (#10005)
* feat(promql): run info() natively
info(v, {selector}) fell back to the existing plan, which has no info
and refuses the query ("Unsupported expr type: info", or "Cannot accept
multiple vector as function input" when a selector is given).
Compile it into the native tree and run it on the info kernel:
- Plan: NativeExpr::Info { base, info, selector }. The info side is an
instant selection reporting each selected sample's own time in
milliseconds (InstantValue::InfoTimestamp, which replaces the
BoundInstant.timestamp flag); validation refuses any other info side.
- Compiler: info_input derives the info selector the way Prometheus
does: target_info or one __name__="..." matcher, the data matchers,
and the @ and offset every selector of the first argument shares
(subquery offsets added until an @ anchors them, none when they
differ). Regex or negative name matchers, which can select several
tables, __field__/__schema__/__database__ matchers and multi-field
info tables are refused. The planner's selector walk uses the same
helper, so both bind the info table as the last input.
- Runtime: BinaryRuntime runs the kernel as PairKernel::Info, with the
retention and pool accounting of binary and set nodes; the kernel
reports its workspace through info_with_reservation.
- Planner: the info table is a second table, so it is read through its
own RawSeriesSource and the node runs on the frontend. The output adds
the info table's tags, or only the data labels the selector names.
Partitioned or non-Mito tables and a missing info table keep the
existing plan.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* test(promql): pin region-local placement of info over its own table
info(rate(target_info[5m])) reads the info table twice. Check that the
planner binds both as direct scans under one FusedSeries node bound to
the table, that the node is placed in the region, and the columns it
reports.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): keep sample-less base series out of info's identifying patterns
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* fix(promql): share identifying label strings in info() instead of copying them
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
|
42f69f2d82 |
perf(promql): resolve the vector match result row once per pairing (#10007)
* perf(promql): resolve the result row once per pairing in vector match Signed-off-by: discord9 <discord9@163.com> * docs(promql): state the native node metric naming contract Signed-off-by: discord9 <discord9@163.com> * bench(promql): make the vector match timing reproducible Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
654a44aa9f |
feat(promql): run native statements over any number of selectors (#10004)
* feat(promql): run native trees with more than two selectors Lift the two-selector cap of the native PromQL route. A tree of one-to-one vector matches and set operators, nested under point-wise wrappers, is now admitted for any number of selectors: on one table the node runs in the region over every scan, across tables it runs on the frontend over one RawSeriesSource per selector. Partitioned tables, aggregates, whole-vector operations and many-to-one matches still keep the existing plan. The dist analyzer and commutativity checks accept a same-table node with more than two scans; without that such a plan fails in the analyzer. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): cover nested set operators and metric-engine multi-selector decline Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(promql): wait for aborted partition tasks to release their inputs Signed-off-by: Dennis Zhuang <killme2008@gmail.com> (cherry picked from commit 3f8699782c3116c4063ec70a3522d1f5f3bff897) * docs(promql): say spawned partitions are released after an error, not before Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
87bd2c6e62 |
feat(promql): run cross-table selector pairs natively (#9990)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring (#25): cross-table inputs ("wiring B"). Builds on the two-scan route of #9870 / #9871 / #9896. ## What's changed and what's your intention? A two-selector native statement can now read two different tables: `a / b`, `rate(a[5m]) / rate(b[5m])`, and set operators across tables. A scalar or point-wise wrapper around such a pair is admitted too, e.g. `100 * (1 - a / b)`, `(a / b) * 100`, `abs(a - on(x) b)`, on one table or across two. Before, the planner required every input to share one metric, time index and tag list, and only admitted the pair as the root. Both kinds of query fell back to the legacy plan. - **Placement.** A cross-table `FusedSeries` runs on the frontend. Region-side placement would read the wrong data: `NameAwareDataSourceInjector` rebinds every `TableScan` in a region request to that request's one region, so a two-table node sent to one region would read it for both inputs. - **`RawSeriesSource`.** New logical node and exec, one per table. It wraps that table's scan and carries the table's metric name, tags and fields. It is commutative, so the dist analyzer pushes each one into its own `MergeScan`. - In the region, it builds the `SeriesScanExec`, materializes labels, and emits one explicit-label raw frame stream declared `Contiguous`, reading the scan partitions in order. A series never spans scan partitions, so each series stays contiguous. - Its declared frame schema is built the same way as what the region emits, which `MergeScan`'s exact `validate_remote_schema` requires. - `plan_fused_series` accepts these frame children, and `ScanHintRule` gives their scans the per-series distribution. - **Planner.** - Each input gets its own scan time filter on its own time index; the output time index is the first input's. - Output labels are worked out over the whole tree from each input's own tags: - one-to-one `on` / `ignoring` narrows the left side's labels; - `and` / `unless` keep the left side's labels; - `or` takes both sides' tags, and a series' missing tag is NULL, not invented. `FusedSeriesExec` takes its label columns from the output schema; before, it narrowed them from the root binary alone, which was wrong for wrapped trees. - A tree whose labels cannot be named declines: an aggregate or vector operation, or an `or` union whose columns would collide. - **Comparisons.** Vector–vector comparisons (`a > on(host) b`, `a == bool b`, and wrappers around them) are admitted on one table and across two, with Prometheus semantics: a filter keeps the left sample and labels, `bool` returns 0/1 and drops the name. - **Route precedence with #9973.** The whole statement is tried natively first, so a pair over two unpartitioned Mito tables now takes this cross-table route instead of #9973's hybrid route; the hybrid route still covers what this route declines (e.g. partitioned or metric-engine tables). #9973's hybrid fixtures use a partitioned right table to keep exercising it. A right side with a duplicate match group now returns Prometheus' "found duplicate series for the match group" error on this route, where legacy returns a result. - **Unchanged.** - Same-table pairs still run inside the region. - Partitioned tables with two selectors still fall back. - `group_left` / `group_right`, aggregates on both sides, more than two selectors and `info()` are not covered and keep the legacy plan. In an o11ybench plan probe (standalone, Mito, single-region tables), vmbench-lite-05, lite-06, lite-08 and long-range-03 now plan natively. That makes 22 of the 30 queries native, all with results equal to legacy. Legacy itself fails `a / on(host) b` when the two tables' time indexes have different names ("Error parsing timestamp from 'h0'"); native answers it. That case is checked against explicit rows only. - **Concurrent inputs.** The multi-input gather of `FusedSeriesExec` and `RawSeriesSourceExec` used to drain scan partitions (and inputs) one after another. That stalled mito2's legacy series distributor, which sends into one `channel(1)` per partition. On regions with overlapping SSTs, every scan took ~480ms, and a same-table pair took up to 12.8s. A new `SeriesGather` polls every partition and input concurrently, each partition on its own query task. - It hands the executor series-contiguous input: a chunk is held while another partition of the same input has a series open. - What it holds is at most one series per partition, charged to the memory pool. - Inputs do not constrain each other. - An error frees everything; dropping the stream aborts the partition tasks; a panicking task reports an error. The distributor itself is fixed separately on main in #9991; this PR no longer needs it. Measured on standalone single-region Mito tables with overlapping SSTs (o11ybench data, median of 10, ON vs legacy OFF): | query | sequential gather (before) | this PR | legacy | | --- | --- | --- | --- | | vmbench-lite-05 | 479 ms | 3.03 ms | 7.4 ms | | vmbench-lite-06 | 970 ms | 4.19 ms | 10.4 ms | | vmbench-lite-08 | 971 ms | 3.30 ms | 8.0 ms | | vmbench-long-range-03 | 98.6 ms | 39.6 ms | 36 ms | | same-table cpu pair | 12.8 s | 15.3 ms | 7.6 ms | Same-table pairs are still slower than legacy (0.5–0.7x), because native scans the region twice while legacy plans `x / x` with one scan. Results are equal to legacy in every compared query. One exception: legacy returns no series for `x{region="r"} / x{region=~"r"}`, where both sides select the same series; native returns them, as Prometheus does. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
02eb3aa60c |
chore: merge narthex into feat/narthex-cross-table
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
b29a471575 |
fix(promql): repair native execution contracts and remove obsolete paths (#9980)
* fix(promql): restore native scan statistics and remove unused raw merger Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): repair native execution contracts and remove obsolete paths Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): leave fused frame output repairs to separate PR Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs: restore native contracts and regenerate config reference Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix: slice nullable sparse frame fields after span selection Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): intersect anchored timestamp selection windows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): name anchored timestamp boundary case type Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): align partitioned pair checks with hybrid operands Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@163.com> |
||
|
|
ba8156be47 |
fix(promql): keep the right side's labels for a filled left side across tables
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
dd79052c7f |
fix(mito2): stop the series distributor from stalling on unread partitions (#10003)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Cherry-pick of #9991 (merged into `main`) onto `narthex`. ## What's changed and what's your intention? `FusedSeriesExec` with two inputs reads every scan partition one after another. On `narthex` this hits the series distributor's round-robin `send_timeout(100us)`. Tokio timers tick at 1 ms, so after a merge (one batch per series) each series batch costs about 1 ms per unpolled partition. With 16 partitions, two-selector native queries over regions with several SSTs spent ~470 ms to 12.7 s in `SeriesScanExec`, while legacy took ~10 ms. #9991 waits on all senders at once and sends to the first free slot. The cherry-pick applies cleanly and the diff is identical to #9991. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
3d5f99596c |
chore: merge narthex into feat/narthex-cross-table
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
70638a769f |
feat(promql): add explicit local canonical merge operator (#9969)
* feat(promql): add explicit local canonical merge operator Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): tighten canonical merge accounting and validation Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): remove redundant merge tree whitelist Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): retain canonical merge output label reservations Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): finish canonical merge review simplifications Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the parent barrier over canonical merge Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): assert the two-selector binary stays unfused Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
1dbc2c15ce |
fix(mito2): stop the series distributor from stalling on unread partitions (#9991)
When every partition channel is full, the distributor walked the senders
round-robin with a 100us send timeout each. Tokio timers fire on a 1ms
tick, so a consumer that leaves some partitions unpolled costs about one
millisecond per unpolled partition for every series batch.
SeriesScan::build_stream and scan_all_partitions drain partitions one
after another, and SortPreservingMergeExec above a multi-partition
SeriesScan only pulls the partition holding the current minimum key.
With 16 partitions and one batch per series after a merge, a 40-series
scan read partition by partition took ~480ms (measured on the native
PromQL two-input route).
Wait on all senders at once and send to the first one that frees a slot.
The send-timeout metric goes away with the timeout.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
|
7bcdec1a83 |
chore: merge narthex into feat/narthex-cross-table
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/extension_plan/fused_series.rs # src/promql/src/extension_plan/fused_series/tests.rs |
||
|
|
04a5aab7c1 |
fix(promql): charge the chunks a spawned partition reads ahead
A spawned partition queues up to two chunks and its task holds a third while the queue is full. None of them was charged to a reservation, and since they carry materialized labels their size is unbounded even though their count is not. Each chunk is now charged to the query's memory pool when the task reads it, and the charge travels with the chunk: it is released when the chunk is handed to the gather, which charges what it holds back itself, or when the chunk is dropped in the queue or in an aborted task. A pool that cannot hold the next chunk ends the partition in an error. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
ed562fc905 |
feat(promql): run vector-vector comparisons natively
A one-to-one comparison between two selectors, such as `a > on(host) b` or `a == bool b`, kept the existing plan: the pair admission accepted only arithmetic operators. The kernel already pairs comparisons by the same matching rules and only differs in the value a pair keeps (the left sample when the comparison holds, or 1/0 with `bool`), so the operator check is dropped. This covers one table and two, and the point-wise wrappers around a pair, such as `(a > b) * 100`. Adds an end-to-end case over two tables comparing explicit rows, the legacy answer and the native placement for a filter and a `bool` comparison. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
48797ab132 |
fix(promql): delegate fused frame output to the shared packer (#9968)
Reuse pack_evaluated_series for fused Frame output, preserving the single native driver and PlainRows/partial timing. Reject mismatched logical frame schemas and use the reset scan plan in integration tests. Add evaluated-packer lifecycle and invalid-sequence regressions, and address the review follow-ups. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
a9266b2da9 |
feat(promql): fuse independent operands of hybrid binary queries (#9973)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## What's changed and what's your intention? Enable vmbench-lite 05/06/08 cross-metric arithmetic using two independently planned native operands and the existing DataFusion binary join, PromQL matching and arithmetic projection. Physical shape: `SeriesScan A(N) -> Fused A(N)` and `SeriesScan B(M) -> Fused B(M)` execute region-side; separate MergeScan branches feed the existing frontend Join. No native Join, second matching kernel or Exchange is added; same-table fused admission is preserved. A bounded analyzer pass locates each direct Fused(TableScan) through Alias/Projection/Filter/Window wrappers, rewrites only the fused subtree, and leaves all wrappers frontend-side. This preserves global uniqueness checks. Subquery bodies disable native admission and restore the prior flag before returning either success or error, avoiding use of outer evaluation bounds. ### Verification - `make fmt-check` - `cargo check --locked -p query -p tests-integration --all-targets` - Focused query admission/analyzer tests: 2 passed - `cargo nextest run --locked -p tests-integration --lib -E 'test(native_hybrid_binary_05_06_08)'`: passed, independently rerun on final source - `cargo clippy --locked -p query --all-targets --all-features -- -D warnings` - Independent source review approved. Standalone integration asserts two independent region-local Fused->SeriesScan branches, four nonempty partitions per input, frontend join placement, native/legacy complete-row parity, unmatched-series removal (05), outer scalar projection (06), and independent rate inputs (08). Additional cases cover subquery legacy fallback, valid reduced-label `on(job)` matching, frontend uniqueness Window plus `prom_assert_unique_match_group` Filter above MergeScan, and meaningful duplicate-match execution errors under both routes. ### Scope and limitations Standalone single-region per table, multiple scan partitions. Arithmetic and admitted one-to-one matching only. No distributed Join/Exchange, cross-region routing, group_left/right, set operators, native subqueries or multi-field expansion. No full workspace suite or performance benchmark is claimed. Based directly on narthex, independent of the global-TopK PR. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the matching labels. |
||
|
|
6ca3d32090 |
feat(promql): finalize topk over globally merged aggregates (#9972)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## What's changed and what's your intention? Enable vmbench-lite 09, `topk(10, sum by(hostname)(rate(...[5m])))`, through the existing native aggregate-finalize path. Split TopK/BottomK into the global suffix above fully merged aggregate values. Other vector operations retain their existing admission behavior. No new physical operator or persisted/wire format is introduced. Physical shape: `SeriesScan(N) -> FusedSeries(N, partial sum) -> merge/coalesce -> AggregateFinalize(global TopK) -> rows`. TopK is never partition-local. The standalone regression uses one region, four nonempty scan partitions, 28 series and two evaluation timestamps with different winning hosts. It compares complete native/legacy results and independently expected values, checks producer/finalizer plans, and records original partial-state streams to prove the same hostname contributes from distinct partitions at both timestamps. Partition output counts were [14,14,14,14]. ### Verification - `make fmt-check` - `cargo check --locked -p promql -p tests-integration --all-targets` - `cargo nextest run --locked -p promql -E 'test(aggregate_finalize)'`: 28 passed - `cargo nextest run --locked -p tests-integration --lib -E 'test(native_aggregate_test)'`: 2 passed - `cargo clippy --locked -p promql --all-targets --all-features -- -D warnings` - Independent source review approved; main-agent targeted execution checks passed. ### Scope and limitations Standalone single-region, multiple scan partitions only. No distributed Exchange, cross-region merge/routing, or new matching/subquery support. No full workspace suite or performance benchmark is claimed. This PR is independent of the hybrid-binary PR and is based directly on narthex. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the matching labels. |
||
|
|
e7a9badd29 |
perf(promql): read native inputs and scan partitions concurrently
A multi-input FusedSeriesExec drained every partition of its first input
before it touched the second, and RawSeriesSourceExec drained its scan
partitions one after another. The series distributor feeding those
partitions then waited on the one partition being read, the two inputs'
scans never overlapped, and all decoding ran on the one consuming task.
Read every partition through SeriesGather instead. It polls all of them
and hands a chunk on unless another partition of the same input has a
series open; such a chunk is held until that series closes, so each
input still sees contiguous series. A partition holding a closed series
is not polled again until it is handed over, which bounds what is held
to one series per partition, and that is charged to a memory
reservation. Inputs do not constrain each other.
When there is more than one partition, each is driven on its own query
task with a two-chunk channel, so the scan conversion, frame codec and
label decode of different partitions run in parallel. Dropping the
stream aborts those tasks, and a task that panics ends its partition in
an error rather than a silent end.
On the o11ybench vmlite/vmlong data (standalone, native ON, median of 10
alternating runs; results byte-equal to the sequential reader):
sequential* this change
vmbench-lite-05 4.4ms 3.0ms
vmbench-lite-06 7.3ms 4.2ms
vmbench-lite-08 5.0ms 3.3ms
vmbench-long-range-03 54.4ms 39.6ms
cpu same-table pair 27.6ms 15.3ms
long fs same-table pair 104.2ms 83.6ms
* with the series distributor fix; without it the sequential reader
stalled for up to 12.8s on these queries, and this change does not
need that fix.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
|
||
|
|
c9423f0193 |
fix(promql): decline cross-table outputs the native node cannot name
Under `or` the right table's tags join the left side's columns, and one of them can share a name with the left side's field or time index; building the output schema then failed instead of leaving the query to the existing plan. Such a union is now declined. The output-label walk also declines an aggregate or whole-vector node instead of passing its input's labels through, which would describe the wrong schema if one were ever reached. Adds an end-to-end case over two tables whose tags, field and time index are all named differently, partly flushed and partly in the memtable, under both series scan modes. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
4ce757b611 |
feat(promql): run cross-table selector pairs natively
A binary or set operation between selectors of two tables, such as `rate(a[5m]) / rate(b[5m])`, fell back to the existing plan: the native route required every input to share one metric, time index and tag list, and a fused node is placed inside the one region it reads. Two tables are two regions, so such a pair now runs on the frontend. Each input becomes a `RawSeriesSource` that rides in its own merge scan: the region binds its scan to that table's metric, tags and fields, decodes the labels and answers one explicit-label raw frame stream, reading its scan partitions one after another so each series stays contiguous. The fused node consumes those frames directly. Pairs on one table keep running in the region; partitioned tables keep the existing plan. A point-wise wrapper around an admitted pair, such as `100 * (1 - a / b)`, is admitted too, on one table and across two. The output labels are now derived from the whole tree with each input's own tags, and the physical node takes them from the output schema instead of narrowing by the root alone. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
6b7d1bca4c |
test(promql): strengthen hybrid binary review regressions
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
1c6c288936 |
test(promql): address global topk review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
45852dbf67 |
feat(promql): compile and run set operators natively (#9970)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring (#25): set operators. The kernel came in #9679; the two-input route comes from #9870 / #9871 / #9896. ## What's changed and what's your intention? `and`, `or` and `unless`, with `on(...)` / `ignoring(...)`, now compile into the native tree and run natively. Before, the compiler refused them and the query fell back to the legacy plan. - **Plan.** New `NativeExpr::Set { left, right, op: BoundSetOp { op, labels } }`. There is no cardinality, `bool` or fill field, because Prometheus rejects those modifiers on set operators. Validation requires two evaluated vectors on the same grid; the output takes the left side's fields. - **Runtime.** `BinaryRuntime` runs either kernel (`PairKernel::{Binary, Set}`), so a set node gets the same operand retention, assembly, reservation, queueing and contiguity checks as vector–vector binary. Errors name the set node. - **Kernel.** `set_vectors` charged no memory. `set_vectors_with_reservation` now charges it the way the binary kernel does: output schema, filter-side group table, kept rows, the sort phase, error text and the merge. Membership is decided from one presence bitmap per filter match group (one bit per grid point), so each kept sample is an O(1) lookup. Before, a coarse `on()` scanned every filter candidate at every kept point, O(K·P·F). Results are unchanged; the Prometheus golden set cases still pass. - **Compiler.** `set_operation` binds the operator and the matching labels, and nests on either side and above. It refuses a fill value: promql-parser accepts one, but Prometheus rejects "filling in missing series not allowed for set operators". It also refuses any side that is not a single field. Legacy refuses multi-field set operations (both sides for `or`, the left side for `and` / `unless`), and a per-field presence rule is not defined by Prometheus, so those queries keep the legacy plan. - **Planner.** Same-table set operations take the existing two-scan route, where `FusedSeriesExec` reads every partition of both scans into one execution, so a set operation sees every series of both sides. - **Partitioned tables.** A two-selector native plan on a table with partition columns failed in `DistPlannerAnalyzer` ("only accept plan with at most one child") instead of falling back. This already happened for arithmetic pairs such as `x{kind="a"} - on(host) x{kind="b"}`, and set operators would have hit it too. The planner now declines two-selector plans on such tables, so both keep the legacy plan. Not covered yet: set operations across two tables (`rate(x[5m]) and on(job) up`), and those with aggregate operands (`sum by (job)(a) or sum by (job)(b)`). Both still use the legacy plan. The presence bitmaps take `groups × grid points / 8` bytes. That can exceed the filter samples only when groups are very sparse (under about 1/64 occupancy); such a query is refused by the memory pool rather than running over budget. Allocating each bitmap over its group's position span would tighten this, and is left for later. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
f882c1182d |
refactor(promql): reuse prepared native operand context
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
9270b1ef7c |
test(promql): narrow global topk partition recording
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
78dadff86d |
test(promql): pin per-step or matching through the native entry
A range query `a or on(host) b` where one host's left series covers only the middle steps: native must keep the left sample where it exists and take the right sample with its own labels at the other steps, matching the explicit rows and the legacy plan. The set test's row reader moves to module level so both cases share it. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7f545344c4 |
fix(promql): hold the labels of queued results until they are handed on
A node that reads its whole input (binary, set, vector) queues the series its kernel produced and settles its reservation on the queue. Each queued result was held for `evaluated_bytes`, which covers the chunk and its fields but not the label set the chunk owns, so once the operands were released the reservation dropped below what the queue actually held. A set over many long-labelled one-point series made the gap large. Hold each result that does not share the input for `series_bytes`, the complete owned estimate including labels. The figure stored with the entry is still the one released when it is handed on. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
c25b6a9fbd |
feat(promql): fuse independent operands of hybrid binary queries
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
9d281b401f |
perf(promql): decide set membership from a per-group presence bitmap
The set kernel checked every filter series of a kept series' match group at every kept point, so a coarse `on()` / `on(...)` cost O(kept series x points x group size). Prometheus decides a step from the set of signatures present on the filter side. Build one bitmap per filter match group, one bit per grid point, from the filter side's rows, and look each kept sample up in O(1). Each bitmap is charged to the walk phase when its group is first met and is released with the group table before the sort phase. Results are unchanged. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
c24997e543 |
fix(promql): leave multi-field set operations to the legacy plan
The legacy planner refuses `and`, `or` and `unless` over several fields (MultiFieldsNotSupported), and Prometheus has one value per sample, so per-field presence has no reference semantics. The compiler now declines a set operation unless each side has exactly one field, and the query keeps the legacy plan and its error. Errors raised by the shared two-operand runtime now name the set node instead of calling every such node "the binary node". Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
cef469b450 |
fix(query): keep two-selector native plans off partitioned tables
A two-scan FusedSeries over a table split into several regions has no placement: DistPlannerAnalyzer rejects it with "only accept plan with at most one child", so the query failed instead of falling back. This hit the existing arithmetic pair and, since set operators now compile, set operators too. Decline the native route for two selectors when the table has partition columns. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
1be0df046c |
feat(promql): compile and run set operators natively
Bind `and`, `or` and `unless` (with `on`/`ignoring`) to a new `NativeExpr::Set` node instead of refusing them. The node runs the existing MATCH-SET kernel through the binary runtime, which retains both operands until the input ends, so a set operation reads every series of both sides in one execution. The set kernel now reports its workspace like the binary kernel does: kept rows, the filter side's match groups, the sort and the merge are charged to the node's reservation. The compiler refuses a fill value on a set operator, which Prometheus rejects and the parser accepts, and sides with different field counts. Same-table set operations over two scans take the existing two-scan native route. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
db5e31a005 |
feat(promql): finalize topk over globally merged aggregates
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
1cab5f1624 |
feat(promql): add a series frame output to the fused native node (#9943)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - **Stacked on #9942** (PR A, the unified SeriesFrame codec). - GitHub cannot target a fork branch, so this PR's base reads `narthex`. **Review #9942 first.** - The commits unique to this PR are `b32573f9cbf`, `16b3e428951`, `d5547c55bdf` and `f2041d8d9de` (test-only lint cleanup). The chain is `narthex ( |
||
|
|
f2041d8d9d |
test(promql): keep the frame output tests lint-clean
The workspace denies the default Clippy warnings, and the new tests tripped three of them without changing what any of them proves: - an expected value was built through `scale as f64` even though the tuple already carries an `f64`, so the conversion is spelled in the literal and the cast is gone; - the standalone replay printed the executed plan and the per-partition counts even though the surrounding assertions already require four nonempty partitions, 24 rows in total and eight closing frames, so the diagnostics are gone rather than exempted from the workspace lint. No production code, assertion or fixture changes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
949eeeff38 |
fix(promql): stream evaluated frames with batch-level backpressure
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
e3c6193e57 |
test(promql): pin the frame output's boundaries and its pending hand-over
The frame output was covered for its happy path and for the plan-level decisions, but three things the spec asks about had no test of their own at this boundary, and one test did not prove what it claimed. The pending test is the one that mattered. It awaited the stream, and `StagedStream` wakes itself right after reporting pending, so the await ran straight through the boundary: every later assertion held whether or not the buffered rows were handed over at the pending poll. It now polls that single poll by hand and requires it to answer the rows it already encoded. Deleting the `flush_frames` call from the pending branch makes it fail, which it did not before. The three boundaries now have their own tests: a chain that closes with no position survives as one row where the row table answers nothing at all; a sparse series carries exactly the positions the row table answers and no filled hole; and two chains that close on one label set are both kept, with values that differ so a lost chain would show. Also drops the `as_str` helper, which only asked whether to show a format and then spelled the same string twice, and corrects two comments: `convert` produces three outputs rather than two, and a built plan is not a promise that execution cannot fail. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
2f173c58bd |
feat(promql): add a series frame output to the fused native node
The fused node answers one way: a row table with the grid timestamps, one column per series field and one per declared tag. That is what a query returns, but a consumer of the native path needs to tell series and positions apart, and the row shape flattens both away - which series a value belongs to, and which grid positions that series actually holds. This adds a second output format, fixed when the plan is built rather than chosen at run time: PlainRows the existing row table, unchanged and still the only one a query reaches SeriesFrame the same evaluated chunks handed over as the unified series frame The format is part of the plan because it decides the node's output schema, so it takes part in the node's equality, its EXPLAIN text and its rebuild over new children. A row-table plan prints exactly what it printed before; a frame plan says so on its own line. A frame is encoded straight from each evaluated chunk, never by flattening it into rows and packing it again, so a sparse series keeps its runs and a closing chunk with no position is still one row. The output keeps one partition per input partition: a frame is not a reason to coalesce, and no exchange is inserted in either format. The partial-state output that a complete aggregate reads is a third thing again, and the format does not reach it. When a node exports partition-local state, the format is `PlainRows` by construction, `convert` hands every chunk to the fold before any format is consulted, and the EXPLAIN line keeps its `partial state:` suffix. Nothing in the aggregate path is rewritten to make room for frames. A tree whose root is not evaluated samples is refused while the plan is built, because aggregate state is a contribution to a later merge and a scalar has no series to name. So is a frame output over more than one raw source, which this node does not run in parallel yet. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
eeca48bc43 |
refactor(series): unify raw and evaluated SeriesFrame codec (#9942)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Spec: Narthex Unified SeriesFrame refactor, PR A of two. - #9914, #9868, #9937, #9896, #9933, #9936, #9882 and #9886 are all merged, so this is a plain increment on `narthex` with no stacked dependency. ## What's changed and what's your intention? SeriesFrame is the internal transport of the Narthex native PromQL path, and it could only carry raw samples: an evaluated result had to fall back to a plain row table. This makes **one self-describing Arrow format carry both**, so a receiver decodes a frame from the frame alone, without the scanner, region, or planning context its producer had. What is unified is the **physical representation and its codec**, not the runtime data structures. `RawSeriesChunk`, `Series` and `EvaluatedChunk` are unchanged; only the adapter boundaries branch on the sample shape. ### Base - base: `narthex` @ `771347df5d9` - HEAD: `8bc53c48714` - **no stacked dependency**: everything this PR built on has landed This is a plain increment on the current `narthex` tip, with focused follow-up fixes for review and CI. Nothing from the earlier stack is carried: #9914 is merged, so no `MetricName` rename appears here. The codec itself, the adapters, the tests and the two frozen wire fixtures are all that moved, plus the call sites that had to follow the renamed API. ### The format (`layout = v2`) One frame family, three legal combinations. Every other combination is refused on both the writing and the reading side: | content | identity | time | producer | | --- | --- | --- | --- | | `raw` | `key` | `timestamps` | a storage raw-series scan | | `raw` | `explicit` | `timestamps` | a raw input whose labels are already decoded | | `evaluated` | `explicit` | `grid_spans` | a compute node's output | `evaluated` + `key` is refused because a computation may rewrite its labels, so it cannot claim a storage key as its identity. `raw` + `grid_spans` and `evaluated` + `timestamps` are refused because raw samples have no grid and computed values have no timestamps of their own. Column layout (`series_frame_schema` in `src/datatypes/src/series/frame.rs`): ```text read_key: Binary (NOT NULL for key mode; NULLABLE for explicit mode) is_last: Boolean NOT NULL time: Struct<timestamps: List<Int64>> // raw Struct<spans: List<Struct<start: u64, len: u64>>> // evaluated fields: Struct<<field_name>: SamplePayload, ...> name: Utf8 NULLABLE // explicit only, __name__ labels: List<Struct<name, value>> // explicit only, no __name__, no duplicates ``` `read_key` must be populated for every raw row, including raw/explicit, and must be NULL for evaluated rows. `SamplePayload` (float / native histogram / mixed) keeps its existing representation and is not redesigned. Protocol metadata lives under `greptime.series.*`. `SeriesFrameDeclaration::parse` reads the layout, identity, time and ordering declaration. Schemas explicitly declare `contiguous` by default; raw streams may declare `read_key_then_time`, while evaluated streams accept only `contiguous`. Missing ordering, unknown values and unknown reserved keys are refused by declaration parsing, direct decoding (including empty batches), and stream admission. | key | value | | --- | --- | | `layout` | `v2` | | `content` | `raw` / `evaluated` | | `labels` | `key` / `explicit` | | `time` | `timestamps` / `grid_spans` | | `grid` | required for evaluated, forbidden for raw; canonical `GridSpec` JSON | | `label_decode` | required for key, forbidden for explicit | | `ordering` | `contiguous` / `read_key_then_time` | ### Time axis - raw: every row carries its own strictly increasing `i64` millisecond timestamps. - evaluated: the `GridSpec` is declared once in schema metadata (`{start_ms, step_ms, len}`), and a row carries only the **absolute grid positions** as spans. A sparse grid is never expanded and its holes are never filled; `start + len` is checked against the grid with checked arithmetic. - A zero-sample closing chunk is legal for evaluated and **is kept** as one frame row with zero positions and `is_last = true`, because dropping it would drop the finality it exists to carry. A zero-sample chunk that closes nothing is refused, and a raw frame still refuses an empty chunk. ### Series chains and `is_last` `is_last` closes **one chunk chain**, not the end of a PromQL label set. After a native label rewrite, two different input series can produce **the same complete labels**, and each closes itself. The writer allows that and the reader does not treat "same labels" as a re-opened series. A chain that changes identity before closing, and EOF with an open chain, are both errors. ### Key code locations - `src/datatypes/src/series/frame.rs`: shared schema, declaration parser, bounded builder, cursor and raw/evaluated decoder. - `src/common/query/src/series_frame.rs`: typed pack/unpack adapters and chain validators. - `src/datatypes/src/series/time.rs`: canonical `GridSpec` serialization. ### Review and CI fixes - Repaired the two table PK-binding fixtures with real `host` and `rack` Tag columns and matching primary-key metadata. Production `LabelDecodeSpec::validate()` stays strict. The mismatch test retains all six cases, including changed tag order, and checks rejection before polling. - Removed the missing-ordering fallback and added one private ordering validator shared by the declaration parser and direct decoder. Invalid evaluated ordering cannot bypass stream checks. - Compatibility metadata is tested exactly as written, without merging canonical metadata into malformed inputs. The direct-decoder matrix covers raw/evaluated, empty/nonempty, missing, unknown and interleaved ordering, and valid raw versus invalid evaluated `read_key_then_time`. - Independent source and anti-over-design review: **APPROVE** for `771347df5d9..8bc53c48714`. Review approval is separate from the CI status of the new head. ### Limitations - Not reachable from a query yet: `FusedSeriesExec` still answers `PlainRows` by default. The second output format is PR B. - Not verified: Windows; the results below are Linux. - Deliberately **not** done, per the spec: evaluated **input** wiring in `FusedSeriesExec`, Complete Aggregate's SeriesFrame input, Partial/Final physical plans, cross-region exchange, remote NativeTree serialization, NodePath rework, #9892 storage-interface changes, #9844 remote read wiring. - The unreleased v1 frame format is **not** read. In-tree producers, consumers, tests and the two frozen wire fixtures all move to v2 together. ### Does this break API or data compatibility? No published format changes. SST, WAL, manifest and the generic Flight/Protobuf formats are untouched. The only versioned thing here is the unreleased internal SeriesFrame layout, which goes from `v1` to `v2` in one step and is not read in both directions. ### Tests The existing tests include real pack → batch stream → unpack round trips, not only schema snapshots: - raw/key with its decoded PK, raw/explicit with complete labels, both across batches and through Arrow IPC - evaluated dense, sparse (multiple spans, holes not filled), multi-field (a position can hold a value in one field and nothing in another; NULL ≠ NaN), native histogram and mixed payloads (payload kind and values survive, no silent float-ification) - evaluated empty closing chunk kept as one row; multi-batch streams keeping order and finality; two consecutive chains with equal labels each closing independently; EOF with an open chain - malformed frames refused: non-increasing timestamps, overlapping/out-of-order/out-of-bounds/zero length spans, payload length mismatch, span-count overflow, missing grid, malformed `GridSpec`, missing label decode, contradictory metadata, wrong layout version, an evaluated row carrying a storage key, an empty row that closes nothing <!-- CURRENT_VERIFICATION_START --> Verified on Linux at `8bc53c48714`; every command below exited 0: ```text make fmt-check -> clean (rustfmt and repository source checks) cargo nextest run -p datatypes -p common-query -p table --retries 0 -> 690 passed, 0 skipped; includes both reported CI failures cargo nextest run -p mito-codec -p promql -p query --retries 0 -> 2055 passed, 2 skipped cargo nextest run -p mito2 -E 'test(engine::scan_test::test_mito_handle_query_raw_series) | test(engine::scan_test::test_two_phase_native_raw_stream) | test(engine::scan_test::test_two_phase_raw_) | test(engine::scan_test::test_mito_raw_series_)' --retries 0 -> 8 passed, 1733 skipped (focused raw-scan selection) cargo check --workspace --all-targets -> passed cargo clippy -p datatypes -p common-query -p table -p mito-codec -p promql -p query --all-targets -- -D warnings -> passed (scoped crates, default features) ``` License headers, TOML formatting and `git diff --check` also pass. These are local Linux results, not a claim that the new-head GitHub checks are green. Full `make test`, workspace all-features Clippy and Windows execution were not run locally in this round. <!-- CURRENT_VERIFICATION_END --> ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
23dc866868 |
fix(test): port MinIO conditional-write test fix to narthex (#9930)
fix(test): avoid MinIO connection reuse after conditional writes (#9877) Signed-off-by: WenyXu <wenymedia@gmail.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: Weny Xu <wenymedia@gmail.com> |