mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
3ba4476b2f04f8a6762b85dfa18ef98889471fbd
6676
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dennis Zhuang
|
3ba4476b2f |
chore: merge feat/narthex-select-compile into feat/narthex-histogram-quantile-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
bd719b0859 |
chore: merge feat/narthex-select-absent-sort into feat/narthex-select-compile
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
f067b252da |
fix(promql): read the grid length from its spec now that timestamps are deferred
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
05442cc722 | chore: merge feat/narthex-select-node into feat/narthex-select-absent-sort | ||
|
Dennis Zhuang
|
4976ffa9b9 |
chore: merge narthex into feat/narthex-select-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs # src/promql/src/native/execute/tests.rs |
||
|
dennis zhuang
|
c209feda3c |
feat(promql): count native histograms in count_values by their Prometheus text (#9855)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native operator kernels (#27). Builds on the Go float formatting from #9840 (merged). Removes the histogram caveat noted in #9854. ## What's changed and what's your intention? Prometheus `count_values` counts a native histogram sample under the text `FloatHistogram.String()` writes it as. The native kernel refused histogram input with `NotImplemented`; it now counts histograms the same way. - `go_format::format_float_g`: Go's `%g` / `%v` for a float64 (shortest digits, exponent below 1e-4 and from 1e6 on). `format_open_metrics_float` now builds on it, unchanged. - `histogram_eval::histogram_string`: Go `FloatHistogram.String`. Count and sum come first, then the non-empty buckets from the lowest up (negative, zero, positive), each written as `Bucket.String`. Bounds stay as the bucket index gives them (not clipped by the zero bucket), with the inclusiveness `baseBucketIterator.at` gives, so custom buckets print `[-Inf,b0]` first. - `count_values` keys a histogram by its text, so equal histograms share a group. Series are validated with `map::validate_grid_series` (from #9845), which accepts any sample type. ### Tests - Golden `histogram_eval/string_cases.txt` (17 KB, 47 cases) from `promql-compat/cmd/narthex-golden-histstring`, Prometheus v3.15.0. Fixed cases cover the zero bucket at threshold 0 (`[-0,0]`), negative bucket order, the exponent form, custom bounds, ±Inf and NaN, a zero-length span, a NaN bucket count, and buckets under a wide zero threshold; 40 random cases follow. Dropping the negative-bucket reversal or the custom-bucket inclusiveness fails it. - `count_values` groups two equal histograms together and a float apart. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 discord9
|
14f0ead84c |
feat(promql): execute constant scalar broadcasts in the native executor (#9859)
Execute compiler-emitted literal Scalar/vector arithmetic, filtering and bool comparisons, and unary negation in NativeExecutor. Reuse existing ScalarOperand and map kernels while pulling only the vector side; preserve sparse/missing values, correct labels, and series closure. Reuse existing reservations, metrics and merge handling. Track name-dropping collisions, merge disjoint collapsed series before a downstream Window, and retain no vector stream in the broadcast. Reject unsupported computed-scalar operands at construction. Add real compiler-to-executor and direct executor regressions for both operand directions, folded bool constants, exact 21-point results, streaming/collecting Window lifecycle, nonclosing empty filters, final closure, bounded retention and reservation release. No duplicate compiler lowering, dynamic scalar/scalar-scalar execution, Aggregate runtime or public Fused entrypoint wiring. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 Yingwen
|
cff91d99bf |
feat(mito): implement columnar series index format (#9841)
* feat(mito): implement columnar series index format Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito): remove columnar compatibility fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): use recorded artifact sizes and inline columnar codecs Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): bound columnar series writer memory Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
bf97de2ad1 |
feat(promql): compile subqueries with deferred grid materialization (#9835)
Compile range functions over explicit/default-step, offset, anchored, and nested subqueries using the shared range-evaluator grid calculation. Retain complete sampled coverage and reject empty child grids explicitly. Keep compilation allocation proportional to plan structure, not timestamp count: validate deferred root and child grids; preserve eager EvalGrid::try_new; materialize shared timestamps fallibly only when a consumer requests them. Migrate Rust timestamp-buffer callers to Result without changing wire or persisted formats. Regression coverage includes grid phase/coverage, rounded sampled endpoints, offset overflow, huge and nested compile-only grids, sparse gathering, sharing, and allocation-error propagation. Deferred compilation does not bound execution memory. No public Fused entrypoint or Scalar runtime is added here. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
10dc8940f6 |
feat(promql): bind the remaining range kernels to their PromQL names (#9834)
* feat(promql): bind the remaining range kernels to their PromQL names `RangeFunction` already names 26 functions and every one of them answers: the kernels for `first_over_time`, `mad_over_time`, the four `ts_of_*` functions, `predict_linear` and `double_exponential_smoothing` are all in `range_eval`. What was missing was the compiler's name table, so these queries were refused as names with no kernel although the kernel existed. Six of them are the functions Prometheus marks experimental. This crate implements them unconditionally — `RangeFunction` names them without a gate and the kernels compute them — so binding them is the same decision as binding `rate` rather than a new policy. If a gate is ever wanted it belongs where a query is admitted, not scattered across the compiler's name table. The two parameterised ones needed a second look at where the selector sits, which is not one place in PromQL: `quantile_over_time(0.9, x[5m])` writes its quantile before the selector, while `predict_linear(x[5m], 300)` and `double_exponential_smoothing(x[5m], 0.5, 0.1)` write theirs after it. A single "parameters come first" rule read the selector as a constant for the second group and rejected an ordinary query, so the signature now states the selector's position as data instead of relying on a convention. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
discord9
|
3876831342 |
feat(promql): compile sum over an optionally grouped expression (#9833)
* feat(promql): bind every aggregator that folds a group's values
`AggregateKind` named only `Sum`, so the compiler bound `sum` and refused every
other aggregator by name. Six of the ones it refused fold exactly as `sum` does —
each reports one value per group per grid point — and two more (`stddev`,
`stdvar`) plus `quantile` fold the same way without a kernel yet. Binding them is
the same decision as binding `sum`: the compiler says what an aggregate means, and
the runtime refuses a kind it cannot execute. A kernel landing later then changes no
binding.
`quantile` is the one of the nine with a parameter, and it carries it in the
variant (`Quantile { quantile: f64 }`) as `RangeFunction::QuantileOverTime` does,
because `BoundAggregate` has no field a constant could go in and a field that meant
nothing for the other eight kinds would split a kind from its own parameter. That
costs `AggregateKind` `Eq`, which nothing on this tree needed: the only exhaustive
match is the display, and no collection is keyed on it.
The five that do not fold are refused with the reason rather than mapped to a kind
that would compute something else: `topk`, `bottomk`, `limitk` and `limit_ratio`
select some of the input's series, so they do not report one value per group per
grid point, and `count_values` writes one output label per distinct value.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit f1d1f3124ede003ae6df18fa0617aa13a56f3907)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
# Conflicts:
# src/query/src/promql/native_compile.rs
# src/query/src/promql/native_compile/tests.rs
* test(promql): keep folded argument regressions in their compiler layer
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
|
||
|
discord9
|
70201c9cf9 |
feat(promql): compile constants, negation and scalar-vector arithmetic (#9832)
* feat(promql): compile a constant, a negation and a scalar-vector operation Three shapes reach the tree now: a constant binds as a `Scalar` on the statement's grid, a negation binds as the multiplication by `-1` it is, and a binary operation between one vector and one constant binds as a `Binary` node whose output is the vector side's own binding. The arithmetic and comparison operators all map, and `return_bool` is carried as its own field so `x > 0` (which filters) and `x > bool 0` (which reports zero or one) are one node with one field changed. The operands keep the order they were written in, because most operators are not commutative: `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them apart. A constant is bound by the same `constant_value` fold a call's parameters already use, so `x * (1 + 2)` carries `3` rather than an addition the tree would have to know how to run. Three shapes are refused rather than approximated. An operation between two vectors needs each `Input` to name the selector it reads, which the one binding does not carry, so `a + b` and `a + a` would compile to the same tree; a set operator combines two vectors' series and has no node at all; and a statement whose value is a scalar has nothing to scan, so a constant is only bound next to a vector that gives it something to broadcast over. Each refusal names what it found. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
Dennis Zhuang
|
c10159cf10 |
chore: merge narthex into feat/narthex-count-values-histograms
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/map.rs |
||
|
dennis zhuang
|
0b6ba623f0 |
feat(promql): partial state and merge for native float aggregates (#9826)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task AGG-STATE. ## What's changed and what's your intention? This adds partial aggregation state for native float aggregates, so an execution partition can fold its series and a later stage can merge the states into the result. Frame encoding, Exchange and plan wiring are not part of it. - `FloatAggregator::finish_partial()` returns a `PartialAggregate`. Its groups are in first-seen order, and each group holds per grid point one `Option<AccumulatorState>` per field; `None` means the field had no contribution there. - `AggregateOp::state_fields()` declares which components each aggregation's state carries. A merge reads nothing else: - `sum`: value, compensation. - `count`: count. - `avg`: value, mean, compensation, count, and the incremental-mean flag. - `min` / `max`: value. - `group`: nothing. Its result is 1 wherever a state exists, so `finish` no longer reads the value. - `stddev` / `stdvar`: value (sum of squared deviations), mean, count. - `FloatMerger` merges partials in any order and identifies groups by their full label set. It rejects a partial with another aggregation, another field count, or points outside the grid, and leaves its state unchanged. - Merge rules, compared with the serial fold: - `sum`: the error of adding the two partial sums goes into the compensation, together with both compensations. - `avg`: sums are added while the result stays finite. Otherwise the merge continues with the count-weighted mean, as the serial update does once its sum overflows. - `stddev` / `stdvar`: Chan's pairwise update, computed as `δ·(δ·w)`, so `[0]` merged with `[1.5e154]` stays finite. - `min` / `max`: the serial comparison. For equal `0` and `-0`, the partial merged first wins. - `merge_count_values` adds the counts of `count_values` partials that share a full output label set. `quantile` has no partial and stays on the complete path. Regrouping changes the rounding, and it can also move an overflow of finite values: a serial `sum` of `[1e308, 1e308, -1e308]` is `+Inf`, while the merge of `[1e308]` and `[1e308, -1e308]` is finite. `stddev` / `stdvar` can also differ beyond that when samples differ by about the rounding of their mean: Prometheus' serial Welford rounds the mean per sample, and for `[1e16] | [1e16 + 2]` it gives a variance of 2 where the merge gives the exact 1. Explicit NaN and infinite inputs give the same class as the serial fold. The module docs state this contract. ### Tests - **Random partitions.** 300 seeds × 8 aggregations, plus `count_values`: 1–3 fields of sparse input, split into random partitions and merged in a shuffled order. Before merging, each partial is stripped to its `state_fields`. Results must match the serial fold: - groups, points and fields exactly; - values within 1e-10; - bit for bit when one partition holds every series. - **Directed cases:** - compensation carried by a partial and created by a merge, for `sum` and `avg`; - `avg` with unequal counts after one partition switched to a mean; - `stdvar` of `[0]` and `[1.5e154]`; - `min` / `max` sign of zero following merge order; - rejected partials. A mean-mode compensation is at most the last bit of a mean near 1e307, so dropping its weights is not observable within this contract and is not pinned by a test. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
99bf6a8403 |
feat(promql): add histogram_quantiles and share Go float formatting (#9840)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task NEW-MAP-FN, remaining part. ## What's changed and what's your intention? Adds the native kernel for the experimental `histogram_quantiles(v, label, q...)` from Prometheus v3.15.0. - `histogram_eval::histogram_quantiles` evaluates `histogram_quantile` for each `q` with the existing kernel and sets `label` to `q`, formatted as Prometheus' `labels.FormatOpenMetricsFloat` does. Results that then share a label set merge through `merge_same_labelset`. A repeated `q`, or a `label` that overwrites the label telling two histograms apart, fails at a point both hold, as in Prometheus. - The Go float formatting from #9799's `count_values` moves to a `go_format` module, next to `go_sort`. It now offers both `FormatFloat(v, 'f', -1, 64)` and `FormatOpenMetricsFloat`, which is Go's shortest `'g'` format: an exponent below 1e-4 and from 1e6 on, at least two exponent digits, `.0` appended to integers, and hardcoded `0.0`, `1.0`, `-1.0`. Both share the round-trip-checked shortest digits. ### Tests - `histogram_eval/quantiles_cases.txt` (37 KB, 30 cases): classic histograms evaluated by the Prometheus engine with experimental functions on (promql-compat `narthex-golden-histogram -quantiles`). It includes repeated quantiles and `job` as the quantile label; 13 cases end in the label set collision. - `go_format/open_metrics_cases.txt` (13 KB, 358 values): `FormatOpenMetricsFloat` outputs for the exponent switch points, the hardcoded values, powers of two from 2^-30 to 2^30 with neighbours, and random values (`narthex-golden-formatfloat -openmetrics`). The existing `'f'` fixture is unchanged. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
6afb0283cd |
feat(promql): project selected sample times for timestamp() and info() (#9836)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task EVAL-TS. It also provides the input #9829's `info()` kernel expects. ## What's changed and what's your intention? `timestamp()` needs the original time of the sample an instant selection chose. `select_instants` already returns the source row of each selected point, so this adds kernels that project from that selection onto the evaluation grid, next to `select_instants` in `instant_eval.rs`: - `timestamp_of_selection`: `timestamp(selector)`. The value is the original time of the selected sample in seconds. An offset or `@` changes which sample is selected, not the time reported. Metric name, `__type__` and `__unit__` are dropped. - `info_sample_times`: the series `info()` reads. The value is the original time in milliseconds, as Prometheus' `evalSeries(..., recordOrigT=true)` records it. Labels are kept. A native histogram fails with Prometheus' "this should be an info metric, with float samples". - `timestamp_of_evaluated`: `timestamp()` over an evaluated expression such as `timestamp(abs(x))`. The value is the evaluation time in seconds where the input has a value. Metadata labels are dropped. A field without a sample in the selected row has no value at that point. Lookback and stale handling stay those of `select_instants`. A selection must lie on the grid, with points and rows increasing; that lets one cursor per field read each selected row once, even when consecutive points select the same row. Out of scope: wiring into the Instant runtime and the native compiler (including `@` binding). ### Tests `instant_eval/timestamp_cases.txt` (22 KB) comes from the Prometheus engine via promql-compat `narthex-golden-timestamp`. Samples sit at arbitrary milliseconds, so original and evaluation times differ. It has 40 random cases plus fixed ones for: - a sample exactly one lookback before a point (excluded); - a stale marker; - an offset, over a selector and over an evaluated expression. Unit tests cover: - the millisecond output, labels and missing fields of `info_sample_times`; - histogram samples (rejected for `info`, accepted for `timestamp`); - rejection of selections off the grid or going back. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
43fdb88124 |
feat(promql): time-varying round/clamp parameters and histogram-skipping math (#9845)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task MAP-MATH, the two remaining gaps: time-varying `round` / `clamp*` parameters, and math over native histogram input. ## What's changed and what's your intention? **Time-varying parameters.** Prometheus evaluates the scalar arguments of `round(v, to_nearest)`, `clamp(v, min, max)`, `clamp_min(v, min)` and `clamp_max(v, max)` at every step. The existing kernels only bind constants. `map::map_math_with_parameters` takes a `ParameterizedMath` that holds the parameter series, so the number of parameters is fixed by its type. At each point it reads the parameters, builds the `MathOp` of that point, and applies it: - `clamp` with `max < min` at a point has no result there, as Prometheus returns an empty vector for that step; - a point where a parameter has no value has no result; - the metric name is dropped, as with constant parameters. **Native histogram samples.** Prometheus' math functions (`simpleFloatFunc`, `clamp`) process only float samples and skip histograms. `map_math` rejected any histogram or mixed field. Now: - a histogram sample is a point without a result in that field; - floats next to it are mapped as before; - a series with no float left yields `None`. Float-only input keeps the existing chunk-wise path; a field with histograms is rebuilt point by point. `map_math_with_parameters` does the same. Arithmetic and negation still reject histograms: Prometheus does compute those on histograms, and that is outside this change. ### Tests - `map/scalar_parameter_cases.txt` (17 KB, 41 cases): generated by the Prometheus engine via promql-compat `narthex-golden-mathparam`, with parameters as `scalar()` of one series. It covers `max < min` points, NaN / ±Inf parameters, `round` to 0, negative, NaN and ±Inf, and a fixed case for ties (`2.5 → 3`, `-2.5 → -2`, `-0.5 → 0`). - Unit tests: - sparse parameter axes with an absent and a null value, across two fields; - histogram samples skipped by `map_math`, including all-histogram input; - histogram samples skipped by `map_math_with_parameters`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
0279e12889 |
chore: merge feat/narthex-select-compile into feat/narthex-histogram-quantile-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
abb55be608 |
refactor(promql): follow the Vector rename in the compiler
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bf3ae6ab60 | chore: merge feat/narthex-select-absent-sort into feat/narthex-select-compile | ||
|
Dennis Zhuang
|
b7dfb8ec3d |
fix(promql): keep a shared histogram table charged until the last queued result leaves
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
5dbda4d9cd |
chore: merge feat/narthex-select-node into feat/narthex-select-absent-sort
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs |
||
|
Dennis Zhuang
|
90478ec493 |
refactor(promql): name the whole-vector node Vector
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bec5f76637 |
fix(promql): keep the input order for limitk and merge label-set pieces instead of sorting
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bc782ac5da |
feat(promql): count native histograms in count_values by their Prometheus text
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
7e0325780e |
chore(promql): return the group slot directly
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
8bc2c8aa55 |
feat(promql): run histogram_quantile and histogram_fraction in the native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
3d19a1f3fc |
feat(promql): compile sort and absent into the native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
353149cf63 |
feat(promql): run sort and absent in the native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
63a7d814a0 |
feat(promql): run topk, bottomk, limitk and limit_ratio as a native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
d79a5f6bfe |
test(promql): keep Go float format tests with go_format and pin histogram_quantiles constants
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
607719dc9d |
fix(promql): validate instant selections in one pass and format info errors like Prometheus
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
e61a6f42eb |
chore: merge narthex into feat/narthex-eval-ts
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
7c139a2ccc |
feat(promql): skip native histogram samples in math functions and cover review gaps
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
e4d1b79b76 |
feat(promql): evaluate round and clamp with time-varying scalar parameters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
f839acb7e5 |
chore: merge narthex into feat/narthex-histogram-quantiles
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
b968070070 |
chore: merge narthex into feat/narthex-agg-state
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
6f94a869a9 |
feat(promql): add the info() combine kernel to native operators (#9829)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task NEW-MATCH-INFO (kernel part). ## What's changed and what's your intention? Adds the combine step of Prometheus v3.15.0's experimental `info(v, {selector})` (`combineWithInfoSeries` / `combineWithInfoVector`) as a native kernel, `info::info`. The caller selects the info series by info metric name and data matchers, with lookback, `@` and offset. It passes them as single-field series whose value at each point is the original timestamp of the selected sample, as Prometheus' `evalSeries(..., recordOrigT=true)` produces them. The kernel then does everything else `evalInfo` does: - **Info metric names.** Name matchers are completed as Prometheus does: none means `target_info`, and only-negative matchers are prefixed with `.+_info`. A base series that is itself an info metric passes through unchanged. - **Which info series count.** The kernel keeps only those Prometheus would fetch (`infoIdentifyingMatcherSets`), so the caller may select by name and data labels alone. Each identifying-label presence pattern among the base series admits the values its series have; a base series with neither `instance` nor `job` matches nothing. - **Joining.** A base series is matched per info metric on `instance` and `job`. When several info series with that signature are present at a point, they are compared in input order, like Prometheus compares them in its storage order: a newer one replaces the kept one, and an equally new one is an error, but only at points where the base has samples. - **Labels.** Only the data labels named by the selector are copied, or all of them when it names none. Labels already on the base series are kept. Two info metrics setting one label to different values is an error. A base series with no matching info series is kept only when every data matcher matches the empty string. - **Results.** Results sharing a label set merge field by field; one field held twice at a point is an error. Errors follow Prometheus' order: the first failing step, and within a step info duplicates, then label conflicts, then label set collisions. Name regexes use `^(?s:...)$` with the `regex` crate, as `map_label` does. Its dialect differs from Go's RE2 in the same ways (Unicode-aware `\w`, `\d`, `\s`, `\b`). Out of scope: planning and reading the info series. ### Tests `info/prometheus_cases.txt` is generated by the Prometheus engine (promql-compat `narthex-golden-info`, lookback 25s on a 10s step). The base is the engine's own evaluation of the base expression; the info input is the series matching the name and data matchers, with original sample timestamps. - 400 random cases, plus fixed cases for: - the input-order rule (timestamps 5, 5, 6 fail; 6, 5, 5 succeed); - equal timestamps where no base sample is present; - a label set collision reported before a later conflict; - a base without a metric name; - `=~` / `!~` name matchers. - Unit tests: merging by field for multi-field bases, and histogram base samples. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
a5c91db63a |
feat(promql): add stddev, stdvar, quantile and count_values native aggregates (#9799)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task AGG-STAT + AGG-CV. ## What's changed and what's your intention? Adds the native float kernels for `stddev`, `stdvar`, `quantile` and `count_values` to `src/promql/src/aggregate.rs`, on the existing `Grouping` and validation. Semantics follow Prometheus v3.15.0 `aggregation`: - `stddev` / `stdvar` extend `aggregate_float` with the same uncompensated Welford update Prometheus uses. The running sum of squared deviations starts at NaN when the first sample is NaN or ±Inf, as in Prometheus. - `quantile` collects each group's values in input order and reuses the `quantile` kernel from `range_eval`. q is taken per point from a scalar series; a point without q has no result. - `count_values` sets the value label to the sample formatted like Go's `strconv.FormatFloat(f, 'f', -1, 64)`. Rust's shortest `Display` rounds ties up while Go rounds them half-even, so the digits are recomputed with exact formatting. With `by`, the value label is added to the grouping; with `without`, it is not, so listing it there drops it. An empty label name is an error. `FloatAggregator` exposes `aggregate_float` as construct / `update(&Series)` / `finish`, so the native Aggregate runtime can fold series as they arrive instead of buffering all input. `aggregate_float` now runs on it, with unchanged semantics. The aggregator owns its grouping labels and borrows nothing. A series that fails validation leaves its state unchanged. With several fields, each field is aggregated on its own. For `count_values`, a group row has a count only in the fields whose value produced its label. Out of scope: partial/final aggregation state (AGG-STATE) and native tree compilation and runtime wiring. ### Tests - `stat_cases.txt`: golden cases generated by the Prometheus engine (promql-compat `narthex-golden-aggstat`), 12 per operation × {none, by, without, by(i)}, including NaN/±Inf/±0, ties, time-varying and out-of-range q, and `without (…, <value label>)`. - `format_float_cases.txt`: 1810 values with Go `FormatFloat` output, including rounding ties and exponent boundaries. - Multi-field, missing-q and empty-label cases in the existing kernel tests. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
fa9838da65 |
feat(promql): execute vector-vector binary nodes natively (#9813)
* feat(promql): execute a bound native tree node by node, over sparse evaluated series Ports the native runtime onto the sparse evaluated-series API: a chunk is addressed by a `TimeAxis` that may hold several runs, so a consumer reads positions through the axis instead of assuming `start + row`. The driver and both node kinds move to that API: - `chunk.range().len()` becomes `chunk.series.time.len()`, the compact row count, in every metric and byte estimate. - `occupied_runs` walks `time.positions()` with one field cursor, so a chunk of several runs records each run where it belongs. A run open across a hole now ends where the hole begins; leaving it open to the last row made two runs look like one and refused a series that occupies the hole. - A window keeps the pieces its evaluator decided in a queue: one push can decide several runs, and reporting only what the push returned stranded the rest until the next push. Queued pieces are charged by `retained_bytes`, and the charge is recomputed when one leaves, so a released piece is not held. - `map_chunk` answers `Option`: an operation that empties every position (`clamp` with `max < min`) yields no chunk at all, unless the chunk closes its series -- a closing chunk holding no row is handed over so a consumer can tell "nothing more here" from "no more input". `RangeEvaluator::retained_bytes` is restored with the three byte helpers it reads: a field's buffers, a series' identity and the pending batch's vectors, all by capacity, and still counted as storage rather than as a peak over a push. Tests read absolute positions from the axis (`decided` no longer adds a row number to a start), and two sparse cases cover what the old reading got wrong: a `Window -> Map -> Window` chain over a chunk of two runs, and a series that occupies the hole between another's runs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop the window queue the sparse evaluator never needed `RangeEvaluator::push` and `push_series` each return `Result<Option<EvaluatedChunk>>`: the evaluator gathers every run it decided into one chunk and hands back at most one. So `pending.extend(decided)` inserted at most one chunk and `take_pending` removed it on the spot, leaving the queue empty after every successful push. That makes the queue dead weight: a `VecDeque` allocation per productive window, an accounting term that is always zero, a second drain site in `pull`, and doc comments claiming a problem the API does not have. The driver's "ask the input again when a window decides nothing" loop was already the whole story. Back to returning the evaluator's own answer, and drop `pending`, `take_pending` and `take_window_output`. Covers the empty-map protocol this port introduces while it is here: an operation that empties every position hands on nothing while its series is open, and one empty chunk when it closes. A non-closing empty chunk would be rejected downstream and a dropped closing one would leave the series unclosed. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the empty-map output count at one The test checked that every output was empty and that exactly one closed, which a stray extra non-closing empty output would also satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge the memory the native evaluator actually retains Four omissions made the reservation blind to most of what a window over native histograms keeps, and the existing incremental-equals-recomputed test could not see any of them because both sides used the same formula. - `FloatHistogram` owns four heap allocations -- both bucket lists, the custom bounds, and the span layout -- that `FieldSamples::bytes` never looked at. A retained 4096-bucket sample is tens of kilobytes against a ledger that saw a struct header. `heap_bytes` counts them, and both the retained sample slice and the histograms a row boxes are charged for what they own. - Queued chunks were estimated as one float per field per row, which is not what a histogram field holds: it holds indices into a shared `Arc<HistogramBatch>` and the buffers behind the Arrow arrays. The figure now walks the samples, and counts a shared table or a shared buffer slice once per pass rather than once per chunk that references it. - `history_bytes` included `size_of::<History>()` while the map slot term charged a `History` per entry as well, so every entry paid two headers. The per-entry figure now covers only the heap the entry owns. - The primary key is charged for the allocation its `Buffer` keeps alive rather than the bytes it shows: a scan slices the key out of a decoded dictionary-values buffer, so a short key can pin a batch-sized allocation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep the source queue's charge incremental The queue's reservation was recounted by walking every queued chunk on each enqueue and each pop. `push_raw` and `push_evaluated` are public and put no bound on how deep the queue gets, so a caller that queues a whole series before pulling it paid `N(N+1)/2` chunk visits to hold `N` chunks -- the same shape of defect this branch fixes in `absorb`. `QueuedCharge` keeps a reference count and the bytes behind each shared allocation: a table or buffer is charged when the first queued chunk reaches it and given back when the last one leaves, so an operation walks the one chunk it names. The count is per chunk rather than per field, because a pop re-walks the same chunk with the same dedupe and must remove exactly the references that were added. A chunk's own struct bytes are counted per chunk (`own_bytes`); only the allocations behind its fields are counted by address, since a struct has no address to dedupe on. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): append a forward-moving chunk to the emitted history `absorb` rebuilt the whole stored run array on every non-empty chunk. Legal input made that quadratic across chunks with no label collision: one series in `K` chunks, chunk `i` occupying only grid position `2*i`, stores one more run per chunk and moves `K(K+1)/2` of them -- about five billion for a hundred thousand rows. A chunk that starts at or past the end of the last stored run only ever adds to the end, so its runs are appended and the boundary run is joined where it touches. One that can interleave or reach backwards still takes the full one-pass merge, which can split a run. Both leave the runs sorted, disjoint and joined, which is what `Emitted::occupy` looks a position up by. The test records the chunks through `Emitted::occupy` itself and reads the history back from it, covering the forward case, the touching boundary, a hole that has to join both neighbours, and a position before everything stored. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): report a truncated input from a tree with no window `close` only asked windows whether a series was still open, so a tree of an input alone, or of a map over one, accepted a final chunk without `is_last` and handed the consumer a series that never closed. The input node now follows the series of the chunk it last handed over, and closing such a tree reports the truncation against it, which is the same refusal a window makes for the input it reads. `occupied_runs` read a row the axis names with `expect`, which panics on a `Series` whose fields are shorter than its axis; the fields are public, so that chunk is constructible by a caller. It now returns the misalignment as an error naming the node, the field and both lengths. The chunk protocol says one series closes once, which reads as one close per label set. State that it is per input series: a node that rewrites labels can hand two series over under one label set and both of them close. Also say in `Emitted`'s doc that the table is one node's own, so a label set costs its bytes again in every node that keeps one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute an instant selection as a tree node The tree could not run `NativeExpr::Instant`: the state builder recorded `not_implemented` and refused the whole tree before reading a chunk, while the compiler already produced the node for a bare selector. A query the plan and the compiler both accept therefore had no runtime path at all. `InstantRuntime` buffers one series to its `is_last` chunk -- the selection kernel reads a complete axis -- then calls the shared `select_instants` once and gathers the rows it picked onto the binding's own grid. Positions come from the grid lattice the kernel was asked for, and a timestamp off that lattice is an error rather than a skipped row. A series that selects nothing still closes, with an empty chunk on the grid; an input that ends inside a series is reported on `finish()`. The node does not record label-set occupancy: it keeps the metric verbatim, so it cannot map two series onto one label set, and the occupancy key drops empty-valued tags -- two genuinely distinct series would be refused as a collision. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved input, and close a label set only for its owner Two series-level identity holes, both reported against `42ea525`. The truncation flag followed the last chunk handed over, which is the same as "is a series open" only while at most one series can be open. A tree with no window still accepted `Interleaved`, and `a` open, `b` closed, end of input then passed while `a` never closed. Tracking the open set would need an identity that a raw chunk in bound-label mode does not have (`metric` is `None`, the key holds it), so the ordering is refused when the state is built instead, for every tree shape and not only under a window. The flag's doc now says what it means and why that is the whole answer. `History` closed on any series' `is_last`, so with `abs` dropping the metric name a series that contributed nothing could close a label set another series was still filling: `a{job="x"}` value, `b{job="x"}` empty close, `a{job="x"}` continuation was refused with "already closed the label set". Contiguous input hands over exactly that sequence, so it was not an interleaving artefact. A history now remembers the series that filled it and closes only for that series' closing chunk; a series that produced nothing under an open set is passed over, as it already was under a closed one. The owner is charged as part of the entry. The ordering check also reads every input node rather than only the first: a tree with two inputs would otherwise have its second declaration read and dropped. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve the instant gather's workspace and keep the histogram start Two findings from the review of the instant node. The gather allocates after the last chunk arrives: the concatenated axis, the descriptors that read it, the two arrays the selection returns, and the rows the builder keeps. Nothing asked the pool for any of it, so with one sample selected by every point of a million-point grid the node took tens of megabytes while its reservation fell to the buffered chunks' size -- nothing at all, on the chunk that ends the series. `gather_bytes` now derives a bound from the values in hand (rows, chunks, segments, fields, points, and the widest histogram row per field, since one row can be selected by every point), and the reservation grows by it before the gather runs. It shrinks back on both the success and the error path, so the workspace never outlives the call. A selection reads an existing sample, so it has to carry that sample's payload. `FloatHistogram` had no `start_timestamp` and `push_to` wrote `None`, which lost it for any histogram that had one: `detect_start_timestamp_reset` reads that field to spot a counter reset, so a window over this output would mis-detect resets. The field is now carried through a view and written back; the kernels that compute a histogram instead of reading one leave it `None`, which is what they produced before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): describe the gather test by what this branch does The doc on `an_oversized_selection_is_refused_by_the_pool` said the kernel has no point limit any more. It does on this branch: the limit is removed on its own branch, which this one does not carry, so the comment described a state that has not arrived. What is true here is the order: the estimate now applies to every gather, so the pool is asked before the kernel is, and the pool's refusal is the one the caller sees. That is worth saying on its own -- a bound in bytes on the path that decides the points, rather than a count that says nothing about what they cost -- and it stays true after the limit goes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute vector binary nodes in the native runtime Checkpoint the vector-vector runtime and single shared matching kernel. Keep addressed input delivery, whole-input buffering, nested composition and error cleanup together. The existing arithmetic/comparison implementation remains the only computation path. Local review checkpoint only, not ready for publication: all-targets check passes; the current run has three new fixture failures. Independent review requires current-live workspace resizing, field-local matched storage and kept-value merge budgeting before approval. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): infer the binary result fixture element type Drop a redundant complex type annotation flagged by the actual strict clippy run. The collection and literal assertions are unchanged. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve binary workspace along the shared kernel Remove the separate matching prepass and reserve current live workspace at the existing kernel allocation boundaries. Release per-sample and per-field scratch after ownership ends; charge actual retained rows, values, histograms and errors. Unbudgeted public and set callers bypass accounting without another computation path. Validate with cargo fmt -p promql -- --check; cargo check -p promql --all-targets; cargo nextest run -p promql --retries 0 --no-fail-fast; cargo clippy -p promql --all-targets -- -D warnings. The public nested Binary test also fails on swapped left/right operands and passes after byte-identical restoration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): scope binary reservations to live scans and output rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge binary buffers at actual allocation boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): skip accounting ledger for unbudgeted vector merges Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): simplify native execution memory estimates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): match binary operands by logical series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): cover flattened operand descriptor capacity Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge rewritten logical series before window evaluation Accept disjoint closed-source fragments under one output label set. Retain occupancy checks and merge affected window inputs once at EOF using shared typed sample segments. Keep direct input windows and root maps streaming, and release collected fragments on finish or error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): clarify approximate merge workspace estimate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate series boundaries before collected input merges Keep each collecting window's open metric and last position so EOF grouping cannot hide reversed runs or identity changes before closing. Preserve source and root-map piece semantics, and test malformed input release, empty closing chunks, and EOF truncation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): supply the default binary fill in the unimplemented-node fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): end a subtree before the node above it computes A window that collects its input answers only once its own input is finished, so ending the input now flushes the deepest ready window and lets the following pull carry its answer up; a binary node computes only when both operands are settled. Previously a binary node computed on an empty operand and marked itself computed, dropping what the window produced afterwards. Also stream the answer instead of collecting it: end_of_input() ends the input, and each pull runs the next stage, so the caller pulls the result and the last None closes the execution. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve an instant selection for the points it decides The gather reserved the whole planned grid, while select_instants clips the range to the series' first and last sample and its lookback: a week-long query at a one-minute step reserved ~504 KB for a series that produces five points, and a small pool refused it before gathering. Reserve for the clipped candidate count, derived with the same alignment the kernel uses. Also pin the start-timestamp semantics the histogram change spread to: last_over_time and first_over_time hand back the sample they selected with its start timestamp, and limitk and limit_ratio keep it on the sample they keep, so a reader downstream can still place a counter reset. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): reserve binary memory per phase, not per value The binary kernel adjusted the pool and the metrics once per matched point, per filed row and per merged row, so a million plain float points meant roughly three million reserve calls inside a numeric loop. Reserve at three boundaries instead: the operands as chunks are retained, one figure per phase (pairing, sort, merge) with an estimate over rows/fields/segments/payload, and each result buffer when a block of it is allocated. Delete the code that modelled another component's allocator: index_column_growth and its shadow index_capacity, BinaryOperand::ends_growth, and the flatten-capacity model in assembly_bytes. Nothing pre-reserves the left-times-right product and no large allocation happens before its reservation. Tests that pinned the callback sequence or exact small-object arithmetic go; the resource properties stay: a genuinely large input or output is refused, a sparse small result is admitted, and reservations are released on finish, error, truncation and drop. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): move the histogram_quantile doc back to its own test The start-timestamp test was inserted into the middle of another test's doc comment, leaving the earlier test without one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): refuse an interleaved input under a selection when the tree is bound The merged plan validates an input ordering in the arm that reads it, so a selection over an interleaved input no longer binds at all. Assert that refusal where it happens instead of asserting a runtime rejection the tree cannot reach, and keep the map-over-a-selection case: a map above a selection does not shield the check, because the check looks at the input the selection itself reads. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): say what the start timestamp pins, and cover every path that carries it The field is GreptimeDB sample metadata, so the comments now say that instead of arguing from what Prometheus omits: a FloatHistogram there has no such field, while its rate and reset evaluation takes start timestamps through a separate channel, and this crate's native range kernel does not read the field at all. Comparison filtering and set selection were the two paths that carry the timestamp without a test. Both keep the payload of the operand they hand back, including the right side of an or at a point the left side does not cover, and a bool comparison answers with a float so no histogram reaches the output. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * style(promql): drop the blank line the appended test block left behind Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): drop the start timestamp a computed histogram never had A window holding one histogram became the result as it arrived, while two or more went through an add that clears the timestamp, so whether a summed histogram carried the payload said how many samples the window held rather than what the function did. Clear it where the fold returns, leaving the bucket layout to the caller. Also correct two comments that described behaviour this branch does not have: the selection kernel carries no point limit of its own, and the interleaved map sits above the selection rather than between it and the input. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): bill a binary operand the histogram tables it keeps alive A chunk figure leaves out the tables its segments index into, because a table is one allocation however many chunks share it. The operand then billed only the indices, so tables it kept alive cost nothing: a pool holding room for one table admitted a second and a third, each arriving from a source whose own reservation was released as it handed the chunk over. Bill each table once, to the chunk that first references it, and charge the one slot that remembers it. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
2e7ed9cbcd |
feat(promql): execute an instant selection as a tree node (#9798)
* feat(promql): execute a bound native tree node by node, over sparse evaluated series Ports the native runtime onto the sparse evaluated-series API: a chunk is addressed by a `TimeAxis` that may hold several runs, so a consumer reads positions through the axis instead of assuming `start + row`. The driver and both node kinds move to that API: - `chunk.range().len()` becomes `chunk.series.time.len()`, the compact row count, in every metric and byte estimate. - `occupied_runs` walks `time.positions()` with one field cursor, so a chunk of several runs records each run where it belongs. A run open across a hole now ends where the hole begins; leaving it open to the last row made two runs look like one and refused a series that occupies the hole. - A window keeps the pieces its evaluator decided in a queue: one push can decide several runs, and reporting only what the push returned stranded the rest until the next push. Queued pieces are charged by `retained_bytes`, and the charge is recomputed when one leaves, so a released piece is not held. - `map_chunk` answers `Option`: an operation that empties every position (`clamp` with `max < min`) yields no chunk at all, unless the chunk closes its series -- a closing chunk holding no row is handed over so a consumer can tell "nothing more here" from "no more input". `RangeEvaluator::retained_bytes` is restored with the three byte helpers it reads: a field's buffers, a series' identity and the pending batch's vectors, all by capacity, and still counted as storage rather than as a peak over a push. Tests read absolute positions from the axis (`decided` no longer adds a row number to a start), and two sparse cases cover what the old reading got wrong: a `Window -> Map -> Window` chain over a chunk of two runs, and a series that occupies the hole between another's runs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop the window queue the sparse evaluator never needed `RangeEvaluator::push` and `push_series` each return `Result<Option<EvaluatedChunk>>`: the evaluator gathers every run it decided into one chunk and hands back at most one. So `pending.extend(decided)` inserted at most one chunk and `take_pending` removed it on the spot, leaving the queue empty after every successful push. That makes the queue dead weight: a `VecDeque` allocation per productive window, an accounting term that is always zero, a second drain site in `pull`, and doc comments claiming a problem the API does not have. The driver's "ask the input again when a window decides nothing" loop was already the whole story. Back to returning the evaluator's own answer, and drop `pending`, `take_pending` and `take_window_output`. Covers the empty-map protocol this port introduces while it is here: an operation that empties every position hands on nothing while its series is open, and one empty chunk when it closes. A non-closing empty chunk would be rejected downstream and a dropped closing one would leave the series unclosed. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the empty-map output count at one The test checked that every output was empty and that exactly one closed, which a stray extra non-closing empty output would also satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge the memory the native evaluator actually retains Four omissions made the reservation blind to most of what a window over native histograms keeps, and the existing incremental-equals-recomputed test could not see any of them because both sides used the same formula. - `FloatHistogram` owns four heap allocations -- both bucket lists, the custom bounds, and the span layout -- that `FieldSamples::bytes` never looked at. A retained 4096-bucket sample is tens of kilobytes against a ledger that saw a struct header. `heap_bytes` counts them, and both the retained sample slice and the histograms a row boxes are charged for what they own. - Queued chunks were estimated as one float per field per row, which is not what a histogram field holds: it holds indices into a shared `Arc<HistogramBatch>` and the buffers behind the Arrow arrays. The figure now walks the samples, and counts a shared table or a shared buffer slice once per pass rather than once per chunk that references it. - `history_bytes` included `size_of::<History>()` while the map slot term charged a `History` per entry as well, so every entry paid two headers. The per-entry figure now covers only the heap the entry owns. - The primary key is charged for the allocation its `Buffer` keeps alive rather than the bytes it shows: a scan slices the key out of a decoded dictionary-values buffer, so a short key can pin a batch-sized allocation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep the source queue's charge incremental The queue's reservation was recounted by walking every queued chunk on each enqueue and each pop. `push_raw` and `push_evaluated` are public and put no bound on how deep the queue gets, so a caller that queues a whole series before pulling it paid `N(N+1)/2` chunk visits to hold `N` chunks -- the same shape of defect this branch fixes in `absorb`. `QueuedCharge` keeps a reference count and the bytes behind each shared allocation: a table or buffer is charged when the first queued chunk reaches it and given back when the last one leaves, so an operation walks the one chunk it names. The count is per chunk rather than per field, because a pop re-walks the same chunk with the same dedupe and must remove exactly the references that were added. A chunk's own struct bytes are counted per chunk (`own_bytes`); only the allocations behind its fields are counted by address, since a struct has no address to dedupe on. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): append a forward-moving chunk to the emitted history `absorb` rebuilt the whole stored run array on every non-empty chunk. Legal input made that quadratic across chunks with no label collision: one series in `K` chunks, chunk `i` occupying only grid position `2*i`, stores one more run per chunk and moves `K(K+1)/2` of them -- about five billion for a hundred thousand rows. A chunk that starts at or past the end of the last stored run only ever adds to the end, so its runs are appended and the boundary run is joined where it touches. One that can interleave or reach backwards still takes the full one-pass merge, which can split a run. Both leave the runs sorted, disjoint and joined, which is what `Emitted::occupy` looks a position up by. The test records the chunks through `Emitted::occupy` itself and reads the history back from it, covering the forward case, the touching boundary, a hole that has to join both neighbours, and a position before everything stored. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): report a truncated input from a tree with no window `close` only asked windows whether a series was still open, so a tree of an input alone, or of a map over one, accepted a final chunk without `is_last` and handed the consumer a series that never closed. The input node now follows the series of the chunk it last handed over, and closing such a tree reports the truncation against it, which is the same refusal a window makes for the input it reads. `occupied_runs` read a row the axis names with `expect`, which panics on a `Series` whose fields are shorter than its axis; the fields are public, so that chunk is constructible by a caller. It now returns the misalignment as an error naming the node, the field and both lengths. The chunk protocol says one series closes once, which reads as one close per label set. State that it is per input series: a node that rewrites labels can hand two series over under one label set and both of them close. Also say in `Emitted`'s doc that the table is one node's own, so a label set costs its bytes again in every node that keeps one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute an instant selection as a tree node The tree could not run `NativeExpr::Instant`: the state builder recorded `not_implemented` and refused the whole tree before reading a chunk, while the compiler already produced the node for a bare selector. A query the plan and the compiler both accept therefore had no runtime path at all. `InstantRuntime` buffers one series to its `is_last` chunk -- the selection kernel reads a complete axis -- then calls the shared `select_instants` once and gathers the rows it picked onto the binding's own grid. Positions come from the grid lattice the kernel was asked for, and a timestamp off that lattice is an error rather than a skipped row. A series that selects nothing still closes, with an empty chunk on the grid; an input that ends inside a series is reported on `finish()`. The node does not record label-set occupancy: it keeps the metric verbatim, so it cannot map two series onto one label set, and the occupancy key drops empty-valued tags -- two genuinely distinct series would be refused as a collision. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved input, and close a label set only for its owner Two series-level identity holes, both reported against `42ea525`. The truncation flag followed the last chunk handed over, which is the same as "is a series open" only while at most one series can be open. A tree with no window still accepted `Interleaved`, and `a` open, `b` closed, end of input then passed while `a` never closed. Tracking the open set would need an identity that a raw chunk in bound-label mode does not have (`metric` is `None`, the key holds it), so the ordering is refused when the state is built instead, for every tree shape and not only under a window. The flag's doc now says what it means and why that is the whole answer. `History` closed on any series' `is_last`, so with `abs` dropping the metric name a series that contributed nothing could close a label set another series was still filling: `a{job="x"}` value, `b{job="x"}` empty close, `a{job="x"}` continuation was refused with "already closed the label set". Contiguous input hands over exactly that sequence, so it was not an interleaving artefact. A history now remembers the series that filled it and closes only for that series' closing chunk; a series that produced nothing under an open set is passed over, as it already was under a closed one. The owner is charged as part of the entry. The ordering check also reads every input node rather than only the first: a tree with two inputs would otherwise have its second declaration read and dropped. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve the instant gather's workspace and keep the histogram start Two findings from the review of the instant node. The gather allocates after the last chunk arrives: the concatenated axis, the descriptors that read it, the two arrays the selection returns, and the rows the builder keeps. Nothing asked the pool for any of it, so with one sample selected by every point of a million-point grid the node took tens of megabytes while its reservation fell to the buffered chunks' size -- nothing at all, on the chunk that ends the series. `gather_bytes` now derives a bound from the values in hand (rows, chunks, segments, fields, points, and the widest histogram row per field, since one row can be selected by every point), and the reservation grows by it before the gather runs. It shrinks back on both the success and the error path, so the workspace never outlives the call. A selection reads an existing sample, so it has to carry that sample's payload. `FloatHistogram` had no `start_timestamp` and `push_to` wrote `None`, which lost it for any histogram that had one: `detect_start_timestamp_reset` reads that field to spot a counter reset, so a window over this output would mis-detect resets. The field is now carried through a view and written back; the kernels that compute a histogram instead of reading one leave it `None`, which is what they produced before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): describe the gather test by what this branch does The doc on `an_oversized_selection_is_refused_by_the_pool` said the kernel has no point limit any more. It does on this branch: the limit is removed on its own branch, which this one does not carry, so the comment described a state that has not arrived. What is true here is the order: the estimate now applies to every gather, so the pool is asked before the kernel is, and the pool's refusal is the one the caller sees. That is worth saying on its own -- a bound in bytes on the path that decides the points, rather than a count that says nothing about what they cost -- and it stays true after the limit goes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): simplify native execution memory estimates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge rewritten logical series before window evaluation Accept disjoint closed-source fragments under one output label set. Retain occupancy checks and merge affected window inputs once at EOF using shared typed sample segments. Keep direct input windows and root maps streaming, and release collected fragments on finish or error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): clarify approximate merge workspace estimate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate series boundaries before collected input merges Keep each collecting window's open metric and last position so EOF grouping cannot hide reversed runs or identity changes before closing. Preserve source and root-map piece semantics, and test malformed input release, empty closing chunks, and EOF truncation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): supply the default binary fill in the unimplemented-node fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve an instant selection for the points it decides The gather reserved the whole planned grid, while select_instants clips the range to the series' first and last sample and its lookback: a week-long query at a one-minute step reserved ~504 KB for a series that produces five points, and a small pool refused it before gathering. Reserve for the clipped candidate count, derived with the same alignment the kernel uses. Also pin the start-timestamp semantics the histogram change spread to: last_over_time and first_over_time hand back the sample they selected with its start timestamp, and limitk and limit_ratio keep it on the sample they keep, so a reader downstream can still place a counter reset. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): move the histogram_quantile doc back to its own test The start-timestamp test was inserted into the middle of another test's doc comment, leaving the earlier test without one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): refuse an interleaved input under a selection when the tree is bound The merged plan validates an input ordering in the arm that reads it, so a selection over an interleaved input no longer binds at all. Assert that refusal where it happens instead of asserting a runtime rejection the tree cannot reach, and keep the map-over-a-selection case: a map above a selection does not shield the check, because the check looks at the input the selection itself reads. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): say what the start timestamp pins, and cover every path that carries it The field is GreptimeDB sample metadata, so the comments now say that instead of arguing from what Prometheus omits: a FloatHistogram there has no such field, while its rate and reset evaluation takes start timestamps through a separate channel, and this crate's native range kernel does not read the field at all. Comparison filtering and set selection were the two paths that carry the timestamp without a test. Both keep the payload of the operand they hand back, including the right side of an or at a point the left side does not cover, and a bool comparison answers with a float so no histogram reaches the output. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): drop the start timestamp a computed histogram never had A window holding one histogram became the result as it arrived, while two or more went through an add that clears the timestamp, so whether a summed histogram carried the payload said how many samples the window held rather than what the function did. Clear it where the fold returns, leaving the bucket layout to the caller. Also correct two comments that described behaviour this branch does not have: the selection kernel carries no point limit of its own, and the interleaved map sits above the selection rather than between it and the input. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dennis Zhuang
|
799bcbfdd7 |
feat(promql): add histogram_quantiles and share Go float formatting
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
3264f560fa |
feat(promql): project selected sample times for timestamp() and info()
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
db26fce1f7 |
docs(promql): state where merged variance departs from Prometheus' serial rounding
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
e1da852c46 |
fix(promql): add a partial's compensation as a Kahan addend when merging sums
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
67dfdb6c36 |
chore: merge feat/narthex-agg-stat into feat/narthex-agg-state
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
84d0dde4cb |
chore: merge narthex into feat/narthex-agg-stat
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
4a4afeef07 |
feat(promql): execute a bound native tree over sparse evaluated series (#9777)
* feat(promql): execute a bound native tree node by node, over sparse evaluated series Ports the native runtime onto the sparse evaluated-series API: a chunk is addressed by a `TimeAxis` that may hold several runs, so a consumer reads positions through the axis instead of assuming `start + row`. The driver and both node kinds move to that API: - `chunk.range().len()` becomes `chunk.series.time.len()`, the compact row count, in every metric and byte estimate. - `occupied_runs` walks `time.positions()` with one field cursor, so a chunk of several runs records each run where it belongs. A run open across a hole now ends where the hole begins; leaving it open to the last row made two runs look like one and refused a series that occupies the hole. - A window keeps the pieces its evaluator decided in a queue: one push can decide several runs, and reporting only what the push returned stranded the rest until the next push. Queued pieces are charged by `retained_bytes`, and the charge is recomputed when one leaves, so a released piece is not held. - `map_chunk` answers `Option`: an operation that empties every position (`clamp` with `max < min`) yields no chunk at all, unless the chunk closes its series -- a closing chunk holding no row is handed over so a consumer can tell "nothing more here" from "no more input". `RangeEvaluator::retained_bytes` is restored with the three byte helpers it reads: a field's buffers, a series' identity and the pending batch's vectors, all by capacity, and still counted as storage rather than as a peak over a push. Tests read absolute positions from the axis (`decided` no longer adds a row number to a start), and two sparse cases cover what the old reading got wrong: a `Window -> Map -> Window` chain over a chunk of two runs, and a series that occupies the hole between another's runs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop the window queue the sparse evaluator never needed `RangeEvaluator::push` and `push_series` each return `Result<Option<EvaluatedChunk>>`: the evaluator gathers every run it decided into one chunk and hands back at most one. So `pending.extend(decided)` inserted at most one chunk and `take_pending` removed it on the spot, leaving the queue empty after every successful push. That makes the queue dead weight: a `VecDeque` allocation per productive window, an accounting term that is always zero, a second drain site in `pull`, and doc comments claiming a problem the API does not have. The driver's "ask the input again when a window decides nothing" loop was already the whole story. Back to returning the evaluator's own answer, and drop `pending`, `take_pending` and `take_window_output`. Covers the empty-map protocol this port introduces while it is here: an operation that empties every position hands on nothing while its series is open, and one empty chunk when it closes. A non-closing empty chunk would be rejected downstream and a dropped closing one would leave the series unclosed. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the empty-map output count at one The test checked that every output was empty and that exactly one closed, which a stray extra non-closing empty output would also satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge the memory the native evaluator actually retains Four omissions made the reservation blind to most of what a window over native histograms keeps, and the existing incremental-equals-recomputed test could not see any of them because both sides used the same formula. - `FloatHistogram` owns four heap allocations -- both bucket lists, the custom bounds, and the span layout -- that `FieldSamples::bytes` never looked at. A retained 4096-bucket sample is tens of kilobytes against a ledger that saw a struct header. `heap_bytes` counts them, and both the retained sample slice and the histograms a row boxes are charged for what they own. - Queued chunks were estimated as one float per field per row, which is not what a histogram field holds: it holds indices into a shared `Arc<HistogramBatch>` and the buffers behind the Arrow arrays. The figure now walks the samples, and counts a shared table or a shared buffer slice once per pass rather than once per chunk that references it. - `history_bytes` included `size_of::<History>()` while the map slot term charged a `History` per entry as well, so every entry paid two headers. The per-entry figure now covers only the heap the entry owns. - The primary key is charged for the allocation its `Buffer` keeps alive rather than the bytes it shows: a scan slices the key out of a decoded dictionary-values buffer, so a short key can pin a batch-sized allocation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep the source queue's charge incremental The queue's reservation was recounted by walking every queued chunk on each enqueue and each pop. `push_raw` and `push_evaluated` are public and put no bound on how deep the queue gets, so a caller that queues a whole series before pulling it paid `N(N+1)/2` chunk visits to hold `N` chunks -- the same shape of defect this branch fixes in `absorb`. `QueuedCharge` keeps a reference count and the bytes behind each shared allocation: a table or buffer is charged when the first queued chunk reaches it and given back when the last one leaves, so an operation walks the one chunk it names. The count is per chunk rather than per field, because a pop re-walks the same chunk with the same dedupe and must remove exactly the references that were added. A chunk's own struct bytes are counted per chunk (`own_bytes`); only the allocations behind its fields are counted by address, since a struct has no address to dedupe on. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): append a forward-moving chunk to the emitted history `absorb` rebuilt the whole stored run array on every non-empty chunk. Legal input made that quadratic across chunks with no label collision: one series in `K` chunks, chunk `i` occupying only grid position `2*i`, stores one more run per chunk and moves `K(K+1)/2` of them -- about five billion for a hundred thousand rows. A chunk that starts at or past the end of the last stored run only ever adds to the end, so its runs are appended and the boundary run is joined where it touches. One that can interleave or reach backwards still takes the full one-pass merge, which can split a run. Both leave the runs sorted, disjoint and joined, which is what `Emitted::occupy` looks a position up by. The test records the chunks through `Emitted::occupy` itself and reads the history back from it, covering the forward case, the touching boundary, a hole that has to join both neighbours, and a position before everything stored. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): report a truncated input from a tree with no window `close` only asked windows whether a series was still open, so a tree of an input alone, or of a map over one, accepted a final chunk without `is_last` and handed the consumer a series that never closed. The input node now follows the series of the chunk it last handed over, and closing such a tree reports the truncation against it, which is the same refusal a window makes for the input it reads. `occupied_runs` read a row the axis names with `expect`, which panics on a `Series` whose fields are shorter than its axis; the fields are public, so that chunk is constructible by a caller. It now returns the misalignment as an error naming the node, the field and both lengths. The chunk protocol says one series closes once, which reads as one close per label set. State that it is per input series: a node that rewrites labels can hand two series over under one label set and both of them close. Also say in `Emitted`'s doc that the table is one node's own, so a label set costs its bytes again in every node that keeps one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved input, and close a label set only for its owner Two series-level identity holes, both reported against `42ea525`. The truncation flag followed the last chunk handed over, which is the same as "is a series open" only while at most one series can be open. A tree with no window still accepted `Interleaved`, and `a` open, `b` closed, end of input then passed while `a` never closed. Tracking the open set would need an identity that a raw chunk in bound-label mode does not have (`metric` is `None`, the key holds it), so the ordering is refused when the state is built instead, for every tree shape and not only under a window. The flag's doc now says what it means and why that is the whole answer. `History` closed on any series' `is_last`, so with `abs` dropping the metric name a series that contributed nothing could close a label set another series was still filling: `a{job="x"}` value, `b{job="x"}` empty close, `a{job="x"}` continuation was refused with "already closed the label set". Contiguous input hands over exactly that sequence, so it was not an interleaving artefact. A history now remembers the series that filled it and closes only for that series' closing chunk; a series that produced nothing under an open set is passed over, as it already was under a closed one. The owner is charged as part of the entry. The ordering check also reads every input node rather than only the first: a tree with two inputs would otherwise have its second declaration read and dropped. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): simplify native execution memory estimates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge rewritten logical series before window evaluation Accept disjoint closed-source fragments under one output label set. Retain occupancy checks and merge affected window inputs once at EOF using shared typed sample segments. Keep direct input windows and root maps streaming, and release collected fragments on finish or error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): clarify approximate merge workspace estimate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate series boundaries before collected input merges Keep each collecting window's open metric and last position so EOF grouping cannot hide reversed runs or identity changes before closing. Preserve source and root-map piece semantics, and test malformed input release, empty closing chunks, and EOF truncation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): supply the default binary fill in the unimplemented-node fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge label groups in first-member order and drop unused accessors A HashMap iteration order made the merged series order, and the first collision reported, differ between runs; the root chunk stream exposes that order. Also drop len/is_empty, which had no callers and is_empty was always false. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
7309800aef |
feat(datatypes): describe label decoding in the raw series frame schema (#9786)
* feat(datatypes): describe label decoding in the raw series frame schema A raw series frame written with key-encoded labels carries the primary key as opaque bytes; only the binding held by the scanner says how to turn those bytes back into a metric name and labels. A consumer that receives the frame without that binding cannot decode it, so the schema now describes its own label layout. The description is compact JSON under `greptime.series.label_decode`, carrying a version, the metric name, the primary-key encoding, the ordered key columns with their physical types, and the logical tags to expose -- exactly what decoding reads. It is derived from the binding, so there is one writer, and a reader that does not implement the version it finds rejects the frame instead of guessing. `unpack_raw_series` keeps its binding argument and now requires it to agree with the description the schema declares, so the two cannot drift. `PrimaryKeyLabelDecoder` reduces both of its inputs to one internal key layout, so the metadata-backed and description-backed paths share their validation instead of growing a second decoder. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(datatypes): compare label descriptions by what their order means The frame's declared label-decode description was compared to the binding's with plain equality, but the description has one part whose order carries no meaning: the decoder sorts logical tags by name when it binds them, so the same tags listed in another order denote the same labels and were being refused. `same_label_layout` canonicalises exactly that part and leaves `primary_key` in place, because for a dense key the array position is the encoding position: the same columns in another order are a different key, not the same key reordered. Both fields now say which order they mean. Three checks go with it. The binding stays for a stated reason -- it establishes identity, the physical region key layout a decoded identity came from -- rather than because a count of call sites kept it. The post-pack schema comparison in the series scan could not fire: the leaf already re-derives that schema and compares it, and every `SeriesSchema` field is covered by the checks between them. And two test groups re-asserted what the tracked v1_metadata fixture pins case by case. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): decode a raw frame from the frame alone `unpack_raw_series` took the label mode, the ordering and the caller's binding along with the stream, so a consumer that received a frame from a peer still needed a local copy of the binding to read it — the thing the frame's own description exists to avoid, and the reason the description was freezing a wire format for nobody. The stream is now the only input: the field schema, the label mode, the label decode description and the producer ordering all come from the frame's schema, and every frame is validated whether or not a caller asked for it. `expected` stays as an optional check on the schema that is declared — field names and their order, and no evaluation grid — and supplies nothing to the decoder, so a caller cannot smuggle in a layout the frame does not declare. Three call sites compared the caller's binding against the frame and all three were load-bearing: a peer's response is not trusted to be the region that was planned, and a native plan's child is an arbitrary execution plan. Those became `ensure_frame_matches_label_source`, which states what it compares: the label layout, not the region. Two regions with the same tag ids and names compare equal, and the doc now says so instead of claiming to establish identity. A stream with no scan-bound identity reports `NO_BOUND_LABEL_SOURCE` and answers `label_decode_spec()` instead; every consumer that decides whether to decode reads the description, so the placeholder never stands in for "these labels are complete". The merge contract is the description, not the binding: inputs are compared by the layout they declare, and a binding that could not be expressed on the wire — a `RegionMetadata` snapshot — no longer makes the merge stricter than the frames it merges. `logical_tags` gets one rule rather than two: the writer canonicalizes a copy, because the caller's binding keeps its own order and the table scan compares producer tag lists positionally, and readers compare by equality. The text embeds `ConcreteDataType`'s serde, which `LABEL_DECODE_VERSION` has to cover. Rather than mirror all 25 datatype variants into a type this module owns, a frozen fixture records the text of every variant and an exhaustive match makes a new variant fail to compile until it is recorded. The version policy is written next to the constant: exact-text comparison across nodes, unknown versions rejected, any same-version change breaking — with the planning-time fallback stated as intended and marked not implemented. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(store-api): make every key-encoded raw stream declare its layout A raw series stream whose chunks carry key-encoded labels had to be decoded against a scan-bound `RawLabelSource`, and two consumers compensated for producers that stayed silent by deriving a description from that binding: `pack_raw_series` and the merge's `declared_layout`. `materialize_raw_labels` did not compensate, so an unframed producer stream reached it with no description and a chunk carrying no labels, and failed. The stream itself is the authority for its chunks. A key-encoded producer knows its own region metadata and can state the layout it writes, exactly as a frame states the layout it carries, so the split contract — "a producer declares its identity, a frame declares its layout" — is replaced by one rule: a stream whose chunks carry key-encoded labels declares the description, and a stream whose chunks carry complete labels answers `None`. mito2's `RawStream` caches the derivation at construction, so the declaration exists before the first poll and for an empty stream. The trait documentation states the rule and no longer exempts unframed producers. Both fallbacks are gone: `pack_raw_series` reads the stream's declaration only, and rejects a key-encoded stream that declares none rather than packing it as an explicit-label frame, which would describe chunks carrying no labels as if they carried all of them. The merge reads the declaration only. Test fixtures that model key-encoded producers now declare their layout too; fixtures that carry complete labels keep answering `None`. The new mito2 test decodes a real produced key through the declared description for both dense and sparse encodings, and checks the declaration is present before polling even for an empty stream. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(common-query): let the declaration decide the frame mode alone `pack_raw_series` rejected a key-encoded stream that declared no description outright, so a stream that stays silent could not be packed at all. That check was redundant: the declaration already decides the frame's label mode, and the first chunk whose labels disagree with it is refused by `RawSeriesChunkCursor` with "raw chunk label mode does not match frame mode", so no such stream can pack data either way. What the check added was an early error for a producer that already breaks the contract, in exchange for reading the stream's scan binding here — in the one function whose rule is that the declaration is the only decode input. Drop it, so pack keeps deciding the mode from the declaration and nothing else. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(store-api): assert the fixture's declared label decode description The mock prepared source derives a `LabelDecodeSpec` from its label source and reports it through `RawSeriesStream::label_decode_spec`, but no assertion read it, so the derivation could drift from the chunks the fixture actually builds. Check it against the description the fixture's own key columns imply: an explicit source declares nothing, a primary-key source declares the metric, the dense encoding, the one key column and its tag. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): let the frame's declaration be the only label authority Two things still treated a caller-supplied binding as authoritative after the declaration became the decode input: - A merged stream forwarded its first input's scan-bound identity, though a merge is not a region scan and decodes from the declaration alone. It now reports no bound identity. - `KeyColumn.tag` repeated an invariant every key column is a tag. The check moves to the one place a decoder is bound, so a snapshot built without validation is still rejected instead of silently decoded. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f235434b7f |
feat(promql): compile nested expressions recursively (#9782)
* feat(promql): compile nested expressions instead of fixed query shapes The native compiler matched three shapes: a bare selector, `range_fn(x[r])`, and `range_fn(n, x[r])`. Every composition of those kernels was refused, so `abs(rate(x[5m]))` and `clamp_min(rate(x[5m]), 0)` had no path through it even though the kernels, the `Map` node and the `Sum` node already existed. `compile` is now a thin entry over one recursive rule, `compile_expr`: a parenthesis compiles its inner expression, a vector selector becomes the instant selection it already was, a call dispatches on its signature, and anything else is refused naming what was found. Composition follows from that rule rather than from a list of accepted query shapes -- `abs(rate(x[5m]))` is a map over whatever its argument compiles to, and `clamp_min(abs(rate(x[5m])), 0)` is two maps over one window because the inner call compiles once. Math functions lower onto the node that already computes them: one `Map` carrying one `MathOp`, over the node its vector argument compiles to. The 28 names and their parameters are one explicit table, so supporting another is a decision made in one place; a name outside it is refused as having no native kernel rather than silently treated as math. `round` without an argument binds the kernel's documented default. A scalar parameter is folded from the statement: number literals, parentheses, unary negation, and arithmetic whose two sides both fold. The operator goes through `ArithmeticOp::calculate`, the same function the kernels use, so `%`, `^` and `atan2` keep Prometheus' semantics instead of being redefined in the compiler -- `(-7) % 3` is `-1`, signed as Go's `math.Mod` is. A parameter the statement cannot decide, `time()` being the real case, is refused rather than bound to a value that would be wrong at every point but one. `ArithmeticOp::calculate` widens from `pub(crate)` to `pub` for the fold; its body is unchanged. Nothing else is added: the plan nodes, the kernels and the tree validation are the ones that were there, and validation still happens once per statement, at the root. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): match the arithmetic token by its id, not by its text `arithmetic_op` read the operator through the token's `Display` output. That is a second source of truth for what an operator is: the id is stable and is what the rest of the planner matches on (`prom_token_to_binary_expr_builder`), while the rendered text is a presentation detail. A formatting change would silently stop `%`, `^` and `atan2` from folding -- not wrong, but quietly less than it says it does. Also corrects the doc comment that claimed the parser keeps its token ids to itself. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): say what the validator compares, and make the fold test bite The map's doc claimed `NativeTree::validate` compares the map's output grid with its input's by identity. It compares them by `EvalGrid::spec`, and only an input node's output is compared by identity, so a map that allocated an equal grid of its own would validate the same way. The comment says that now, and the two assertions in the nested-map test compare spec and field names instead of `Arc::ptr_eq`, which failed a map that allocated an equal grid for no behavioural reason. The constant-fold test could not catch what its doc said it guarded: `go_math`'s modulo is `x % y` and its power agrees with `f64::powf` on `2 ^ 10`, so replacing the kernel arithmetic with a hand-written match still passed. The power case is now `1.0000006850274261 ^ 9.80019037e8`, where Go's repeated squaring accumulates error the standard library's `pow` does not, and where the two disagree above the last bit. Replacing the fold with `%`/`powf` fails it. `pi()` and `time()` have no vector to be pointwise over, so the "no arguments" branch explained them wrongly; without it they fall to "has no native kernel", which is accurate, and a listed math name called with no argument at all is covered by the signature's arity check. `math_op` now reads the name first and returns the vector it found, so both rejections come from one place. `argument_position` indexes a fixed array: its three callers pass literals, so there is no fourth position to default to. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved scan at the compiler entry Every accepted shape reads one series to its last chunk before the next, so the ordering has to be refused before the statement's shape is dispatched on; the check sat in the range function branch, which let the bare selector and a math function over it compile under Interleaved. The window branch of NativeTree::validate stays as the backstop for a tree built another way. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |