I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test
reference; review only `feat(promql): evaluate time, date and label
rewriting functions over grid series`.
## What's changed and what's your intention?
### Problem
The native path has no kernels for `time()`, the date functions
(`minute`, `hour`, `day_of_month`, `day_of_week`, `day_of_year`,
`days_in_month`, `month`, `year`) or `label_replace`/`label_join`.
### Change
- `map_time::time` and `map_time::date_of_eval_time` produce one
unlabelled series over the schema's grid from the evaluation timestamps
(`ms / 1000` for `time()`, whole seconds truncated towards zero for the
date functions, as Prometheus `funcTime`/`dateWrapper`).
- `map_time::map_date` reads every float value as Unix seconds and drops
`__name__`, `__type__` and `__unit__`. The value goes through Go's
`int64(f)` as on amd64 (NaN and out-of-range values become `i64::MIN`)
and a port of Go's `time` absolute-date arithmetic, including its
wrapping for instants Go cannot represent, which decides the result for
those inputs. All-float series are mapped chunk by chunk without copying
the time axis; histogram samples are dropped and the series is trimmed
again.
- `map_label::label_replace` anchors the regex as `^(?s:...)$`, expands
`$1`/`${name}`/`$$` with Go's `regexp.Expand` rules, and leaves
unmatched series unchanged. `map_label::label_join` joins source values,
an absent label joining as `""`. An empty result removes the label in
both. Empty source or destination label names and invalid regexes are
`Plan` errors with Prometheus' messages.
- `grid_series::merge_same_labelset` follows Prometheus
`mergeSeriesWithSameLabelset`: series that share a label set (empty
values counting as absent) become one series, merged field by field; a
field with values from two of them at one point fails with "vector
cannot contain metrics with the same labelset". The label functions call
it; `map_date` leaves it to the caller, as other per-series kernels that
drop `__name__` do.
- `promql` now depends on the workspace `regex` crate, which is already
in the build graph.
Review focus:
- `map_time::DateFunction::apply` and `civil`: the port of Go's
`absSeconds`/`absDays` arithmetic; `days_in_month` uses the proleptic
Gregorian leap rule on the computed year.
- `map_time::go_int64`: NaN and values outside `[-2^63, 2^63)` map to
`i64::MIN`.
- `merge_same_labelset`: the fast path returns the input untouched when
no label sets collide; colliding groups are rebuilt row by row and fail
on any shared point.
- `label_replace` expands its template with Go's `regexp.Expand` rules:
a reference with a leading zero (`$01`) or of more than nine digits
names a group, unlike `regex::Captures::expand`.
### Non-goals
- `timestamp()` (EVAL-TS), which needs sample timestamps from instant
selection.
- Merging histogram series that collide after a label function
(`NotImplemented`).
- Planner wiring.
### Constraints
- Every per-series kernel that drops `__name__` (range functions, date
functions, math) can leave two series with one label set. Prometheus
merges or rejects them per step; on the native path that is
`merge_same_labelset`, which the caller must apply after such kernels.
- Regexes compile with the Rust `regex` crate, as the ordinary planner
does. Perl classes (`\d`, `\w`, `\s`, `\b`) are Unicode-aware there and
ASCII-only in Go's RE2, so non-ASCII label values can match differently.
The dialects also differ in what compiles: `\Q...\E` is rejected, `(?x)`
and repeat counts above 1000 are accepted, and `\<`, `\>` are word
boundaries instead of literal characters. The rustdoc of `label_replace`
lists the same differences.
- `go_int64` follows amd64. On arm64 Go saturates (`NaN` becomes 0), so
Prometheus' own results for non-finite inputs differ by architecture.
### Verification
- `cargo nextest run -p promql`: 315 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`, `hawkeye check`: clean.
- `time_functions_match_prometheus`: 200 fixtures produced by the
Prometheus v3.15.0 engine built for amd64
(`map_time/prometheus_cases.txt`): `time()` and argument-less date
functions over grids from 1685 to 2254, and date functions over values
including NaN, ±Inf, ±1e300, ±9.2e18, leap-day edges of
1600/1900/2000/2100/2400 and years 0 and 10000. Values compare bit for
bit after `merge_same_labelset`; 6 cases expect the duplicate-labelset
error.
- `label_functions_match_prometheus`: 200 fixtures
(`map_label/prometheus_cases.txt`) with random destinations including
`__name__`, absent sources, templates with valid and invalid `$`
references, anchoring-sensitive regexes and invalid ones; 25 expect
errors, the rest compare label sets and value bits.
- Mutation check: each of these fails a test: Go's NaN and out-of-range
conversion replaced with a saturating cast or NaN-to-0; floor instead of
truncating division of the evaluation time; whole-second `time()`;
dropping the 400-year rule; leap day counted in January and February;
hour from the Unix seconds instead of Go's absolute seconds; no leap
February in `days_in_month`; leap year taken from the March-based year;
keeping `__name__`; mapping only the first chunk; regex without the
`(?s:...)` group or without anchors; setting an empty label instead of
removing it; skipping absent `label_join` sources; skipping the merge;
skipping the duplicate check; comparing empty label values as present.
- `replacements_expand_like_go`: 16 templates with expectations taken
from Go, including `$1²`, where Go's Unicode classes end the name before
`²`.
- `merge_combines_disjoint_fields_at_one_point`: different fields at one
point merge, one field present twice fails.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test
reference; review only `feat(promql): evaluate topk, bottomk, scalar,
absent and instant sorts over evaluated series`.
## What's changed and what's your intention?
### Problem
The native path has no operators that depend on the whole set of series
at each point: `topk`/`bottomk`, `scalar`, `absent` and the
instant-result sorts.
### Change
`select` module, following Prometheus (`aggregationK`, `funcScalar`,
`funcAbsent`, `createLabelsForAbsentFunction`, `funcSort*`):
- `topk(k, top, grouping, schema, series)`: at every point each
`by`/`without` group keeps the `k` largest (or smallest) values, NaN
ranking last for both. `k` is truncated and capped at the number of
input series; below 1 there is no result. Selected samples keep their
series' labels and values.
- `scalar(schema, series)`: the value of the only series present at each
point, Go's NaN otherwise; the result is a label-less series with a
value at every point. A scalar in this representation is already
`vector(scalar)`.
- `absent(schema, series, labels)`: 1 with `labels` at every point where
no series is present, in every field. `absent_over_time(m[r])` is
`absent` over the `present_over_time` kernel of #9674.
- `absent_labels(matchers)`: the result labels from the selector's
`LabelMatcher`s (`name`, `MatchOp`, `value`, `matches_empty`).
- `sort_instant(order, schema, series)`: `sort`, `sort_desc` (NaN last),
`sort_by_label` and `sort_by_label_desc` (Go `natsort` order, then the
label sets) for single-point results. Multi-point results are returned
unchanged, as Prometheus does not sort matrices.
- `go_sort::sort_by`: a port of Go's pdqsort (`sort.Sort` and
`slices.SortFunc`). The instant sorts use it with Prometheus'
comparators, so ties, ±0, several NaNs and `natsort`'s inconsistent
order come out in Go's order, and an inconsistent comparator cannot
panic.
`topk`, `scalar` and the value sorts take one field and reject others
with `NotImplemented`; histogram samples are rejected the same way.
Review focus:
- `topk`: NaN ranking and the `k` conversion.
- `absent_labels`: the matcher order (see Constraints).
- `natural_less` and `chunks` against `facette/natsort.Compare`.
- `topk`/`bottomk` return nothing for `k` below 1 and reject a NaN `k`
and one of 9223372036854774784 (Prometheus' `maxInt64`) or more with
Prometheus' errors.
- `absent` marks each field absent on its own.
### Non-goals
- Non-constant `k`, `limitk`, `count_values`, planner wiring.
### Constraints
- Ties in `topk`/`bottomk` go to the series whose label set sorts first.
Prometheus resolves them by the order its storage returns series in
(posting order), which is not a property of the series.
- `absent_labels` applies Prometheus' rule (the first equality matcher
of a name sets it, any other matcher of the name removes it) to the
matchers in the order Prometheus actually reads them: TSDB's
`PostingsForMatchers` stably moves narrowing matchers before subtracting
ones, in place, before `absent` builds its labels. Over an empty storage
Prometheus keeps the query order instead. The two orders differ only
when one name has both an equality and another matcher.
### Verification
- `cargo nextest run -p promql`: 315 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`, `hawkeye check`: clean.
- `select_operators_match_prometheus`: 180 cases from the Prometheus
v3.15.0 engine built for amd64 (60 each of `topk`/`bottomk` with
`by`/`without` and several `k`, `scalar`, and `absent` with equality,
inequality and repeated matchers), over series with holes, special
values and NaN, sampled on the grid with a 1ms lookback. Result label
sets and value bits match. The fixture is `select/prometheus_cases.txt`.
- `matches_go_sort_on_comparators_that_are_not_total_orders`: 208 orders
produced by Go's `sort.Sort` and `slices.SortFunc`
(`go_sort_cases.txt`), with consistent, NaN-like, tie-heavy and
inconsistent comparators, up to 257 elements.
- `sort_instant` and `natural_less` checks for the cases the fixture
cannot carry (instant order, Go `natsort` behavior including digit runs
beyond `i64`).
- Mutation check: NaN ranked first, `topk`/`bottomk` swapped, `k`
rounded up, `__name__` kept by `without`, `scalar` with several series,
repeated matchers kept, query-order matchers, and `absent` ignoring
presence each fail a test.
- `topk_validates_k_like_prometheus` and
`absent_evaluates_fields_independently` fail without these changes.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test
reference; review only the two `vector_match` commits.
## What's changed and what's your intention?
### Problem
The native path has no vector–vector operations: arithmetic,
comparisons, `group_left`/`group_right` and `and`/`or`/`unless` between
two sets of evaluated series.
### Change
`vector_match::binary_vectors` and `vector_match::set_vectors` take both
operands as evaluated series on one grid and return trimmed grid series.
Semantics follow Prometheus `VectorBinop`, `resultMetric`,
`signatureFunc`, `VectorAnd`, `VectorOr` and `VectorUnless` at every
grid point:
- Match groups: `on(...)` keeps the listed labels (`__name__` when
listed); `ignoring(...)` and the default keep every other label except
`__name__`. Empty label values count as absent.
- Cardinality is checked per point, never over a series' whole lifetime:
- The "one" side (right, or left with `group_right`) may hold one series
per match group at a point where both operands are non-empty.
- One-to-one matching may match a group once per point, filtered
comparisons included.
- `group_x` results must have unique label sets per point.
Series of one group that are present at disjoint points never conflict.
- Result labels: arithmetic and `bool` comparisons drop `__name__`,
`__type__` and `__unit__`; one-to-one keeps the `on` labels or drops the
`ignoring` ones; `group_x` labels are copied from the "one" side, and
removed when it has no value for them.
- Values: comparisons without `bool` keep the left value where they
hold; `bool` returns 1 or 0. `^` and `atan2` use the standard library.
- Errors: the error of the earliest failing grid point is returned. At
one point a one-side duplicate is reported before a one-to-one or
`group_x` violation, as Prometheus checks the one side before matching.
- Fields are paired by position; the output carries the shorter side's
field count, named after the left side. Each field is matched on its
own, as if the operands held only that field, and the per-field results
are joined by label set. Set operators require equal field counts and
return `NotImplemented` otherwise.
- `or` requires the same field count on both sides and rejects others
with `NotImplemented`. Histogram samples are rejected with
`NotImplemented`.
Series are processed whole over their own grid ranges: the "one" side is
indexed by match group, and per-point checks run only over the points a
series covers.
Review focus:
- `binary_vectors`: the one-side duplicate check is limited to points
where both operands are non-empty, as Prometheus short-circuits empty
steps before checking.
- `binary_vectors`: the one-to-one check counts a match even when the
comparison filters it out.
- `Results::sort_rows`: duplicate rows of one result label set at a
point are the `group_x` uniqueness error; the earliest point across all
label sets is returned.
- `binary_vectors` error selection: earliest point wins; ties go to the
one-side duplicate.
- Results of different match groups that share a label set once
`__type__` is dropped fail as duplicate label sets after the operator,
as Prometheus does; two results within one group fail the `group_x`
uniqueness check.
### Non-goals
- Scalar operands (MAP-ARITH), `fill` modifiers, histogram operands
(MATCH-HIST), planner wiring.
### Constraints
Results agree with Prometheus within promql-compat's relative tolerance
(1e-10), not bit for bit.
The ordinary planner deviates from Prometheus in ways this kernel does
not reproduce:
- An operand whose plan has no tag columns is joined on time only and
broadcast to every series of the other side (`only_join_time_index`).
Prometheus matches it by label signature, so `sum(m) / m` returns
nothing in Prometheus.
- Labels with an empty value and absent labels join as different values
(`NullEquality::NullEqualsNull` over nullable tag columns). Prometheus
treats them as equal.
### Verification
- `cargo nextest run -p promql`: 317 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`, `hawkeye check`: clean.
- `binary_vectors_match_prometheus_reference`: 3,000 random cases
against a per-point transcription of Prometheus
`VectorBinop`/`resultMetric`/`signatureFunc` that keeps labels in a map
with `__name__` as an ordinary label. The cases cover all 13 operators,
`bool`, random `on`/`ignoring` subsets including `__name__`, all three
cardinalities with random `group_x` labels, 1–2 fields per side, and
series with holes and disjoint lifetimes. Over 1,000 cases succeed and
over 100 hit a cardinality error; results compare label sets, absolute
grid ranges and value bits, and errors compare their class.
- `set_vectors_match_prometheus_reference`: 3,000 random cases for
`and`/`or`/`unless`.
- `vector_operations_match_prometheus`: 180 fixtures (120 binary, 60
set) produced by the Prometheus v3.15.0 engine
(`vector_match/prometheus_cases.txt`); results compare label sets and
grid points exactly and values within the tolerance, errors compare
their class.
- `pow_and_atan2_match_go`: 452 Go `math.Pow`/`math.Atan2` results
(`vector_match/go_math_cases.txt`) agree with the standard library
within the tolerance.
- `reports_the_earliest_failing_point`: a `group_left` violation at the
first point wins over a one-side duplicate at the second.
- Mutation check: dropping `__name__` for comparisons, ignoring the
`ignoring` list, checking one-side duplicates at points where an operand
is empty, removing the one-to-one match check, keeping a `group_x` label
the one side lacks, not filtering comparisons, keeping matched rows in
`or`, swapping `and`/`unless`, keeping all labels under `on`, accepting
duplicate result rows, and taking the wrong "many" side for
`group_right` each fail a test. So do replacing the Go ports with
`powf`/`atan2`, misplacing the fractional-exponent split in `pow`,
returning the first one-side or one-to-one violation found instead of
the earliest, always or never preferring the one-side duplicate on ties,
returning the first duplicate label set instead of the earliest, and
skipping the row sort.
- `colliding_results_fail_by_where_they_collide`: a collision across
match groups reports duplicate label sets, one within a group the
grouping error; the random reference now includes `__type__`.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9677 for `RangeEvaluator` with `@` and
subquery input; review only `feat(promql): evaluate experimental first,
mad and ts_of range functions`.
## What's changed and what's your intention?
### Problem
The native range evaluator has none of Prometheus' experimental range
functions `first_over_time`, `mad_over_time` and
`ts_of_first/last/min/max_over_time`.
### Change
New `RangeFunction` variants with kernels ported from Prometheus:
- `FirstOverTime`: the first sample of the window (`funcFirstOverTime`).
- `MadOverTime`: the median of the absolute deviations from the median,
through the existing `quantile`; NaN when any sample is NaN
(`funcMadOverTime`).
- `TsOfFirstOverTime`, `TsOfLastOverTime`: the timestamp in seconds of
the first and last sample.
- `TsOfMinOverTime`, `TsOfMaxOverTime`: the timestamp of the last sample
holding the minimum or maximum; a NaN so far is replaced by any later
sample (`compareOverTime` with `<=`/`>=`).
Window timestamps are shifted by the offset (or the `@` fold offset)
onto the evaluation timeline. The `ts_of_*` kernels subtract it to
report the sample's own timestamp, as Prometheus does. All six are
step-invariant, so `@` replays one result.
Review focus:
- `ts_of_*` in `RangeFunction::kernel`: the subtracted value is the fold
offset passed to `kernel`, which equals the selector offset without `@`.
- `timestamp_of_extremum`: `>=`/`<=` keep the last extremum, and a NaN
best is always replaced.
- Results drop the metadata labels except for `first_over_time`, as
Prometheus does.
- `ts_of_last_over_time` reports the max of the last timestamp and 0, as
Prometheus 3.15 does by taking the max with a histogram timestamp that
defaults to 0.
### Non-goals
- Gating these functions behind an experimental flag, which belongs with
planner wiring.
- Histogram samples (`first_over_time` and `ts_of_first/last_over_time`
also accept them in Prometheus).
### Constraints
None.
### Verification
- `cargo nextest run -p promql`: 318 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`, `hawkeye check`: clean.
- `range_functions_match_prometheus` now also reads
`range_eval/experimental_cases.txt`. It holds 240 cases (40 per
function) from the Prometheus v3.15.0 engine built for amd64, 125 of
them with `@` and most with an offset. Values compare bit for bit, NaN
as any NaN.
- `ts_of_extremum_moves_past_a_leading_nan` covers a window starting
with NaN.
- Mutation check, each of these fails a test:
- `ts_of_first/max` without subtracting the offset;
- strict `>`/`<` in `ts_of_max/min`, keeping a NaN best;
- no NaN check in `mad`, returning the median instead of the deviation
median;
- `first` returning the last sample, `ts_of_last` returning the first;
- truncating the timestamp to whole seconds.
- `ts_of_last_over_time_reports_zero_for_negative_timestamps` and the
`first_over_time` case of `range_functions_drop_metadata_labels`.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9674 (which is stacked on #9673); review
the three commits after `7a3cfca086`.
## What's changed and what's your intention?
### Problem
`RangeEvaluator` reads one window per grid point from raw series. PromQL
also reads range windows fixed by `@`, and windows over the evaluated
result of a subquery.
### Change
`@` (`range_eval::RangeWindow::anchor_ms`):
- `anchor_ms` is the resolved `@` time; `start()` and `end()` are
resolved by the caller.
- As in Prometheus (`setOffsetForAtModifier`) and the ordinary planner,
samples are shifted by `start - (anchor - offset)` and only the first
grid point's window is read.
- Every function except `predict_linear` evaluates that window once and
reports the result at every grid point. `predict_linear` evaluates the
same window at every grid point, centered on that point's time.
- An anchor that leaves `i64` with the offset is a `Plan` error.
Subquery:
- `range_eval::subquery_grid(outer, window, step_ms)` returns the inner
evaluation grid: the multiples of `step_ms` after `start - F - range` up
to the last window end, where `F` is the fold offset of `window` (with
`@`, the only window end is the anchored one). It returns `None` when no
multiple falls inside.
- `RangeEvaluator` accepts an input schema with an evaluation grid; such
an evaluator takes complete evaluated series through `push_series` and
treats each grid value as a sample at that time. Stale bit patterns are
kept as values, as the ordinary subquery plan does (`SeriesNormalize`
with `filter_stale_markers = false`).
- `push` and `push_series` reject the other kind of input.
Review focus:
- `RangeEvaluator::evaluate`: with `@`, only grid index 0 is read;
`Replay::Once` copies its result, `Replay::EveryPoint` re-evaluates the
same window at each point.
- `subquery_grid`: the start bound is exclusive and the end bound
follows the anchored or last outer point.
- `RangeEvaluator::ingest`: samples are no longer retained once every
window that reads them has been evaluated.
### Non-goals
- Resolving `start()`/`end()`, choosing the subquery step default, and
planning the inner expression; these stay with the planner.
- `@` on instant selectors (built on the instant selection of #9649) and
`timestamp()`.
- Wiring into execution.
### Constraints
- Follows Prometheus and the `main` planner for `@` on subqueries
(#9446). The `narthex` branch planner predates #9446 and ignores `@` on
subqueries.
- No change to existing public behavior; `RangeWindow` gains a field.
### Verification
- `cargo nextest run -p promql`: 317 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`: clean.
- The randomized reference tests draw `@` for a quarter of the seeds.
The probe reference checks window membership and the evaluation time a
kernel sees, both for step-invariant functions (first point's window
evaluated once) and for `predict_linear`'s per-point evaluation; the UDF
comparison covers the functions whose UDF agrees with Prometheus.
- `subquery_series_match_references`: 300 seeds of random evaluated
series, with stale bit patterns, on the grid from `subquery_grid`; each
is compared with the probe reference and with the UDF-compatible
functions.
- `subquery_grid_covers_every_step_multiple_in_the_windows`: 2,000 seeds
against brute-force enumeration of step multiples.
- Mutation check: a wrong fold offset sign, ignoring the anchor, moving
the `predict_linear` window with the step, reporting only the first
point, treating `predict_linear` as step-invariant, dropping stale
values from subquery input, including the open bound in the inner grid,
ending an anchored inner grid at the last outer point, and skipping the
input grid check each fail a test.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9673; review the commits after #9673's
head. This PR also carries the `*_over_time` functions from the closed
#9675.
## What's changed and what's your intention?
### Problem
`RangeEvaluator` from #9673 needs window kernels for the float range
functions PromQL defines.
### Change
`range_eval::RangeFunction` lists the supported functions;
`RangeEvaluator::try_new` takes one, and `FloatWindow`, `WindowKernel`
and `with_kernel` are `pub(crate)`:
- Rate family: `Rate`, `Increase`, `Delta`, `Irate`, `Idelta`.
- Over time: `AvgOverTime`, `MinOverTime`, `MaxOverTime`, `SumOverTime`,
`CountOverTime`, `LastOverTime`, `PresentOverTime`, `StddevOverTime`,
`StdvarOverTime`, `QuantileOverTime(f64)`.
- Change: `Resets`, `Changes`, `Deriv`, `PredictLinear(f64)`,
`DoubleExponentialSmoothing { smoothing, trend }`.
Parameters are constants. Each kernel follows the Prometheus function of
the same name (`promql/functions.go`, v3.15.0), including the order of
floating point operations. Arithmetic where GreptimeDB's range UDFs
already agree with Prometheus is shared with them: `add_counter_resets`,
`instant_delta`, `extremum` (moved out of `scan_extremum`) and
`count_edges` (now taking a slice). Those UDFs keep their results.
`double_exponential_smoothing` factors outside `(0, 1)` fail with
`Execution` at the first window that holds a sample, where Prometheus
checks them; input with no sample in any window evaluates to nothing.
`predict_linear` regresses around `eval_ts + offset`, Prometheus'
evaluation time on the shifted timeline, and `try_new` rejects a grid
where that overflows `i64`.
Review focus:
- `range_eval/function.rs`: each kernel against its Prometheus
counterpart, in particular `kahan_inc` (`kahansum.Inc`, which resets the
compensation on infinity), `avg`, `variance`, `quantile`,
`extrapolated_rate` and `linear_regression`.
- `functions/aggr_over_time.rs`, `edge_count.rs`: the moved code keeps
the same expression sequence.
- Results drop `__name__`, `__type__` and `__unit__` (Prometheus
`DropReserved(schema.IsMetadataLabel)`), except for `last_over_time`;
the metric was passed through unchanged before.
### Non-goals
- Histogram and mixed samples, `absent_over_time`, non-constant function
parameters, planner or executor wiring.
- A reset index for heavily overlapping `rate` windows.
### Constraints
Where the ordinary range UDFs differ from Prometheus, the kernels follow
Prometheus:
- `rate`/`increase`/`delta` extrapolate in seconds as Prometheus does;
the UDF computes the ratio in milliseconds. The two differ in the last
bits.
- `sum_over_time` and `avg_over_time` use Kahan summation (and the
incremental mean on overflow for `avg`); the UDFs use Arrow's lane-wise
sum.
- `stdvar_over_time` uses the compensated variance `stddev_over_time`
uses; the UDF uses an uncompensated one.
- `quantile_over_time` sorts NaN first; the UDF sorts by `total_cmp`,
which puts positive NaN last.
- `deriv`/`predict_linear` return NaN for a constant infinite window,
which the UDFs drop. `predict_linear` takes a fractional horizon, which
the planner truncates to `Int64`.
- `double_exponential_smoothing` computes two-sample windows, returns
nothing for one sample or an empty window, and rejects factors outside
`(0, 1)`. The UDF returns NaN in the first three cases and maps
out-of-range factors to ±Inf.
- Every kernel reads only each field's valid samples, so the UDFs'
dependence on where null slots sit (`deriv`'s intercept, smoothing over
null payloads, lane-wise sums) does not carry over.
### Verification
- `cargo nextest run -p promql`: 314 passed, including the existing
tests of every UDF touched.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`, `hawkeye check`: clean.
- `range_functions_match_prometheus`: 400 cases (20 per function, 1,183
result points) from the Prometheus v3.15.0 engine built for amd64, over
random single series with counters, gauges with special values,
constants, large magnitudes, wide spreads, NaN-heavy windows and stale
markers, random ranges and offsets. Values compare by bits, NaN by being
NaN. The fixture is `range_eval/prometheus_cases.txt`.
- `functions_match_range_udfs`: the nine functions whose UDF agrees with
Prometheus, 300 random seeds each against the UDFs on brute-force
windows, by bits.
- `cargo sqlness bare` over the 34 case names that use these functions:
78 cases passed across standalone and distributed (run before the UDF
code was narrowed back to the four shared helpers).
- Mutation check: Kahan without the infinity reset, `avg` as a plain
mean, uncompensated variance, NaN sorted last, millisecond
extrapolation, dropping NaN slopes, truncating the `predict_linear`
horizon and requiring three samples for smoothing each fail
`range_functions_match_prometheus`; faults in the rate, `irate`,
extremum and edge counting kernels fail the UDF comparison.
- `range_functions_drop_metadata_labels`: `rate` drops the metadata
labels and `last_over_time` keeps them.
- `quantile_keeps_the_window_order_of_signed_zeros`:
`quantile_over_time` sorts as Go does, +0 and -0 equal and in window
order, so `1 / quantile_over_time(1, m[3s])` over +0, -0, -0 is -Inf.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Builds on the series types (#9598, #9599) and the raw
chunk contract (#9603) already on that branch. No dependency on another
open PR.
## What's changed and what's your intention?
### Problem
The native PromQL path has raw series (`RawSeriesChunk`) and evaluated
grid series (`Series` with `TimeAxis::Grid`), but nothing that turns one
into the other for range-vector functions. Operators producing grid
series also need a common way to build output that covers only the
points that have results.
### Change
`grid_series::GridSeriesBuilder` builds one float series on a shared
`EvalGrid` from rows pushed at absolute grid indices:
- indices must be strictly increasing and inside the grid;
- storage is allocated at the first row where any field has a value;
- missing rows between valid rows become nulls, trailing missing rows
are dropped;
- a series with no valid row yields `None`.
`range_eval::RangeEvaluator` evaluates range windows `(T - offset -
range, T - offset]` over raw series, one series at a time:
- `push(&RawSeriesChunk)` takes the chunks of a series in order and
returns `Option<Series>` on the `is_last` chunk; `finish()` fails if the
input ended inside a series.
- A row with a stale marker in any field is dropped, as
`SeriesNormalize` does for range selectors. Each field is then evaluated
over its own valid samples.
- Windows may span chunks and overlap (range > step). Only samples that
a later window can still read are retained, and grid points before the
next sample are skipped.
- The window function is a `WindowKernel` receiving a `FloatWindow`:
offset-shifted timestamps, values, the evaluation timestamp and the
range, the same inputs `RangeManipulate` hands to the range UDFs.
- Errors: non-positive range or a grid-bearing input schema is `Plan`;
field count or length mismatch, unmaterialized labels, a changed
`primary_key` or metric within a series, non-increasing timestamps
across chunks, input ending before `is_last`, and a kept sample whose
`timestamp + offset` overflows `i64` (as `RangeManipulate` reports) are
`Execution`; histogram samples are `NotImplemented`.
`grid_series::reference` (test only) expands grid series into dense
`[grid index][field]` matrices and asserts trimmed range, value bits and
labels against a brute-force reference.
Review focus:
- `RangeEvaluator::evaluate`: a grid point is evaluated only once no
later chunk can add a sample to its window; `head` only drops samples at
or before `T - range` of the current point.
- `RangeEvaluator::ingest`: stale rows are dropped across all fields
before per-field compaction; overflow is checked on kept rows only.
- `GridSeriesBuilder::push`/`finish`: output range equals the span of
valid rows.
- `range_eval/tests.rs` `reference`: window membership is computed
independently of chunking and cursors.
- A grid point that is not ready yet still drops the samples behind its
window, so a long gap between grid points keeps at most the current
chunk.
### Non-goals
- Range functions (`rate`, `increase`, `delta`, `irate`, `idelta`,
`*_over_time`). They follow as separate PRs and plug in as kernels.
- `@` modifiers, subqueries, histogram and mixed samples.
- Planner, stream, or executor wiring; memory pool accounting of the
retained samples.
### Constraints
- New public items only; no existing API, persisted format, or wire
format changes.
- Output field names equal the input fields; the metric is passed
through unchanged.
- Retained samples per field are bounded by the samples of one window
plus the current chunk, independent of grid length.
### Verification
- `cargo nextest run -p promql`: 311 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`:
clean.
- `cargo fmt --check`, `hawkeye check`: clean.
- Randomized reference test: 500 seeds, each with 1-3 fields, random
grid, range below/equal/above step, positive and negative offsets, gaps
longer than the range, missing values, NaN, stale markers, random chunk
splits, sliced arrays and chunked `Samples`, three series per evaluator.
- Mutation check: 16 deliberate faults (window bounds, offset sign,
stale handling, readiness, gap skip, retention, state reset, identity
and order checks, histogram handling, builder trimming and index order)
each fail at least one test.
- `waiting_for_a_point_drops_samples_behind_its_window`: 900 chunks
between two grid points keep at most one chunk; it fails without the
fix.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
Go reads a group name as Unicode letters, decimal digits and _, so in
$1² the name is 1; char::is_alphanumeric also takes ² and other number
characters.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Prometheus treats +0 and -0 as equal and sorts short windows by
insertion, keeping them in window order; sorting -0 before +0 picked
the other zero and flipped the sign of an infinite reciprocal.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Prometheus 3.15 takes the max of the last float timestamp and the
histogram one, which defaults to 0 without histograms.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
label_replace expands its template with Go's regexp.Expand rules, where
$01 or a reference of more than nine digits names a group. Series that
share a label set merge field by field, colliding only when one field
has two values at one point.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
topk and bottomk reject a NaN k and one out of int64, as Prometheus
does, instead of returning results. absent marks each field absent on
its own, as fields are evaluated independently.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Results only need to agree with Prometheus within the relative tolerance
promql-compat uses, so the ports of Go's math.Pow and math.Atan2 go; the
Go results stay as a fixture compared within that tolerance.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The evaluator stopped at a grid point whose window was not complete yet
before advancing past the samples that window cannot use, so a long gap
between grid points kept every sample pushed in the meantime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Arithmetic and bool comparisons drop __type__ and __unit__ with
__name__, as Prometheus does. __type__ stays part of the match group, so
results of different groups can now share a label set; that fails as
duplicate label sets after the operator, while two results within one
group still fail the group_x uniqueness check.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Prometheus drops __name__, __type__ and __unit__ from the result of every
range function except last_over_time and first_over_time; the evaluator
passed the metric through unchanged.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Add first_over_time, mad_over_time and ts_of_first/last/min/max_over_time
to the range evaluator, following Prometheus funcFirstOverTime,
funcMadOverTime, funcTsOf*OverTime and compareOverTime. Timestamps are
reported on the sample timeline, before the offset or @ fold.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Add time(), the date functions and label_replace/label_join for
evaluated series, following Prometheus funcTime, dateWrapper,
evalLabelReplace and evalLabelJoin, and merge series that end up with
one label set as mergeSeriesWithSameLabelset does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Use ports of Go math.Pow and math.Atan2 for ^ and atan2, report the
error of the earliest failing step, and check results against
fixtures generated by the Prometheus engine.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
dennis zhuang