mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-17 16:24:07 +00:00
c500f4697f8a7bd3473cb40d9b040d7bba6551c0
6586
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
c500f4697f |
perf(promql): append ordinary raw chunks to the evaluator in bulk (#9709)
* perf(promql): append ordinary raw chunks to the evaluator in bulk `ingest` copied every input sample into `FieldSamples` one at a time. A `RawSeriesChunk` hands the evaluator Arrow buffers `Samples::from_float` wrapped without copying, and `ingest` read them back through `Samples::iter` only to perform, per sample, an enum match, a stale-marker comparison, an overflow-checked offset addition and two `Vec::push` calls. A chunk of null-free float columns read with no offset now appends with `extend_from_slice`. Every other shape keeps the per-sample loop: a shifted timestamp reads different timestamps, a null has no sample, a field mixing native histograms reports `DenseSamplesRef::Mixed` and must keep its windows on the histogram path where the mixed-window rules decide the result, and a chunk whose values contain a stale marker drops its row and needs the per-row decision. The reset index stays per-sample inside the bulk append, so the recorded resets are the same ones the per-sample path records. Tests assert both that the values still match the brute-force reference and that the bulk path is the one that ran, since both paths produce identical output by construction; a mutation that lets a mixed field through the admission check fails them. Signed-off-by: discord9 <discord9@users.noreply.github.com> * refactor(promql): keep the ingest docs with ingest and drop untested guards Three review findings on the bulk-ingest path, none changing behaviour. The doc comment describing `ingest` sat on top of `ingest_in_bulk` after the bulk path was inserted above it, so `ingest` had lost its docs and the bulk function carried documentation for work it does not do. The comment moves back to `ingest`. `ingest_in_bulk` returned `Result<bool>` but never returned `Err`: the two length checks inside it are already done by `check_chunk` for raw input and by `push_series` for evaluated input. It now returns `bool`, and the length checks go with the `Result`. The stale-marker and offset "is not appended in bulk" tests guarded faults the behaviour tests already catch -- the differential and reference comparisons fail if either condition is admitted. The histogram one stays: it is the only test covering that admission rule. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: discord9 <discord9@users.noreply.github.com> |
||
|
|
3fca17791c |
perf(promql): index counter resets per field instead of rescanning the window (#9708)
* perf(promql): index counter resets per field instead of rescanning the window The fused evaluator rescanned every window for counter resets, once per grid point. Record the resets of each field as the samples are ingested instead, keyed by an absolute sequence number, and read the ones a window covers with a binary search. Only `rate` and `increase` correct counters, so only they maintain the index. The binary search runs over the field's retained reset index rather than over the window's samples: its input is the remaining slice of resets, so its cost follows the number of retained resets, not the window length. A drop across an intervening native histogram sample is recorded as a reset of two samples that are not neighbours. That is unobservable: a window holding both floats also holds the histogram between them, so it is a mixed window, whose counter functions report nothing; and a window starting after the earlier float excludes a reset on its first sample. Verified with `cargo nextest run -p promql` (453 passed) and `cargo clippy -p promql --all-targets --all-features -D warnings`. Tests assert both that the indexed results match the brute-force scan and that the index is what ran. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop per-field clearing the evaluator never needs Two review findings, both about code that does nothing. `finish_series` cleared every field after `evaluate(None)` returned. That loop is redundant: `evaluate` ends with either every point evaluated -- so `head` moved to the end and `advance` dropped the samples -- or no field holding a sample, and either way the fields are empty. It also did not clear `histograms`, which made it read as though it should. Removing it keeps the suite green, and `push` already documents that an evaluator is not reused after an error. `FieldSamples::clear` existed only for that loop. `overlapping_window_spans_chunks` used a `sum` kernel and never touched resets. Windows spanning chunks are already covered by `evaluator_matches_brute_force_reference`, which chunks randomly, and no mutation was caught only by this test. Its `sum` helper went with it. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@users.noreply.github.com> |
||
|
|
04589c175f |
perf(promql): widen the shared extrapolation arithmetic to 64 bits (#9704)
* perf(promql): widen the shared extrapolation arithmetic to 64 bits `extrapolation_factor` is the one place both the float and the native-histogram range paths compute Prometheus' extrapolation. Its three millisecond differences were widened to `i128` and narrowed back with `as f64`; each of those narrowing conversions compiles to a call to the `__floattidf` builtin rather than an instruction, and the function runs once per grid point per series. Use saturating `i64` subtraction instead, which is exact here: a difference too large for `i64` is either a window start below every representable instant, or a distance far past the extrapolation threshold where both values take the same branch. Both callers now share the same arithmetic; neither path carries its own copy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): wrap the extrapolation arithmetic like Go instead of saturating it `extrapolation_factor` widened to `i128`, subtracted, and narrowed back with `as f64`; that compiles to a call to the `__floattidf` builtin once per grid point per series. Doing the subtraction in `i64` removes the calls, but the first form of it clamped with `saturating_sub`, and clamping is not what Go computes: with `eval_ts = i64::MIN + 10`, `range_ms = 100`, `first_t = i64::MIN + 1` and `last_t = i64::MIN + 9`, Prometheus returns 1.625 and the clamped subtraction returns 1.25. `wrapping_sub` is Go's `int64` arithmetic bit for bit, at the same cost. Real timestamps are nowhere near that range, so the two agree on every input a query can produce; the difference only shows up where the old code silently disagreed with Prometheus. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@users.noreply.github.com> |
||
|
|
aa4abe554a |
feat(promql): add evaluated series arithmetic and basic aggregates (#9682)
* feat(promql): add evaluated series math kernels Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): add evaluated series scalar arithmetic Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): complete evaluated series math functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): add series comparisons and scalar results Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): add sparse evaluated series basic aggregates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): align time-varying scalar maps on evaluated grids Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop metric name in evaluated series math like Prometheus Prometheus' simpleFloatFunc drops __name__ together with __type__ and __unit__. Check abs, ceil, floor, sqrt and round against fixtures generated by the Prometheus v3.15.0 engine. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop empty labels with the metric name in series math Prometheus has no empty-valued labels, so a kept empty value would hide a same-labelset collision from the caller. Round shares its calculation with the UDF, so check it only against the Prometheus fixtures. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): align evaluated series map kernels with Prometheus Follow Prometheus v3.15.0 simpleFloatFunc, clamp, VectorscalarBinop and vectorElemBinop: drop __name__, __type__ and __unit__ for math, arithmetic and bool comparisons; use ports of Go math (including the amd64 Log and Exp assembly) for ^, %, atan2 and the transcendental functions; empty a clamp with max < min; keep -0 in sgn and Prometheus' operation order in deg and rad. Fields are compared independently and filtered output is trimmed with GridSeriesBuilder. Check the kernels against fixtures generated by the Prometheus engine and Go math results built for amd64. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): align evaluated series aggregation with Prometheus Follow Prometheus aggregation and generateGroupingLabels: no modifier is by (), without drops __name__, empty label values are absent, sum and avg use Neumaier compensation and avg switches to an incremental mean on overflow. Fields are aggregated independently, so a field without values at a row is null also for count and group. Output is trimmed with GridSeriesBuilder. Check against fixtures generated by the Prometheus engine and drop the main UDAF oracle and the common-function dependency. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): validate whole segments in series comparisons map_compare checked sample types one by one, so a histogram or mixed segment without a present sample passed. Validate segments up front as the other kernels do, check them before an empty clamp, share the empty-label rule with aggregation, and pin exact bits in the deg, rad and incremental avg tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: exclude promql golden fixtures from typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(promql): point series round at Prometheus funcRound Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): keep Go math ports only where std leaves the tolerance Series math now follows the relative-1e-10 comparison rule: functions use Rust's standard library unless Go's amd64 result differs beyond the tolerance or in overflow and sign. asin, atan, tanh, asinh, acosh and atanh use std; the Exp assembly port is reduced to its overflow edge, and sinh/cosh build on it above 21. Golden and Go math fixtures are compared with almost.Equal. The Go cases are evaluated through the kernels' own per-sample functions, and each retained port must have a case where std fails, so a port that is no longer needed, or a swap back to std, fails a test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(promql): validate aggregate input like series maps aggregate_float repeated map's schema, grid-axis and float-segment checks in a different order, so one bad series could fail with a different error category in each. Both now use validate_float_series, and map_math/map_negate/map_arithmetic run it before mapping instead of their own axis check. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin avg compensation to Prometheus bits With fixtures compared within the tolerance, dropping or misplacing the Kahan compensation in avg only moves the last bits. Three Prometheus cases where it does are compared exactly: compensation on the incremental mean, the separate division of sum and compensation, and rescaling the compensation when avg switches to the incremental mean. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin the zero sign of clamp and the Neumaier swap Under the fixture tolerance, +0 and -0 compare equal and no case had an addend larger than the running sum, so two branches lost their tests: Go's Max picking +0 over -0, whose sign a later 1/x turns into the sign of an infinity, and the Neumaier swap, without which sum(1, 1e100, 1, -1e100) is 1 instead of 2. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: allow the Unicode category Nd in typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): keep aggregate state in fixed-size grid blocks Aggregation kept one B-tree entry and one Vec allocation per populated evaluation point, plus a tree lookup per sample. State is now stored in 64-point blocks keyed by absolute grid index / 64, holding one state per field and point. A block is allocated only once a sample falls in it, so sparse input never expands to the full grid, and each field's run through a block costs one lookup. Points without samples are skipped on output, which also keeps the last block from reaching past the grid. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): drop reserved and empty labels in one rebuild drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin that aggregation keeps type and unit labels Prometheus generateGroupingLabels drops only the metric name, unlike math and arithmetic. Nothing failed when grouping dropped __type__ or __unit__. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): return aggregate groups in first-seen order Groups were collected in a BTreeMap and returned sorted by label. Prometheus' rangeEvalAgg appends groups in the order their first input appears, and a nested aggregation folds them in that order: for sum(sum by (g) (m)) over 1e308, -1e308, 1e308 with g = a, c, b, Prometheus gives 1e308 and label order overflows to +Inf. Look groups up by label key and keep them in a Vec. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin which zero min and max keep Prometheus replaces the min/max accumulator only on a strict comparison, so of +0 and -0 the first is kept. The golden tolerance treats the zeros as equal, and a non-strict comparison passed every test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
50002c6830 |
feat(promql): add evaluated series math kernels (#9672)
* feat(promql): add evaluated series math kernels Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop metric name in evaluated series math like Prometheus Prometheus' simpleFloatFunc drops __name__ together with __type__ and __unit__. Check abs, ceil, floor, sqrt and round against fixtures generated by the Prometheus v3.15.0 engine. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop empty labels with the metric name in series math Prometheus has no empty-valued labels, so a kept empty value would hide a same-labelset collision from the caller. Round shares its calculation with the UDF, so check it only against the Prometheus fixtures. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: exclude promql golden fixtures from typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(promql): point series round at Prometheus funcRound Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: allow the Unicode category Nd in typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): drop reserved and empty labels in one rebuild drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
a1052c0c0f |
feat(query): add raw-series preflight and execution surface to query traits (#9661)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
cbc2a729a5 |
feat(common-meta): add raw-series preflight and execution surface to Datanode trait (#9659)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
5f0f344908 |
feat(table): scan-bound typed raw series production via SeriesScanExec (#9664)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9657** (raw request types). Also requires **#9651** (scanner preflight), **#9655** (frame ordering), **#9656** (recordbatch adapter). ## What's changed and what's your intention? Add scan-bound typed raw series production to `TableScanExec`: - When the scan binding has a `RawSeriesRequest` and preflight passes, produces `SeriesScanExec` that streams typed `RawSeries` frames. - Shares MVCC/projection/filter semantics with the existing RecordBatch path. - Strictly opt-in: absent or declined preflight returns the ordinary plan unchanged. **2 files, +475/-2 production + 1506 test lines.** Note: diff includes cherry-picked commits from #9651 and #9656. Once they merge, the diff will shrink. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
|
18e11944c9 |
feat(promql): add native instant manipulate execution over raw series (#9660)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9654** (source materializer). Also requires **#9649** (instant eval) and **#9655** (frame ordering). ## What's changed and what's your intention? Add native instant manipulate execution over raw series streams: - `NativeInstantMapping`: preflighted mapping from typed raw series to query output schema. - `NativeInstantManipulateExec`: ExecutionPlan that consumes `SendableRawSeriesStream`, materializes labels, selects instants, and emits aligned RecordBatches. - `InstantManipulate::try_to_native_execution_plan()`: validates the raw frame schema and constructs the native plan. **5 files, +1830/-0 lines.** No callers wired yet. Note: diff currently includes cherry-picked commits from #9649 and #9655. Once those merge into narthex and the base is updated, the diff will shrink to only the 5 native exec files. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p promql` → **357/357 passed** ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
|
b3707c28ac |
feat(promql): merge ordered raw series inputs (#9653)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Depends on #9650** (labels decoder). Includes `materialize_raw_labels` — if the source-only PR merges first, this rebases cleanly. ## What's changed and what's your intention? Add `promql::source::merge_ordered_raw_series`: merges multiple ordered raw-series streams into one globally ordered stream by series key and time. All inputs must share the same schema and label binding, be `KeyThenTime` ordered, and have undecoded labels. The merge recomputes per-series `is_last` globally. Errors are terminal; already-emitted prefixes are not rolled back. **8 files, +4118/-0 lines (622 production + 3489 tests + 7 config).** ### Verification - Compiled and tested with #9650 merged: `cargo check --all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D warnings` ✅ - Cannot compile standalone until #9650 merges. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
200ed3ce36 |
feat(table): scan-bound typed raw series production via SeriesScanExec
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
f354e3423a |
feat(promql): add native instant manipulate execution over raw series
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
535923629b |
feat(promql): merge ordered raw series inputs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
83a0d58980 |
feat(query): capture native raw-series routes and extend MergeScan for typed frames (#9663)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9662** (native config). Also requires **#9655** (frame ordering metadata). ## What's changed and what's your intention? Add native raw-series route capture and MergeScan typed-frame support: - `CaptureNativeRawRoute`: captures region routing during planning when `experimental_native_promql` is enabled. - `NativeRawSource`: borrows the captured route for native raw-series execution. - Schema identity validation for `greptime.series.` metadata. - Wires `experimental_native_promql` from `QueryEngineState` → `DfQueryPlanner` → `DistExtensionPlanner`. **4 files, +1746/-113 lines.** Note: diff includes cherry-picked commits from #9655. Once it merges, the diff will shrink. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
|
440ecb909b |
feat(common-query): add raw series request binding and preflight types (#9657)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add raw series request types for the named raw-series preflight contract: - `RawSeriesRequest` / `RawSeriesTag`: versioned binding for a single-region raw-series request. - `PreparedNamedRawSeries` + encode/decode: JSON body for the `PREPARE_RAW_SERIES_ACTION` Flight action. Strict serde (`deny_unknown_fields`), version 1 only. Includes wire compatibility fixtures. **7 files, +661 production + wire fixtures.** ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
e0af59f14c |
feat(promql): materialize complete raw source labels (#9654)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Depends on #9650** (labels decoder). Must merge after (or be rebased onto) #9650. ## What's changed and what's your intention? Add `promql::source::materialize_raw_labels`: a stream decorator that converts raw series chunks with primary-key-encoded labels into chunks with complete explicit labels. Uses `PrimaryKeyLabelDecoder` (from #9650) to decode dense/sparse primary keys against the scan-bound region metadata snapshot. Rejects schemas with eval grids or empty fields, and requires complete (not projected) label decoding. Output always advertises `RawLabelSource::Explicit`. **5 files, +1216/-0 lines (169 production + 1040 tests + 7 config).** The merge module is intentionally excluded. ### Verification - Compiled and tested with #9650 cherry-picked: `cargo check --all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D warnings` ✅ - Cannot compile standalone until #9650 merges. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
bd9f7dc40b |
docs(promql): say what draining the histograms after an early stop is for
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
328218776d |
fix(promql): follow up the post-merge review of native histogram operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
53ef80a516 |
feat(query): capture native raw-series routes and extend MergeScan for typed frames
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
3ef71b5702 |
feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
7c6595d9f7 |
feat(promql): materialize complete raw source labels
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
f8054ab297 |
feat(promql): select instant samples from shared typed fields (#9649)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Builds on the shared series types already present on `narthex` (#9598, #9599). This PR targets `narthex`, not `main`, and has no dependency on another open PR. ## What's changed and what's your intention? Extract instant sample selection into `select_instants`, operating on the existing `RawTimestamps` and `SeriesFields`. It returns source row indices and evaluation timestamps without converting the payload into a new representation. Use it in the existing `InstantManipulate` path when timestamps are non-null, strictly increasing milliseconds and the configured fields are Float64. Other inputs retain the existing path, including duplicate timestamps and Histogram columns. Selection preserves lookback boundaries, offsets, stale markers, ordinary NaNs, null samples, and the existing output-point limit. This is one extracted slice of previously local work: **5 files, +732/-3 lines, including tests**. It does not add native planning, scanner/Flight integration, configuration, storage formats, or any new Histogram type. ### Verification - Passed: scoped Rust formatting, `hawkeye format`, `hawkeye check`, `git diff --check`, and normal commit hooks. - Running on this exact extracted branch; results not yet claimed: `cargo check --locked -p promql --all-targets` and `cargo nextest run --locked -p promql --retries 0`. - Independent source review is in progress. Kept as a draft until verification and review are complete. - Full-workspace tests, full-workspace Clippy, and SQLness have not been run for this extracted branch. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
88a1e684a0 |
feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors (#9656)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Recognize `SeriesScanExec` in query metrics and preserve typed errors through the record batch adapter: - `PlanMetrics`: recognizes `SeriesScanExec` nodes so region-scoped stats attribution works. - `RecordBatchStreamAdapter`: preserves the original typed error via `BoxedError` instead of wrapping into a generic execution error. This lets downstream recover the specific status code (e.g., `InvalidArguments` instead of `EngineExecuteQuery`). **2 files, +203/-9 production + 1 sqlness expected result.** ### Error code change The typed error preservation changes the error code for the `semantic_graph` upper-bound-only query from `3001(EngineExecuteQuery)` to `1004(InvalidArguments)`. The error message text is unchanged. The new code is correct: the original error was an argument validation error, not a generic execution failure. The sqlness expected result has been updated. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-recordbatch` → **65/65 passed** ✅ - sqlness expected result updated for the typed error code change ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
|
5d93b8177c |
feat(common-query): adapt ordered record batches to raw series (#9652)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. Builds on shared series types from #9598/#9599. ## What's changed and what's your intention? Add `record_batches_to_raw_series` in `common-query`: converts an ordinary DataFusion `RecordBatch` stream into contiguous raw series chunks. The adapter validates column types at construction (time must be Timestamp(Millisecond, None), tags must be string/dictionary, fields must be Float64 or native histogram), encodes series identity via Arrow RowConverter, and produces `RawSeriesChunk`s with explicit labels and contiguous ordering. Float64 fields share source buffers (zero-copy); histograms reuse the existing `histogram_samples_from_arrow` bridge. Chunking merges equal-key runs within and across batches; a key change or end-of-input sets `is_last`. Timestamps must be strictly increasing within a series. Empty batches are skipped with a bounded poll loop. **3 files, +1666/-0 lines, including 20 tests.** No production callers yet — consumed by follow-up scan wiring PRs. ### Verification - `cargo check --locked -p common-query --all-targets` ✅ - `cargo nextest run --locked -p common-query --retries 0` ✅ - `cargo clippy --locked -p common-query --all-targets -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
fb9bda2744 |
feat(store-api): add raw-series scanner preflight and ordering contract (#9651)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add two default methods to `RegionScanner` for raw-series scan preflight: - `raw_series_schema()` — non-consuming check whether this scanner can provide raw series for a given label source. Returns `None` (unsupported) by default; overrides promise `scan_series_partition` can execute. - `raw_series_ordering()` — returns the ordering guarantee without consuming scanner state. Defaults to `Contiguous` (conservative for legacy sources). Both are additive-only trait defaults with no behavior change for existing implementors. **32 insertions, 0 deletions, 1 file.** ### Verification - `cargo check --locked -p store-api --all-targets` ✅ - `cargo nextest run --locked -p store-api --retries 0` ✅ - `cargo clippy --locked -p store-api --all-targets -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
e25fa2bd8d |
Merge remote-tracking branch 'upstream/narthex' into feat/narthex-instant-eval-pr
# Conflicts: # src/promql/src/lib.rs |
||
|
|
869a32fdcb |
feat(query): add experimental_native_promql config option (#9662)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Add the `experimental_native_promql` configuration option (default: `false`): - `QueryOptions.experimental_native_promql`: new boolean field, default `false`. - Four example TOML configs updated with the commented-out option. - `config/config.md` regenerated with the new option. The option is stored in process configuration only. Session-level override and the actual gate in query planning are in follow-up PRs. **6 files, +24/-0 lines.** ### Verification - Compiles standalone: no unused fields or methods. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [x] This PR requires documentation updates. (config.md regenerated) - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
|
b27ca7c7bd |
feat(common-query): declare raw series frame ordering in schema metadata (#9655)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add ordering metadata to raw series frame schemas: - `raw_series_frame_schema_with_ordering()`: builds the canonical v1 frame schema and stamps the producer's ordering (`contiguous` or `read_key_then_time`) into Arrow schema metadata. Rejects `Interleaved` (no v1 wire representation). - `raw_series_frame_ordering()`: reads the declaration back, defaulting absent metadata to `Contiguous` (legacy frames), rejecting unknown values. - `pack_raw_series` now declares ordering on every yielded batch via `RecordBatch::with_schema`. Includes a wire compatibility fixture (`tests/compatibility/wire/raw_series_ordering/`) with 4 cases. **4 files, +277/-7 lines.** ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
2d9b2a970f |
feat(mito-codec): decode scan-bound primary key labels (#9650)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Builds on `narthex` base types (#9598, #9599, #9602, #9603). No dependency on other open PRs. ## What's changed and what's your intention? Add `PrimaryKeyLabelDecoder` to `mito-codec`: given a scan-bound `RegionMetadata` snapshot and a primary key byte sequence, decode the complete metric name and label set (or a projected subset). Supports both dense and sparse primary key encodings. Sparse keys get a new `visit_label_ids` method that validates the full tail (rejecting duplicates, reserved IDs, and truncated framing) before reading values. Dense keys check that all fields are present and decode each value with exact type fidelity. This is one extracted slice of previously local work: **4 files, +1046/-13 lines, including tests**. It has no production call site yet — it will be consumed by the raw series scan wiring in follow-up PRs. ### Verification - `cargo check --locked -p mito-codec --all-targets` ✅ - `cargo nextest run --locked -p mito-codec --retries 0` → **62/62 passed** ✅ - `cargo clippy --locked -p mito-codec --all-targets -- -D warnings` ✅ - `hawkeye format`, `hawkeye check`, `git diff --check` ✅ - Full-workspace tests and SQLness not run for this extracted branch. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
0ce358dde5 |
feat(promql): match vectors holding native histograms (#9696)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) MATCH-HIST in internal-docs#27. Builds on #9679 (`vector_match`) and #9686 (`FloatHistogram`). ## What's changed and what's your intention? ### Problem `binary_vectors` and `set_vectors` rejected native histogram samples with `NotImplemented`. ### Change - `vector_match::elem_binop` ports Prometheus `vectorElemBinop` for histograms: - histogram `+`/`-` histogram, compacted, with `-` marking the result as a gauge; - histogram `==`/`!=` histogram, layout-sensitive like Go's `Equals`, keeping the left histogram; - histogram `*` float, float `*` histogram, histogram `/` float, compacted. - Every other pairing that involves a histogram, and a `+`/`-` between exponential and custom buckets, is skipped: no result, `bool` included, and the pair does not count as a match. This follows Prometheus, which reports these as annotations before recording the match. - The one-to-one "multiple matches" check now counts the matches actually made per group and point, so skipped pairs do not count. - Set operators carry histogram values through; results with histograms are built with `GridSeriesBuilder::push_samples`. - Operand presence no longer reads values, and the helpers that could only fail on histograms are infallible now. Review focus: - `elem_binop` against `vectorElemBinop`: which pairings keep, filter or skip. - `binary_field`: the one-to-one match counting and the ordering of the skip against it. ### Non-goals - The experimental trim operators (`</`, `>/`). - Annotations Prometheus attaches to skipped pairs or counter-reset collisions. ### Constraints None. Public signatures are unchanged. ### Verification - `histogram_operators_match_prometheus`: 280 cases (20 per operator: `+ - * / % ^ atan2 == != > <= and or unless`) from the Prometheus v3.15.0 engine over vectors mixing floats and native histograms drawn from small per-case pools, with `on`/`ignoring`, `group_left`/`group_right` and `bool` (`vector_match/histogram_cases.txt`). Results, error classes, floats within 1e-10 and histograms bucket by bucket match. - Mutation check: all 10 faults fail a test (keeping incompatible pairs as filtered matches, not skipping mixed bucket kinds, no gauge hint, no compaction, `/` as `*`, `==` keeping the right side, `!=` as `==`, ignoring `group_right`'s swap, set operators dropping histograms, `float * histogram` unsupported). - `cargo nextest run -p promql`: 359 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
8b85a3c58f |
feat(promql): match vectors holding native histograms
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
cd3292f59e |
feat(promql): evaluate range functions over native histogram samples (#9695)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) EVAL-HIST in internal-docs#27. Builds on #9673/#9674/#9677 (`RangeEvaluator`) and #9686 (`FloatHistogram`). ## What's changed and what's your intention? ### Problem `RangeEvaluator` rejected native histogram samples with `NotImplemented`, so no range function could read a series holding native histograms, alone or mixed with floats. ### Change - `RangeEvaluator` keeps the histogram samples of each field apart from its floats, drops a histogram whose sum is the stale marker, and evaluates windows holding histograms in `range_eval::histogram` following the histogram branches of the Prometheus functions: - `min/max/stddev/stdvar/quantile/mad_over_time`, `deriv`, `predict_linear`, `double_exponential_smoothing`, `ts_of_min/max_over_time` read only the floats, as before; a window without floats has no result. - `count_over_time`, `present_over_time`, `ts_of_first/last_over_time` count both kinds; `first/last_over_time` return the earliest/latest sample, histogram or float. - `resets` and `changes` walk both kinds in time order: a change of kind is one edge, two histograms compare with `DetectReset` / `Equals`. - `sum/avg_over_time` and `rate/increase/delta` give nothing for a mixed window; over histograms alone they port the Kahan sum/mean and `histogramRate` with `extrapolatedRate`'s extrapolation. - `irate/idelta` take the last two samples of either kind; a float and a histogram give nothing. - An evaluator built from a bare kernel (`with_kernel`) still rejects histograms. - `GridSeriesBuilder::push_samples` writes rows holding histograms; fields become mixed samples when they hold any. - `histogram_eval::sum_or_avg` is the histogram sum/mean shared by `aggregate_histograms` and the range functions; the aggregation still compacts its result, the range functions do not, as in Prometheus. - `range_eval::function::extrapolation_factor` is the extrapolation step shared by the float and histogram rates. Review focus: - `range_eval/histogram.rs`: each arm against the Prometheus function it names; `histogram_rate` against `histogramRate` (nulling the first sample on a reset between the first two, schema reduction, adding back at resets). - `RangeEvaluator::evaluate`: retention and the skip to the earliest sample now cover histograms too. - `GridSeriesBuilder::push_samples`: histogram indices stay aligned with the float rows across gaps. ### Non-goals - Anchored and smoothed range selectors (NEW-EVAL-MOD); start-timestamp based resets, which Prometheus only uses behind a feature flag. - Annotations (warnings and infos) Prometheus attaches to mixed or incompatible windows. ### Constraints None. `RangeEvaluator`'s public interface is unchanged. ### Verification - `histogram_range_functions_match_prometheus`: 520 random cases (20 per function, 26 functions) plus 6 fixed counter histories, from the Prometheus v3.15.0 engine over one or two series mixing floats and native histograms, with ranges of 15s to 75s and offsets (`range_eval/histogram_cases.txt`). Floats compare within 1e-10 (1e-6 for `deriv`, `predict_linear`, `stddev`, `stdvar`), histograms bucket by bucket. - Mutation check: 16 of 17 faults fail a test (comparison order of first/last, counting only floats, mixed-edge counting, histogram resets as changes, mixed sums and rates, the counter zero point, adding back at resets, nulling the first sample, the gauge hint, irate's reset check, float-only functions answering for histograms, histogram retention, builder index alignment). The remaining one, skipping the schema reduction before the subtraction, only changes rounding: a later higher-schema sample is a reset whose addition reduces the schema anyway. - `histogram_samples_need_a_function_and_drop_stale_markers` pins the stale histogram drop and the bare-kernel rejection. - `cargo nextest run -p promql`: 358 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
17f30f594a |
feat(promql): evaluate range functions over native histogram samples
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
a78d4aa313 |
feat(promql): cover absent_over_time and vector(), sort quantile_over_time like Go (#9694)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Completes SEL-ABSENT (`absent_over_time`) and SEL-SCALAR (`vector()`) from internal-docs#27, and the `quantile_over_time` limitation left by #9674. ## What's changed and what's your intention? ### Problem - `quantile_over_time` sorted windows with a stable sort. Go's `sort.Sort` is unstable for more than 12 elements, so in a long window holding both +0 and -0 the zero a rank lands on, and the sign of the result, could differ from Prometheus. - `absent_over_time` and `vector()` had no native path. ### Change - `quantile_over_time` sorts with `go_sort::sort_by` and `vectorByValueHeap.Less` (NaN first). - `absent_over_time(v[r])` is `select::absent` over the series of `present_over_time(v[r])`, with labels from `select::absent_labels` on the matchers of `v`; Prometheus derives both the same way. The rustdoc of `absent` states this, and a golden test pins it. No new kernel. - `vector(s)` needs no kernel: a scalar is already a label-less grid series with a value at every point. The `select` module docs state this. ### Non-goals Wiring `absent_over_time` and `vector()` into the planner. ### Constraints None. ### Verification - `quantile_picks_the_signed_zero_go_picks`: a 20-sample window of mixed zeros, a NaN and numbers; the sign of `quantile_over_time(k/19)` matches Go's `sort.Sort` on amd64 for k = 1..19, and fails with the previous stable sort. - `absent_over_time_is_absent_over_present_over_time`: 120 cases from the Prometheus v3.15.0 engine (`range_eval/absent_over_time_cases.txt`) with ranges of 5s to 45s and equality/inequality matchers. - `cargo nextest run -p promql`: 357 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
41cfd237f6 |
test(promql): pin absent_over_time as absent over present_over_time
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
1147fafa5d |
fix(promql): sort quantile_over_time windows with Go's sort
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
7c8c74ac9b |
fix(promql): keep no samples past a skipped anchored window and trim Narthex operators (#9690)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Stacked on #9689; review the commits after `47f89b34b7`. Follow-up to the Narthex operator PRs merged into `narthex` (#9673–#9687). ## What's changed and what's your intention? ### Problem An `@`-anchored range selector whose one window ends before the series starts skips the cursor past the point it reads. `evaluate` cleared retained samples only when `cursor == points_read`, so every later chunk of that series stayed in memory until `is_last`, e.g. `x[5m] @ start()` over a series that starts after `start()`. Results were unaffected. The operator PRs also left duplicated helpers, a dead branch, a redundant reset, a duplicate test and comments that went stale during review. ### Change - `RangeEvaluator::evaluate` drops retained samples once `cursor >= points_read`. - `METADATA_LABELS` and `external` are defined once in `grid_series`; `range_eval`, `vector_match`, `map_time`, `select` and `histogram_eval` use them. - `kahan_inc` (Prometheus `kahansum.Inc`) is defined once in `functions`, on top of `compensated_sum_inc`, replacing the copies in `range_eval::function` and `histogram_eval::native`. - Removed: the unreachable `k == 0` return in `topk`, the field reset in `RangeEvaluator::finish_series` (`evaluate(None)` already leaves the fields empty), and `overlapping_window_spans_chunks`, whose only fault class is also caught by the random-chunk reference and the golden tests. - Fixed comments that no longer match the code: `RangeEvaluator::try_new` input, where the smoothing factors are checked, `binary_vectors` label dropping, `Results`, `rank`. ### Non-goals `quantile_over_time` still sorts with a stable sort; switching it to `go_sort` changes behavior and belongs in its own PR. ### Constraints None. No public interface changes. ### Verification - `anchored_window_before_the_series_retains_nothing` fails before the `>=` change and passes after it. - `cargo nextest run -p promql`: 356 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
d4c4da7b6e |
chore(promql): import parent items through crate paths (#9689)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Follow-up to #9673, #9686 and #9685, merged into `narthex`. ## What's changed and what's your intention? ### Problem `make fmt-check` fails on `narthex` at `4969bca60e`: `scripts/check-super-imports.py` rejects five top-level `use super::` imports in `histogram_eval/{native,float_histogram,aggregate}.rs` and `range_eval/function.rs`. ### Change The five imports use `crate::` paths. No behavior change. ### Non-goals None. ### Constraints None. ### Verification - `make fmt-check`: clean. - `cargo nextest run -p promql`: 356 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`: clean. - On `4969bca60e`, the rest of the full suite already passes: `make clippy`, `make test` (9950 passed; `json2::test_json2_single_mem_range_flush` failed once on a MySQL port bind and passed on retry) and `cargo sqlness bare`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [ ] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
438c44c109 |
chore: merge fix/narthex-super-imports into chore/narthex-ablation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
4761d39690 |
refactor(promql): drop a redundant field reset and a duplicate range window test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
72b06dc730 |
fix(promql): drop retained samples when an anchored window is skipped
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
47f89b34b7 |
chore(promql): import parent items through crate paths
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
a19e8cbd46 |
chore(promql): fix stale docs and drop a dead branch in Narthex operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
b37f47022b |
refactor(promql): share metadata labels, error mapping and Kahan increments across Narthex operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
4969bca60e |
feat(promql): add native histogram arithmetic and sum/avg aggregation (#9686)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9685 for the native histogram bucket bounds and the golden fixture helpers; review only `feat(promql): add native histogram arithmetic and sum/avg aggregation`. ## What's changed and what's your intention? ### Problem Native histogram range functions, aggregations and binary operators all need histogram arithmetic. The ordinary `common_query::native_histogram::NativeHistogram` adds without Kahan compensation and compacts with its own algorithm, so its results differ from Prometheus in value bits and bucket layout. The native path has no `sum`/`avg` over native histograms either. ### Change - `histogram_eval::float_histogram::FloatHistogram` ports Prometheus `FloatHistogram` (`model/histogram/float_histogram.go`): `Add`, `KahanAdd`, `Sub`, `Mul`, `Div`, `Compact(0)`, `CopyToSchema`, `DetectReset`, `Equals` and `HasOverflow`. The port covers: - zero bucket reconciliation (`zeroCountForLargerThreshold`, `trimBucketsInZeroBucket`), including the compensation Go keeps across abandoned passes; - resolution reduction, plain and Kahan; - the intersection of mismatched custom bounds and the matching reset detection; - counter reset hint merging. Each side is kept as `(bucket index, count)` sorted by index. The bucket set and the order of additions match Go's span splicing. A histogram read from a view keeps its spans until its buckets change, because `Equals` compares span layouts; afterwards consecutive indices form one span, the layout `Compact` produces. - `histogram_eval::aggregate_histograms` follows the histogram paths of Prometheus `aggregation` for `sum` and `avg` within one group. Histograms are added in group order with `KahanAdd`. `avg` switches to an incremental mean once the plain sum overflows. A point yields nothing when the group also has a float there or mixes exponential and custom buckets; any other addition error fails the aggregation. Grouping stays with AGG-BASIC. - `native::bound` takes exponential bounds from Go's `exponentialBounds` table with `Ldexp`, as `getBoundExponential` does. Widened zero thresholds are set to these bounds and compared exactly in `DetectReset` and when adding buckets, and some table entries are one ulp off `2^(idx / 2^schema)`. - `sub`, `copy_to_schema`, `detect_reset` and `equals` have no caller on this branch yet. They are for the native histogram range functions and binary operators, which build on other branches, so the module allows dead code outside tests. Review focus: - `add_buckets`: `other` buckets whose upper bound is within the zero threshold are skipped only while they lead; subtraction inserts negated counts, including `-0`. - `zero_count_for_larger_threshold` and `reconcile_zero_buckets`: the widened threshold, the Kahan compensation of the zero count, and the compaction after trimming, which drops only buckets whose count and compensation are both zero. - `kahan_add`: which side is reduced to the lower schema, and the alignment of the compensation buckets with the receiving ones. - `detect_reset`: both sides start from the current zero threshold; merged buckets of a higher-resolution predecessor sum from zero. - `aggregate_point`: the overflow switch to the incremental mean reuses the compensation from before the failed addition. - The aggregation writes its histograms from the first point with a result instead of holding one slot per point of the whole input span. ### Non-goals - Native histogram range functions (EVAL-HIST) and binary operators (MATCH-HIST). - Grouping and float aggregation (AGG-BASIC). - `Equals` after an uncompacted `Add`/`Sub` compares the compacted layout. PromQL compacts every such result before using it. ### Constraints - Sums of floats and histograms depend on the order of the series. Prometheus adds them in the order the head index returns them (creation order). The kernel adds them in the order of `group`, which the caller decides. - Prometheus panics in `addBuckets` on histograms with zero-length spans; the port does not. The aggregation fixtures compact their inputs for that reason. ### Verification - `cargo nextest run -p promql`: 319 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `histogram_arithmetic_matches_prometheus`: 723 operations from Go's `FloatHistogram` methods on random histograms (`histogram_eval/arithmetic_cases.txt`). Inputs include: - schemas -4 to 8 and custom buckets with matching and mismatched bounds; - zero thresholds from 0 to 0.5, mixed exponential and custom operands, and all reset hints; - reset candidates with missing buckets, lower zero counts, lower resolution, wider zero buckets and bounds that underflow to zero; - three fixed cases random inputs do not reach. Results compare every field, the bucket indices and count bits of both the sum and the Kahan compensation, and the collision and reconciliation flags. - `native_histograms_match_prometheus` now also covers 240 `sum`/`avg` cases with and without `by (job)`, from the Prometheus engine with inputs in creation order, plus a fixed `avg` case whose overflow switch carries a nonzero compensation. Aggregated histograms compare bit for bit. - Mutation check, each of these fails a test: - Arithmetic, 21 faults: - no zero bucket reconciliation in `Add`, plain addition inside `KahanAdd`, a target index without the `- 1`; - no zero-threshold skip, no hint collision; - `>` instead of `>=` when mapping custom bounds; - resetting the zero count compensation per pass, no compaction after trimming, a compaction that ignores the compensation; - reset detection that ignores the zero count, missing buckets or schema merging, or skips at a zero threshold; - `Equals` by value or without the layout; - division by zero keeping buckets, `CopyToSchema` keeping the hint, a negative factor not setting the gauge hint; - restarting the custom-bound roll-up, dropping the custom-bucket compensation, plain resolution reduction in `KahanAdd`. - Aggregation, 8 faults: plain `Add` for `sum`, dropping the final compensation, never switching to the incremental mean, discarding the compensation at the switch, output despite floats or incompatible schemas, no final compaction, adding the compensation before dividing. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
db0a531d7d | chore: merge narthex into feat/narthex-histogram-arith | ||
|
|
0d2f1ff6cd |
feat(promql): evaluate classic and native histogram functions over grid series (#9685)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9684 for `merge_same_labelset` and the golden fixture helpers; review only the two commits `feat(promql): evaluate classic histogram_quantile and histogram_fraction over grid series` and `feat(promql): evaluate native histogram functions over grid series`. ## What's changed and what's your intention? ### Problem The native path has no kernels for the histogram functions: `histogram_quantile` and `histogram_fraction` over classic (`le`-bucketed) and native histograms, and `histogram_count`, `histogram_sum`, `histogram_avg`, `histogram_stddev` and `histogram_stdvar` over native histograms. ### Change `histogram_eval::evaluate_histograms` takes the evaluated input series and a `HistogramFunction` (`Quantile(q)` or `Fraction { lower, upper }`, constants as in the ordinary planner) and follows Prometheus `funcHistogramQuantile`, `funcHistogramFraction` and `resetHistograms`: - Float series with the same labels other than `le` form a classic histogram; a series whose `le` does not parse is ignored. Parsing follows Go's `strconv.ParseFloat`: finite literals out of the f64 range (`1e400`) and a signed NaN are rejected, and `_` between digits is accepted. - Native histogram samples are evaluated one by one. At a point where a native histogram and classic buckets of the same labels (`le` aside) are both present, neither is evaluated. - Each field is evaluated on its own. Results drop `le`, `__name__`, `__type__` and `__unit__`; series that then share a label set go through `merge_same_labelset`. Classic buckets (`histogram_eval::classic`) port `BucketQuantile` and `BucketFraction`. At each point the present buckets are sorted with `go_sort::sort_by` and Go's `<` comparator, so +0/-0 and NaN bounds keep Go's order, and buckets with equal bounds are coalesced. For quantiles, counts within a relative 1e-12 of the previous count or below it are raised to it. The quantile bucket is found with Go's `sort.Search` bisection, so NaN counts select the same bucket. Native histograms (`histogram_eval::native`) port `HistogramQuantile`, `HistogramFraction` and `histogramVariance`. The port includes: - the bucket order and zero-bucket clipping of `AllBucketIterator`/`AllReverseBucketIterator`; - bucket bounds `2^(idx / 2^schema)` from the standard library, with `f64::MAX` for the last bucket of finite values as in Go's `getBound`; This covers the reverse iteration for `q >= 0.5`, the count clamp, custom-bucket edge buckets, and the trailing walk that Prometheus runs for NaN observations, which leaves the last bucket in place for interpolation. `histogram_eval::map_native_histogram` applies `NativeHistogramFunction::{Count, Sum, Avg, StdDev, StdVar}` to the histogram samples of one series, dropping float samples and `__name__`; colliding label sets are left to `merge_same_labelset`, as for other per-series kernels. Review focus: - `classic::bucket_quantile`: the `q` checks come before the `+Inf` check, the `b == 0 && upper_bound <= 0` branch, and the interpolation `start + (end - start) * (rank / count)`. - `classic::ensure_monotonic`: `prev` is not advanced when a count is raised. - `native::all_buckets` and `native::bound`: iteration order, clipping to the zero threshold, and the last finite bucket (`f64::MAX`). - `native::quantile`: the last visited bucket (empty or not) is the one interpolated, including after the NaN-observation walk. - `evaluate_histograms`: the classic/native conflict is decided per point and field. ### Non-goals - Native histogram `sum`/`avg` aggregation, rate functions and arithmetic. - Non-constant `q`, `lower` and `upper`. - Planner wiring. ### Constraints - The ordinary `HistogramFold::evaluate_row` differs from Prometheus for classic histograms: it errors when the last bucket is not `+Inf`, has no small-delta tolerance, replaces non-finite counts with the previous count, and interpolates as `lower + (upper - lower) / count * rank`. The ordinary native quantile (`common_query::native_histogram::quantile_histogram_view`) has no reverse iteration for `q >= 0.5` and derives bounds with `powf` instead of Go's table. This PR follows Prometheus 3.15. - `le` values in Go-only float syntax (hex floats such as `0x1p-2`) do not parse with Rust's `f64::from_str` and are ignored. ### Verification - `cargo nextest run -p promql`: 318 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `classic_histograms_match_prometheus`: 300 fixtures produced by the Prometheus v3.15.0 engine built for amd64 (`histogram_eval/classic_cases.txt`). - Bucket layouts: non-positive bounds, a bound written twice (`1`/`1.0`), no `+Inf`, only `+Inf`, unparsable, out-of-range or missing `le`, and the `inf` spelling. - Counts: monotonic, non-monotonic, around the 1e-12 tolerance, NaN, `+Inf`, negative, zero and subnormal. - Inputs are passed in reverse order. 32 cases expect the duplicate-labelset error. - `native_histograms_match_prometheus`: 420 fixtures (`histogram_eval/native_cases.txt`, 60 per function). - Histograms: schemas -4, -1, 0, 1, 3, 8 and custom buckets, with gaps, zero-length spans, empty buckets, zero thresholds from 0 to 0.5, NaN sums with uncounted observations, and counts below the bucket total. - Inputs mix floats and histograms, and include classic buckets of the same labels. - Inputs are read back from the engine, so they carry its span layout. - Values in both fixtures compare within promql-compat's relative tolerance (1e-10, 1e-6 for `histogram_stddev`/`histogram_stdvar`), NaN as any NaN. - Mutation check, classic: each of these fails a test: - no coalescing (quantile or fraction), no forced monotonicity, no small-delta tolerance, an absolute tolerance at zero; - a linear search, no `+Inf` check, `q` checks after it, no non-positive first-bucket branch, the ordinary interpolation order; - accepting `1e400`, keeping `__name__`, a zero lower bound for a non-positive first fraction bucket, interpolating inside a `-Inf`-wide bucket, no merge, sorting with NaN bounds last. - Mutation check, native: each of these fails a test: - always iterating forward, no NaN-observation walk, no zero-threshold clipping, no clamp of the count; - shifted bucket bounds, no `f64::MAX` for the last finite bucket; - a plain variance sum, a geometric midpoint for custom buckets; - the histogram count as the fraction total under NaN observations, linear interpolation in exponential buckets; - no custom `-Inf` branch, no zero-bucket side adjustment; - evaluating conflicting classic and native histograms, keeping `__name__`, no zero bucket in reverse order. - `histogram_functions_drop_metadata_labels`: `histogram_quantile` drops all metadata labels, `histogram_count` only `__name__`. - `classic_bounds_keep_the_first_signed_zero`: bounds +0 and -0 compare equal and keep input order, so `1 / histogram_quantile(0, ...)` over `le` "+Inf", "0", "-0" is +Inf. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
041a48d456 | chore: merge narthex into feat/narthex-classic-histogram | ||
|
|
12e07cf727 |
feat(promql): evaluate time, date and label rewriting functions over grid series (#9684)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test reference; review only `feat(promql): evaluate time, date and label rewriting functions over grid series`. ## What's changed and what's your intention? ### Problem The native path has no kernels for `time()`, the date functions (`minute`, `hour`, `day_of_month`, `day_of_week`, `day_of_year`, `days_in_month`, `month`, `year`) or `label_replace`/`label_join`. ### Change - `map_time::time` and `map_time::date_of_eval_time` produce one unlabelled series over the schema's grid from the evaluation timestamps (`ms / 1000` for `time()`, whole seconds truncated towards zero for the date functions, as Prometheus `funcTime`/`dateWrapper`). - `map_time::map_date` reads every float value as Unix seconds and drops `__name__`, `__type__` and `__unit__`. The value goes through Go's `int64(f)` as on amd64 (NaN and out-of-range values become `i64::MIN`) and a port of Go's `time` absolute-date arithmetic, including its wrapping for instants Go cannot represent, which decides the result for those inputs. All-float series are mapped chunk by chunk without copying the time axis; histogram samples are dropped and the series is trimmed again. - `map_label::label_replace` anchors the regex as `^(?s:...)$`, expands `$1`/`${name}`/`$$` with Go's `regexp.Expand` rules, and leaves unmatched series unchanged. `map_label::label_join` joins source values, an absent label joining as `""`. An empty result removes the label in both. Empty source or destination label names and invalid regexes are `Plan` errors with Prometheus' messages. - `grid_series::merge_same_labelset` follows Prometheus `mergeSeriesWithSameLabelset`: series that share a label set (empty values counting as absent) become one series, merged field by field; a field with values from two of them at one point fails with "vector cannot contain metrics with the same labelset". The label functions call it; `map_date` leaves it to the caller, as other per-series kernels that drop `__name__` do. - `promql` now depends on the workspace `regex` crate, which is already in the build graph. Review focus: - `map_time::DateFunction::apply` and `civil`: the port of Go's `absSeconds`/`absDays` arithmetic; `days_in_month` uses the proleptic Gregorian leap rule on the computed year. - `map_time::go_int64`: NaN and values outside `[-2^63, 2^63)` map to `i64::MIN`. - `merge_same_labelset`: the fast path returns the input untouched when no label sets collide; colliding groups are rebuilt row by row and fail on any shared point. - `label_replace` expands its template with Go's `regexp.Expand` rules: a reference with a leading zero (`$01`) or of more than nine digits names a group, unlike `regex::Captures::expand`. ### Non-goals - `timestamp()` (EVAL-TS), which needs sample timestamps from instant selection. - Merging histogram series that collide after a label function (`NotImplemented`). - Planner wiring. ### Constraints - Every per-series kernel that drops `__name__` (range functions, date functions, math) can leave two series with one label set. Prometheus merges or rejects them per step; on the native path that is `merge_same_labelset`, which the caller must apply after such kernels. - Regexes compile with the Rust `regex` crate, as the ordinary planner does. Perl classes (`\d`, `\w`, `\s`, `\b`) are Unicode-aware there and ASCII-only in Go's RE2, so non-ASCII label values can match differently. The dialects also differ in what compiles: `\Q...\E` is rejected, `(?x)` and repeat counts above 1000 are accepted, and `\<`, `\>` are word boundaries instead of literal characters. The rustdoc of `label_replace` lists the same differences. - `go_int64` follows amd64. On arm64 Go saturates (`NaN` becomes 0), so Prometheus' own results for non-finite inputs differ by architecture. ### Verification - `cargo nextest run -p promql`: 315 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `time_functions_match_prometheus`: 200 fixtures produced by the Prometheus v3.15.0 engine built for amd64 (`map_time/prometheus_cases.txt`): `time()` and argument-less date functions over grids from 1685 to 2254, and date functions over values including NaN, ±Inf, ±1e300, ±9.2e18, leap-day edges of 1600/1900/2000/2100/2400 and years 0 and 10000. Values compare bit for bit after `merge_same_labelset`; 6 cases expect the duplicate-labelset error. - `label_functions_match_prometheus`: 200 fixtures (`map_label/prometheus_cases.txt`) with random destinations including `__name__`, absent sources, templates with valid and invalid `$` references, anchoring-sensitive regexes and invalid ones; 25 expect errors, the rest compare label sets and value bits. - Mutation check: each of these fails a test: Go's NaN and out-of-range conversion replaced with a saturating cast or NaN-to-0; floor instead of truncating division of the evaluation time; whole-second `time()`; dropping the 400-year rule; leap day counted in January and February; hour from the Unix seconds instead of Go's absolute seconds; no leap February in `days_in_month`; leap year taken from the March-based year; keeping `__name__`; mapping only the first chunk; regex without the `(?s:...)` group or without anchors; setting an empty label instead of removing it; skipping absent `label_join` sources; skipping the merge; skipping the duplicate check; comparing empty label values as present. - `replacements_expand_like_go`: 16 templates with expectations taken from Go, including `$1²`, where Go's Unicode classes end the name before `²`. - `merge_combines_disjoint_fields_at_one_point`: different fields at one point merge, one field present twice fails. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
|
6485e9c810 | chore: merge narthex into feat/narthex-map-label-time |