mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 06:46:56 +00:00
50002c6830db32ea2b25728aaa468259d14f03d1
6582
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Lei, HUANG
|
50002c6830 |
feat(promql): add evaluated series math kernels (#9672)
* feat(promql): add evaluated series math kernels Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop metric name in evaluated series math like Prometheus Prometheus' simpleFloatFunc drops __name__ together with __type__ and __unit__. Check abs, ceil, floor, sqrt and round against fixtures generated by the Prometheus v3.15.0 engine. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop empty labels with the metric name in series math Prometheus has no empty-valued labels, so a kept empty value would hide a same-labelset collision from the caller. Round shares its calculation with the UDF, so check it only against the Prometheus fixtures. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: exclude promql golden fixtures from typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(promql): point series round at Prometheus funcRound Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: allow the Unicode category Nd in typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): drop reserved and empty labels in one rebuild drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
 discord9
|
a1052c0c0f |
feat(query): add raw-series preflight and execution surface to query traits (#9661)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
cbc2a729a5 |
feat(common-meta): add raw-series preflight and execution surface to Datanode trait (#9659)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
5f0f344908 |
feat(table): scan-bound typed raw series production via SeriesScanExec (#9664)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9657** (raw request types). Also requires **#9651** (scanner preflight), **#9655** (frame ordering), **#9656** (recordbatch adapter). ## What's changed and what's your intention? Add scan-bound typed raw series production to `TableScanExec`: - When the scan binding has a `RawSeriesRequest` and preflight passes, produces `SeriesScanExec` that streams typed `RawSeries` frames. - Shares MVCC/projection/filter semantics with the existing RecordBatch path. - Strictly opt-in: absent or declined preflight returns the ordinary plan unchanged. **2 files, +475/-2 production + 1506 test lines.** Note: diff includes cherry-picked commits from #9651 and #9656. Once they merge, the diff will shrink. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
18e11944c9 |
feat(promql): add native instant manipulate execution over raw series (#9660)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9654** (source materializer). Also requires **#9649** (instant eval) and **#9655** (frame ordering). ## What's changed and what's your intention? Add native instant manipulate execution over raw series streams: - `NativeInstantMapping`: preflighted mapping from typed raw series to query output schema. - `NativeInstantManipulateExec`: ExecutionPlan that consumes `SendableRawSeriesStream`, materializes labels, selects instants, and emits aligned RecordBatches. - `InstantManipulate::try_to_native_execution_plan()`: validates the raw frame schema and constructs the native plan. **5 files, +1830/-0 lines.** No callers wired yet. Note: diff currently includes cherry-picked commits from #9649 and #9655. Once those merge into narthex and the base is updated, the diff will shrink to only the 5 native exec files. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p promql` → **357/357 passed** ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
b3707c28ac |
feat(promql): merge ordered raw series inputs (#9653)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Depends on #9650** (labels decoder). Includes `materialize_raw_labels` — if the source-only PR merges first, this rebases cleanly. ## What's changed and what's your intention? Add `promql::source::merge_ordered_raw_series`: merges multiple ordered raw-series streams into one globally ordered stream by series key and time. All inputs must share the same schema and label binding, be `KeyThenTime` ordered, and have undecoded labels. The merge recomputes per-series `is_last` globally. Errors are terminal; already-emitted prefixes are not rolled back. **8 files, +4118/-0 lines (622 production + 3489 tests + 7 config).** ### Verification - Compiled and tested with #9650 merged: `cargo check --all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D warnings` ✅ - Cannot compile standalone until #9650 merges. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 discord9
|
200ed3ce36 |
feat(table): scan-bound typed raw series production via SeriesScanExec
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f354e3423a |
feat(promql): add native instant manipulate execution over raw series
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
535923629b |
feat(promql): merge ordered raw series inputs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
83a0d58980 |
feat(query): capture native raw-series routes and extend MergeScan for typed frames (#9663)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9662** (native config). Also requires **#9655** (frame ordering metadata). ## What's changed and what's your intention? Add native raw-series route capture and MergeScan typed-frame support: - `CaptureNativeRawRoute`: captures region routing during planning when `experimental_native_promql` is enabled. - `NativeRawSource`: borrows the captured route for native raw-series execution. - Schema identity validation for `greptime.series.` metadata. - Wires `experimental_native_promql` from `QueryEngineState` → `DfQueryPlanner` → `DistExtensionPlanner`. **4 files, +1746/-113 lines.** Note: diff includes cherry-picked commits from #9655. Once it merges, the diff will shrink. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
440ecb909b |
feat(common-query): add raw series request binding and preflight types (#9657)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add raw series request types for the named raw-series preflight contract: - `RawSeriesRequest` / `RawSeriesTag`: versioned binding for a single-region raw-series request. - `PreparedNamedRawSeries` + encode/decode: JSON body for the `PREPARE_RAW_SERIES_ACTION` Flight action. Strict serde (`deny_unknown_fields`), version 1 only. Includes wire compatibility fixtures. **7 files, +661 production + wire fixtures.** ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
e0af59f14c |
feat(promql): materialize complete raw source labels (#9654)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Depends on #9650** (labels decoder). Must merge after (or be rebased onto) #9650. ## What's changed and what's your intention? Add `promql::source::materialize_raw_labels`: a stream decorator that converts raw series chunks with primary-key-encoded labels into chunks with complete explicit labels. Uses `PrimaryKeyLabelDecoder` (from #9650) to decode dense/sparse primary keys against the scan-bound region metadata snapshot. Rejects schemas with eval grids or empty fields, and requires complete (not projected) label decoding. Output always advertises `RawLabelSource::Explicit`. **5 files, +1216/-0 lines (169 production + 1040 tests + 7 config).** The merge module is intentionally excluded. ### Verification - Compiled and tested with #9650 cherry-picked: `cargo check --all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D warnings` ✅ - Cannot compile standalone until #9650 merges. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 Dennis Zhuang
|
bd9f7dc40b |
docs(promql): say what draining the histograms after an early stop is for
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
328218776d |
fix(promql): follow up the post-merge review of native histogram operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
53ef80a516 |
feat(query): capture native raw-series routes and extend MergeScan for typed frames
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
3ef71b5702 |
feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
7c6595d9f7 |
feat(promql): materialize complete raw source labels
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f8054ab297 |
feat(promql): select instant samples from shared typed fields (#9649)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Builds on the shared series types already present on `narthex` (#9598, #9599). This PR targets `narthex`, not `main`, and has no dependency on another open PR. ## What's changed and what's your intention? Extract instant sample selection into `select_instants`, operating on the existing `RawTimestamps` and `SeriesFields`. It returns source row indices and evaluation timestamps without converting the payload into a new representation. Use it in the existing `InstantManipulate` path when timestamps are non-null, strictly increasing milliseconds and the configured fields are Float64. Other inputs retain the existing path, including duplicate timestamps and Histogram columns. Selection preserves lookback boundaries, offsets, stale markers, ordinary NaNs, null samples, and the existing output-point limit. This is one extracted slice of previously local work: **5 files, +732/-3 lines, including tests**. It does not add native planning, scanner/Flight integration, configuration, storage formats, or any new Histogram type. ### Verification - Passed: scoped Rust formatting, `hawkeye format`, `hawkeye check`, `git diff --check`, and normal commit hooks. - Running on this exact extracted branch; results not yet claimed: `cargo check --locked -p promql --all-targets` and `cargo nextest run --locked -p promql --retries 0`. - Independent source review is in progress. Kept as a draft until verification and review are complete. - Full-workspace tests, full-workspace Clippy, and SQLness have not been run for this extracted branch. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
88a1e684a0 |
feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors (#9656)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Recognize `SeriesScanExec` in query metrics and preserve typed errors through the record batch adapter: - `PlanMetrics`: recognizes `SeriesScanExec` nodes so region-scoped stats attribution works. - `RecordBatchStreamAdapter`: preserves the original typed error via `BoxedError` instead of wrapping into a generic execution error. This lets downstream recover the specific status code (e.g., `InvalidArguments` instead of `EngineExecuteQuery`). **2 files, +203/-9 production + 1 sqlness expected result.** ### Error code change The typed error preservation changes the error code for the `semantic_graph` upper-bound-only query from `3001(EngineExecuteQuery)` to `1004(InvalidArguments)`. The error message text is unchanged. The new code is correct: the original error was an argument validation error, not a generic execution failure. The sqlness expected result has been updated. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-recordbatch` → **65/65 passed** ✅ - sqlness expected result updated for the typed error code change ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
5d93b8177c |
feat(common-query): adapt ordered record batches to raw series (#9652)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. Builds on shared series types from #9598/#9599. ## What's changed and what's your intention? Add `record_batches_to_raw_series` in `common-query`: converts an ordinary DataFusion `RecordBatch` stream into contiguous raw series chunks. The adapter validates column types at construction (time must be Timestamp(Millisecond, None), tags must be string/dictionary, fields must be Float64 or native histogram), encodes series identity via Arrow RowConverter, and produces `RawSeriesChunk`s with explicit labels and contiguous ordering. Float64 fields share source buffers (zero-copy); histograms reuse the existing `histogram_samples_from_arrow` bridge. Chunking merges equal-key runs within and across batches; a key change or end-of-input sets `is_last`. Timestamps must be strictly increasing within a series. Empty batches are skipped with a bounded poll loop. **3 files, +1666/-0 lines, including 20 tests.** No production callers yet — consumed by follow-up scan wiring PRs. ### Verification - `cargo check --locked -p common-query --all-targets` ✅ - `cargo nextest run --locked -p common-query --retries 0` ✅ - `cargo clippy --locked -p common-query --all-targets -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
fb9bda2744 |
feat(store-api): add raw-series scanner preflight and ordering contract (#9651)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add two default methods to `RegionScanner` for raw-series scan preflight: - `raw_series_schema()` — non-consuming check whether this scanner can provide raw series for a given label source. Returns `None` (unsupported) by default; overrides promise `scan_series_partition` can execute. - `raw_series_ordering()` — returns the ordering guarantee without consuming scanner state. Defaults to `Contiguous` (conservative for legacy sources). Both are additive-only trait defaults with no behavior change for existing implementors. **32 insertions, 0 deletions, 1 file.** ### Verification - `cargo check --locked -p store-api --all-targets` ✅ - `cargo nextest run --locked -p store-api --retries 0` ✅ - `cargo clippy --locked -p store-api --all-targets -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
e25fa2bd8d |
Merge remote-tracking branch 'upstream/narthex' into feat/narthex-instant-eval-pr
# Conflicts: # src/promql/src/lib.rs |
||
|
discord9
|
869a32fdcb |
feat(query): add experimental_native_promql config option (#9662)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Add the `experimental_native_promql` configuration option (default: `false`): - `QueryOptions.experimental_native_promql`: new boolean field, default `false`. - Four example TOML configs updated with the commented-out option. - `config/config.md` regenerated with the new option. The option is stored in process configuration only. Session-level override and the actual gate in query planning are in follow-up PRs. **6 files, +24/-0 lines.** ### Verification - Compiles standalone: no unused fields or methods. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [x] This PR requires documentation updates. (config.md regenerated) - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
b27ca7c7bd |
feat(common-query): declare raw series frame ordering in schema metadata (#9655)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add ordering metadata to raw series frame schemas: - `raw_series_frame_schema_with_ordering()`: builds the canonical v1 frame schema and stamps the producer's ordering (`contiguous` or `read_key_then_time`) into Arrow schema metadata. Rejects `Interleaved` (no v1 wire representation). - `raw_series_frame_ordering()`: reads the declaration back, defaulting absent metadata to `Contiguous` (legacy frames), rejecting unknown values. - `pack_raw_series` now declares ordering on every yielded batch via `RecordBatch::with_schema`. Includes a wire compatibility fixture (`tests/compatibility/wire/raw_series_ordering/`) with 4 cases. **4 files, +277/-7 lines.** ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
2d9b2a970f |
feat(mito-codec): decode scan-bound primary key labels (#9650)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Builds on `narthex` base types (#9598, #9599, #9602, #9603). No dependency on other open PRs. ## What's changed and what's your intention? Add `PrimaryKeyLabelDecoder` to `mito-codec`: given a scan-bound `RegionMetadata` snapshot and a primary key byte sequence, decode the complete metric name and label set (or a projected subset). Supports both dense and sparse primary key encodings. Sparse keys get a new `visit_label_ids` method that validates the full tail (rejecting duplicates, reserved IDs, and truncated framing) before reading values. Dense keys check that all fields are present and decode each value with exact type fidelity. This is one extracted slice of previously local work: **4 files, +1046/-13 lines, including tests**. It has no production call site yet — it will be consumed by the raw series scan wiring in follow-up PRs. ### Verification - `cargo check --locked -p mito-codec --all-targets` ✅ - `cargo nextest run --locked -p mito-codec --retries 0` → **62/62 passed** ✅ - `cargo clippy --locked -p mito-codec --all-targets -- -D warnings` ✅ - `hawkeye format`, `hawkeye check`, `git diff --check` ✅ - Full-workspace tests and SQLness not run for this extracted branch. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
0ce358dde5 |
feat(promql): match vectors holding native histograms (#9696)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) MATCH-HIST in internal-docs#27. Builds on #9679 (`vector_match`) and #9686 (`FloatHistogram`). ## What's changed and what's your intention? ### Problem `binary_vectors` and `set_vectors` rejected native histogram samples with `NotImplemented`. ### Change - `vector_match::elem_binop` ports Prometheus `vectorElemBinop` for histograms: - histogram `+`/`-` histogram, compacted, with `-` marking the result as a gauge; - histogram `==`/`!=` histogram, layout-sensitive like Go's `Equals`, keeping the left histogram; - histogram `*` float, float `*` histogram, histogram `/` float, compacted. - Every other pairing that involves a histogram, and a `+`/`-` between exponential and custom buckets, is skipped: no result, `bool` included, and the pair does not count as a match. This follows Prometheus, which reports these as annotations before recording the match. - The one-to-one "multiple matches" check now counts the matches actually made per group and point, so skipped pairs do not count. - Set operators carry histogram values through; results with histograms are built with `GridSeriesBuilder::push_samples`. - Operand presence no longer reads values, and the helpers that could only fail on histograms are infallible now. Review focus: - `elem_binop` against `vectorElemBinop`: which pairings keep, filter or skip. - `binary_field`: the one-to-one match counting and the ordering of the skip against it. ### Non-goals - The experimental trim operators (`</`, `>/`). - Annotations Prometheus attaches to skipped pairs or counter-reset collisions. ### Constraints None. Public signatures are unchanged. ### Verification - `histogram_operators_match_prometheus`: 280 cases (20 per operator: `+ - * / % ^ atan2 == != > <= and or unless`) from the Prometheus v3.15.0 engine over vectors mixing floats and native histograms drawn from small per-case pools, with `on`/`ignoring`, `group_left`/`group_right` and `bool` (`vector_match/histogram_cases.txt`). Results, error classes, floats within 1e-10 and histograms bucket by bucket match. - Mutation check: all 10 faults fail a test (keeping incompatible pairs as filtered matches, not skipping mixed bucket kinds, no gauge hint, no compaction, `/` as `*`, `==` keeping the right side, `!=` as `==`, ignoring `group_right`'s swap, set operators dropping histograms, `float * histogram` unsupported). - `cargo nextest run -p promql`: 359 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
8b85a3c58f |
feat(promql): match vectors holding native histograms
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
cd3292f59e |
feat(promql): evaluate range functions over native histogram samples (#9695)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) EVAL-HIST in internal-docs#27. Builds on #9673/#9674/#9677 (`RangeEvaluator`) and #9686 (`FloatHistogram`). ## What's changed and what's your intention? ### Problem `RangeEvaluator` rejected native histogram samples with `NotImplemented`, so no range function could read a series holding native histograms, alone or mixed with floats. ### Change - `RangeEvaluator` keeps the histogram samples of each field apart from its floats, drops a histogram whose sum is the stale marker, and evaluates windows holding histograms in `range_eval::histogram` following the histogram branches of the Prometheus functions: - `min/max/stddev/stdvar/quantile/mad_over_time`, `deriv`, `predict_linear`, `double_exponential_smoothing`, `ts_of_min/max_over_time` read only the floats, as before; a window without floats has no result. - `count_over_time`, `present_over_time`, `ts_of_first/last_over_time` count both kinds; `first/last_over_time` return the earliest/latest sample, histogram or float. - `resets` and `changes` walk both kinds in time order: a change of kind is one edge, two histograms compare with `DetectReset` / `Equals`. - `sum/avg_over_time` and `rate/increase/delta` give nothing for a mixed window; over histograms alone they port the Kahan sum/mean and `histogramRate` with `extrapolatedRate`'s extrapolation. - `irate/idelta` take the last two samples of either kind; a float and a histogram give nothing. - An evaluator built from a bare kernel (`with_kernel`) still rejects histograms. - `GridSeriesBuilder::push_samples` writes rows holding histograms; fields become mixed samples when they hold any. - `histogram_eval::sum_or_avg` is the histogram sum/mean shared by `aggregate_histograms` and the range functions; the aggregation still compacts its result, the range functions do not, as in Prometheus. - `range_eval::function::extrapolation_factor` is the extrapolation step shared by the float and histogram rates. Review focus: - `range_eval/histogram.rs`: each arm against the Prometheus function it names; `histogram_rate` against `histogramRate` (nulling the first sample on a reset between the first two, schema reduction, adding back at resets). - `RangeEvaluator::evaluate`: retention and the skip to the earliest sample now cover histograms too. - `GridSeriesBuilder::push_samples`: histogram indices stay aligned with the float rows across gaps. ### Non-goals - Anchored and smoothed range selectors (NEW-EVAL-MOD); start-timestamp based resets, which Prometheus only uses behind a feature flag. - Annotations (warnings and infos) Prometheus attaches to mixed or incompatible windows. ### Constraints None. `RangeEvaluator`'s public interface is unchanged. ### Verification - `histogram_range_functions_match_prometheus`: 520 random cases (20 per function, 26 functions) plus 6 fixed counter histories, from the Prometheus v3.15.0 engine over one or two series mixing floats and native histograms, with ranges of 15s to 75s and offsets (`range_eval/histogram_cases.txt`). Floats compare within 1e-10 (1e-6 for `deriv`, `predict_linear`, `stddev`, `stdvar`), histograms bucket by bucket. - Mutation check: 16 of 17 faults fail a test (comparison order of first/last, counting only floats, mixed-edge counting, histogram resets as changes, mixed sums and rates, the counter zero point, adding back at resets, nulling the first sample, the gauge hint, irate's reset check, float-only functions answering for histograms, histogram retention, builder index alignment). The remaining one, skipping the schema reduction before the subtraction, only changes rounding: a later higher-schema sample is a reset whose addition reduces the schema anyway. - `histogram_samples_need_a_function_and_drop_stale_markers` pins the stale histogram drop and the bare-kernel rejection. - `cargo nextest run -p promql`: 358 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
17f30f594a |
feat(promql): evaluate range functions over native histogram samples
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
a78d4aa313 |
feat(promql): cover absent_over_time and vector(), sort quantile_over_time like Go (#9694)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Completes SEL-ABSENT (`absent_over_time`) and SEL-SCALAR (`vector()`) from internal-docs#27, and the `quantile_over_time` limitation left by #9674. ## What's changed and what's your intention? ### Problem - `quantile_over_time` sorted windows with a stable sort. Go's `sort.Sort` is unstable for more than 12 elements, so in a long window holding both +0 and -0 the zero a rank lands on, and the sign of the result, could differ from Prometheus. - `absent_over_time` and `vector()` had no native path. ### Change - `quantile_over_time` sorts with `go_sort::sort_by` and `vectorByValueHeap.Less` (NaN first). - `absent_over_time(v[r])` is `select::absent` over the series of `present_over_time(v[r])`, with labels from `select::absent_labels` on the matchers of `v`; Prometheus derives both the same way. The rustdoc of `absent` states this, and a golden test pins it. No new kernel. - `vector(s)` needs no kernel: a scalar is already a label-less grid series with a value at every point. The `select` module docs state this. ### Non-goals Wiring `absent_over_time` and `vector()` into the planner. ### Constraints None. ### Verification - `quantile_picks_the_signed_zero_go_picks`: a 20-sample window of mixed zeros, a NaN and numbers; the sign of `quantile_over_time(k/19)` matches Go's `sort.Sort` on amd64 for k = 1..19, and fails with the previous stable sort. - `absent_over_time_is_absent_over_present_over_time`: 120 cases from the Prometheus v3.15.0 engine (`range_eval/absent_over_time_cases.txt`) with ranges of 5s to 45s and equality/inequality matchers. - `cargo nextest run -p promql`: 357 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
41cfd237f6 |
test(promql): pin absent_over_time as absent over present_over_time
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
1147fafa5d |
fix(promql): sort quantile_over_time windows with Go's sort
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
7c8c74ac9b |
fix(promql): keep no samples past a skipped anchored window and trim Narthex operators (#9690)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Stacked on #9689; review the commits after `47f89b34b7`. Follow-up to the Narthex operator PRs merged into `narthex` (#9673–#9687). ## What's changed and what's your intention? ### Problem An `@`-anchored range selector whose one window ends before the series starts skips the cursor past the point it reads. `evaluate` cleared retained samples only when `cursor == points_read`, so every later chunk of that series stayed in memory until `is_last`, e.g. `x[5m] @ start()` over a series that starts after `start()`. Results were unaffected. The operator PRs also left duplicated helpers, a dead branch, a redundant reset, a duplicate test and comments that went stale during review. ### Change - `RangeEvaluator::evaluate` drops retained samples once `cursor >= points_read`. - `METADATA_LABELS` and `external` are defined once in `grid_series`; `range_eval`, `vector_match`, `map_time`, `select` and `histogram_eval` use them. - `kahan_inc` (Prometheus `kahansum.Inc`) is defined once in `functions`, on top of `compensated_sum_inc`, replacing the copies in `range_eval::function` and `histogram_eval::native`. - Removed: the unreachable `k == 0` return in `topk`, the field reset in `RangeEvaluator::finish_series` (`evaluate(None)` already leaves the fields empty), and `overlapping_window_spans_chunks`, whose only fault class is also caught by the random-chunk reference and the golden tests. - Fixed comments that no longer match the code: `RangeEvaluator::try_new` input, where the smoothing factors are checked, `binary_vectors` label dropping, `Results`, `rank`. ### Non-goals `quantile_over_time` still sorts with a stable sort; switching it to `go_sort` changes behavior and belongs in its own PR. ### Constraints None. No public interface changes. ### Verification - `anchored_window_before_the_series_retains_nothing` fails before the `>=` change and passes after it. - `cargo nextest run -p promql`: 356 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
d4c4da7b6e |
chore(promql): import parent items through crate paths (#9689)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Follow-up to #9673, #9686 and #9685, merged into `narthex`. ## What's changed and what's your intention? ### Problem `make fmt-check` fails on `narthex` at `4969bca60e`: `scripts/check-super-imports.py` rejects five top-level `use super::` imports in `histogram_eval/{native,float_histogram,aggregate}.rs` and `range_eval/function.rs`. ### Change The five imports use `crate::` paths. No behavior change. ### Non-goals None. ### Constraints None. ### Verification - `make fmt-check`: clean. - `cargo nextest run -p promql`: 356 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`: clean. - On `4969bca60e`, the rest of the full suite already passes: `make clippy`, `make test` (9950 passed; `json2::test_json2_single_mem_range_flush` failed once on a MySQL port bind and passed on retry) and `cargo sqlness bare`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [ ] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
438c44c109 |
chore: merge fix/narthex-super-imports into chore/narthex-ablation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
4761d39690 |
refactor(promql): drop a redundant field reset and a duplicate range window test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
72b06dc730 |
fix(promql): drop retained samples when an anchored window is skipped
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
47f89b34b7 |
chore(promql): import parent items through crate paths
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
a19e8cbd46 |
chore(promql): fix stale docs and drop a dead branch in Narthex operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
b37f47022b |
refactor(promql): share metadata labels, error mapping and Kahan increments across Narthex operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
4969bca60e |
feat(promql): add native histogram arithmetic and sum/avg aggregation (#9686)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9685 for the native histogram bucket bounds and the golden fixture helpers; review only `feat(promql): add native histogram arithmetic and sum/avg aggregation`. ## What's changed and what's your intention? ### Problem Native histogram range functions, aggregations and binary operators all need histogram arithmetic. The ordinary `common_query::native_histogram::NativeHistogram` adds without Kahan compensation and compacts with its own algorithm, so its results differ from Prometheus in value bits and bucket layout. The native path has no `sum`/`avg` over native histograms either. ### Change - `histogram_eval::float_histogram::FloatHistogram` ports Prometheus `FloatHistogram` (`model/histogram/float_histogram.go`): `Add`, `KahanAdd`, `Sub`, `Mul`, `Div`, `Compact(0)`, `CopyToSchema`, `DetectReset`, `Equals` and `HasOverflow`. The port covers: - zero bucket reconciliation (`zeroCountForLargerThreshold`, `trimBucketsInZeroBucket`), including the compensation Go keeps across abandoned passes; - resolution reduction, plain and Kahan; - the intersection of mismatched custom bounds and the matching reset detection; - counter reset hint merging. Each side is kept as `(bucket index, count)` sorted by index. The bucket set and the order of additions match Go's span splicing. A histogram read from a view keeps its spans until its buckets change, because `Equals` compares span layouts; afterwards consecutive indices form one span, the layout `Compact` produces. - `histogram_eval::aggregate_histograms` follows the histogram paths of Prometheus `aggregation` for `sum` and `avg` within one group. Histograms are added in group order with `KahanAdd`. `avg` switches to an incremental mean once the plain sum overflows. A point yields nothing when the group also has a float there or mixes exponential and custom buckets; any other addition error fails the aggregation. Grouping stays with AGG-BASIC. - `native::bound` takes exponential bounds from Go's `exponentialBounds` table with `Ldexp`, as `getBoundExponential` does. Widened zero thresholds are set to these bounds and compared exactly in `DetectReset` and when adding buckets, and some table entries are one ulp off `2^(idx / 2^schema)`. - `sub`, `copy_to_schema`, `detect_reset` and `equals` have no caller on this branch yet. They are for the native histogram range functions and binary operators, which build on other branches, so the module allows dead code outside tests. Review focus: - `add_buckets`: `other` buckets whose upper bound is within the zero threshold are skipped only while they lead; subtraction inserts negated counts, including `-0`. - `zero_count_for_larger_threshold` and `reconcile_zero_buckets`: the widened threshold, the Kahan compensation of the zero count, and the compaction after trimming, which drops only buckets whose count and compensation are both zero. - `kahan_add`: which side is reduced to the lower schema, and the alignment of the compensation buckets with the receiving ones. - `detect_reset`: both sides start from the current zero threshold; merged buckets of a higher-resolution predecessor sum from zero. - `aggregate_point`: the overflow switch to the incremental mean reuses the compensation from before the failed addition. - The aggregation writes its histograms from the first point with a result instead of holding one slot per point of the whole input span. ### Non-goals - Native histogram range functions (EVAL-HIST) and binary operators (MATCH-HIST). - Grouping and float aggregation (AGG-BASIC). - `Equals` after an uncompacted `Add`/`Sub` compares the compacted layout. PromQL compacts every such result before using it. ### Constraints - Sums of floats and histograms depend on the order of the series. Prometheus adds them in the order the head index returns them (creation order). The kernel adds them in the order of `group`, which the caller decides. - Prometheus panics in `addBuckets` on histograms with zero-length spans; the port does not. The aggregation fixtures compact their inputs for that reason. ### Verification - `cargo nextest run -p promql`: 319 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `histogram_arithmetic_matches_prometheus`: 723 operations from Go's `FloatHistogram` methods on random histograms (`histogram_eval/arithmetic_cases.txt`). Inputs include: - schemas -4 to 8 and custom buckets with matching and mismatched bounds; - zero thresholds from 0 to 0.5, mixed exponential and custom operands, and all reset hints; - reset candidates with missing buckets, lower zero counts, lower resolution, wider zero buckets and bounds that underflow to zero; - three fixed cases random inputs do not reach. Results compare every field, the bucket indices and count bits of both the sum and the Kahan compensation, and the collision and reconciliation flags. - `native_histograms_match_prometheus` now also covers 240 `sum`/`avg` cases with and without `by (job)`, from the Prometheus engine with inputs in creation order, plus a fixed `avg` case whose overflow switch carries a nonzero compensation. Aggregated histograms compare bit for bit. - Mutation check, each of these fails a test: - Arithmetic, 21 faults: - no zero bucket reconciliation in `Add`, plain addition inside `KahanAdd`, a target index without the `- 1`; - no zero-threshold skip, no hint collision; - `>` instead of `>=` when mapping custom bounds; - resetting the zero count compensation per pass, no compaction after trimming, a compaction that ignores the compensation; - reset detection that ignores the zero count, missing buckets or schema merging, or skips at a zero threshold; - `Equals` by value or without the layout; - division by zero keeping buckets, `CopyToSchema` keeping the hint, a negative factor not setting the gauge hint; - restarting the custom-bound roll-up, dropping the custom-bucket compensation, plain resolution reduction in `KahanAdd`. - Aggregation, 8 faults: plain `Add` for `sum`, dropping the final compensation, never switching to the incremental mean, discarding the compensation at the switch, output despite floats or incompatible schemas, no final compaction, adding the compensation before dividing. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
db0a531d7d | chore: merge narthex into feat/narthex-histogram-arith | ||
|
dennis zhuang
|
0d2f1ff6cd |
feat(promql): evaluate classic and native histogram functions over grid series (#9685)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9684 for `merge_same_labelset` and the golden fixture helpers; review only the two commits `feat(promql): evaluate classic histogram_quantile and histogram_fraction over grid series` and `feat(promql): evaluate native histogram functions over grid series`. ## What's changed and what's your intention? ### Problem The native path has no kernels for the histogram functions: `histogram_quantile` and `histogram_fraction` over classic (`le`-bucketed) and native histograms, and `histogram_count`, `histogram_sum`, `histogram_avg`, `histogram_stddev` and `histogram_stdvar` over native histograms. ### Change `histogram_eval::evaluate_histograms` takes the evaluated input series and a `HistogramFunction` (`Quantile(q)` or `Fraction { lower, upper }`, constants as in the ordinary planner) and follows Prometheus `funcHistogramQuantile`, `funcHistogramFraction` and `resetHistograms`: - Float series with the same labels other than `le` form a classic histogram; a series whose `le` does not parse is ignored. Parsing follows Go's `strconv.ParseFloat`: finite literals out of the f64 range (`1e400`) and a signed NaN are rejected, and `_` between digits is accepted. - Native histogram samples are evaluated one by one. At a point where a native histogram and classic buckets of the same labels (`le` aside) are both present, neither is evaluated. - Each field is evaluated on its own. Results drop `le`, `__name__`, `__type__` and `__unit__`; series that then share a label set go through `merge_same_labelset`. Classic buckets (`histogram_eval::classic`) port `BucketQuantile` and `BucketFraction`. At each point the present buckets are sorted with `go_sort::sort_by` and Go's `<` comparator, so +0/-0 and NaN bounds keep Go's order, and buckets with equal bounds are coalesced. For quantiles, counts within a relative 1e-12 of the previous count or below it are raised to it. The quantile bucket is found with Go's `sort.Search` bisection, so NaN counts select the same bucket. Native histograms (`histogram_eval::native`) port `HistogramQuantile`, `HistogramFraction` and `histogramVariance`. The port includes: - the bucket order and zero-bucket clipping of `AllBucketIterator`/`AllReverseBucketIterator`; - bucket bounds `2^(idx / 2^schema)` from the standard library, with `f64::MAX` for the last bucket of finite values as in Go's `getBound`; This covers the reverse iteration for `q >= 0.5`, the count clamp, custom-bucket edge buckets, and the trailing walk that Prometheus runs for NaN observations, which leaves the last bucket in place for interpolation. `histogram_eval::map_native_histogram` applies `NativeHistogramFunction::{Count, Sum, Avg, StdDev, StdVar}` to the histogram samples of one series, dropping float samples and `__name__`; colliding label sets are left to `merge_same_labelset`, as for other per-series kernels. Review focus: - `classic::bucket_quantile`: the `q` checks come before the `+Inf` check, the `b == 0 && upper_bound <= 0` branch, and the interpolation `start + (end - start) * (rank / count)`. - `classic::ensure_monotonic`: `prev` is not advanced when a count is raised. - `native::all_buckets` and `native::bound`: iteration order, clipping to the zero threshold, and the last finite bucket (`f64::MAX`). - `native::quantile`: the last visited bucket (empty or not) is the one interpolated, including after the NaN-observation walk. - `evaluate_histograms`: the classic/native conflict is decided per point and field. ### Non-goals - Native histogram `sum`/`avg` aggregation, rate functions and arithmetic. - Non-constant `q`, `lower` and `upper`. - Planner wiring. ### Constraints - The ordinary `HistogramFold::evaluate_row` differs from Prometheus for classic histograms: it errors when the last bucket is not `+Inf`, has no small-delta tolerance, replaces non-finite counts with the previous count, and interpolates as `lower + (upper - lower) / count * rank`. The ordinary native quantile (`common_query::native_histogram::quantile_histogram_view`) has no reverse iteration for `q >= 0.5` and derives bounds with `powf` instead of Go's table. This PR follows Prometheus 3.15. - `le` values in Go-only float syntax (hex floats such as `0x1p-2`) do not parse with Rust's `f64::from_str` and are ignored. ### Verification - `cargo nextest run -p promql`: 318 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `classic_histograms_match_prometheus`: 300 fixtures produced by the Prometheus v3.15.0 engine built for amd64 (`histogram_eval/classic_cases.txt`). - Bucket layouts: non-positive bounds, a bound written twice (`1`/`1.0`), no `+Inf`, only `+Inf`, unparsable, out-of-range or missing `le`, and the `inf` spelling. - Counts: monotonic, non-monotonic, around the 1e-12 tolerance, NaN, `+Inf`, negative, zero and subnormal. - Inputs are passed in reverse order. 32 cases expect the duplicate-labelset error. - `native_histograms_match_prometheus`: 420 fixtures (`histogram_eval/native_cases.txt`, 60 per function). - Histograms: schemas -4, -1, 0, 1, 3, 8 and custom buckets, with gaps, zero-length spans, empty buckets, zero thresholds from 0 to 0.5, NaN sums with uncounted observations, and counts below the bucket total. - Inputs mix floats and histograms, and include classic buckets of the same labels. - Inputs are read back from the engine, so they carry its span layout. - Values in both fixtures compare within promql-compat's relative tolerance (1e-10, 1e-6 for `histogram_stddev`/`histogram_stdvar`), NaN as any NaN. - Mutation check, classic: each of these fails a test: - no coalescing (quantile or fraction), no forced monotonicity, no small-delta tolerance, an absolute tolerance at zero; - a linear search, no `+Inf` check, `q` checks after it, no non-positive first-bucket branch, the ordinary interpolation order; - accepting `1e400`, keeping `__name__`, a zero lower bound for a non-positive first fraction bucket, interpolating inside a `-Inf`-wide bucket, no merge, sorting with NaN bounds last. - Mutation check, native: each of these fails a test: - always iterating forward, no NaN-observation walk, no zero-threshold clipping, no clamp of the count; - shifted bucket bounds, no `f64::MAX` for the last finite bucket; - a plain variance sum, a geometric midpoint for custom buckets; - the histogram count as the fraction total under NaN observations, linear interpolation in exponential buckets; - no custom `-Inf` branch, no zero-bucket side adjustment; - evaluating conflicting classic and native histograms, keeping `__name__`, no zero bucket in reverse order. - `histogram_functions_drop_metadata_labels`: `histogram_quantile` drops all metadata labels, `histogram_count` only `__name__`. - `classic_bounds_keep_the_first_signed_zero`: bounds +0 and -0 compare equal and keep input order, so `1 / histogram_quantile(0, ...)` over `le` "+Inf", "0", "-0" is +Inf. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
041a48d456 | chore: merge narthex into feat/narthex-classic-histogram | ||
|
dennis zhuang
|
12e07cf727 |
feat(promql): evaluate time, date and label rewriting functions over grid series (#9684)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test reference; review only `feat(promql): evaluate time, date and label rewriting functions over grid series`. ## What's changed and what's your intention? ### Problem The native path has no kernels for `time()`, the date functions (`minute`, `hour`, `day_of_month`, `day_of_week`, `day_of_year`, `days_in_month`, `month`, `year`) or `label_replace`/`label_join`. ### Change - `map_time::time` and `map_time::date_of_eval_time` produce one unlabelled series over the schema's grid from the evaluation timestamps (`ms / 1000` for `time()`, whole seconds truncated towards zero for the date functions, as Prometheus `funcTime`/`dateWrapper`). - `map_time::map_date` reads every float value as Unix seconds and drops `__name__`, `__type__` and `__unit__`. The value goes through Go's `int64(f)` as on amd64 (NaN and out-of-range values become `i64::MIN`) and a port of Go's `time` absolute-date arithmetic, including its wrapping for instants Go cannot represent, which decides the result for those inputs. All-float series are mapped chunk by chunk without copying the time axis; histogram samples are dropped and the series is trimmed again. - `map_label::label_replace` anchors the regex as `^(?s:...)$`, expands `$1`/`${name}`/`$$` with Go's `regexp.Expand` rules, and leaves unmatched series unchanged. `map_label::label_join` joins source values, an absent label joining as `""`. An empty result removes the label in both. Empty source or destination label names and invalid regexes are `Plan` errors with Prometheus' messages. - `grid_series::merge_same_labelset` follows Prometheus `mergeSeriesWithSameLabelset`: series that share a label set (empty values counting as absent) become one series, merged field by field; a field with values from two of them at one point fails with "vector cannot contain metrics with the same labelset". The label functions call it; `map_date` leaves it to the caller, as other per-series kernels that drop `__name__` do. - `promql` now depends on the workspace `regex` crate, which is already in the build graph. Review focus: - `map_time::DateFunction::apply` and `civil`: the port of Go's `absSeconds`/`absDays` arithmetic; `days_in_month` uses the proleptic Gregorian leap rule on the computed year. - `map_time::go_int64`: NaN and values outside `[-2^63, 2^63)` map to `i64::MIN`. - `merge_same_labelset`: the fast path returns the input untouched when no label sets collide; colliding groups are rebuilt row by row and fail on any shared point. - `label_replace` expands its template with Go's `regexp.Expand` rules: a reference with a leading zero (`$01`) or of more than nine digits names a group, unlike `regex::Captures::expand`. ### Non-goals - `timestamp()` (EVAL-TS), which needs sample timestamps from instant selection. - Merging histogram series that collide after a label function (`NotImplemented`). - Planner wiring. ### Constraints - Every per-series kernel that drops `__name__` (range functions, date functions, math) can leave two series with one label set. Prometheus merges or rejects them per step; on the native path that is `merge_same_labelset`, which the caller must apply after such kernels. - Regexes compile with the Rust `regex` crate, as the ordinary planner does. Perl classes (`\d`, `\w`, `\s`, `\b`) are Unicode-aware there and ASCII-only in Go's RE2, so non-ASCII label values can match differently. The dialects also differ in what compiles: `\Q...\E` is rejected, `(?x)` and repeat counts above 1000 are accepted, and `\<`, `\>` are word boundaries instead of literal characters. The rustdoc of `label_replace` lists the same differences. - `go_int64` follows amd64. On arm64 Go saturates (`NaN` becomes 0), so Prometheus' own results for non-finite inputs differ by architecture. ### Verification - `cargo nextest run -p promql`: 315 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `time_functions_match_prometheus`: 200 fixtures produced by the Prometheus v3.15.0 engine built for amd64 (`map_time/prometheus_cases.txt`): `time()` and argument-less date functions over grids from 1685 to 2254, and date functions over values including NaN, ±Inf, ±1e300, ±9.2e18, leap-day edges of 1600/1900/2000/2100/2400 and years 0 and 10000. Values compare bit for bit after `merge_same_labelset`; 6 cases expect the duplicate-labelset error. - `label_functions_match_prometheus`: 200 fixtures (`map_label/prometheus_cases.txt`) with random destinations including `__name__`, absent sources, templates with valid and invalid `$` references, anchoring-sensitive regexes and invalid ones; 25 expect errors, the rest compare label sets and value bits. - Mutation check: each of these fails a test: Go's NaN and out-of-range conversion replaced with a saturating cast or NaN-to-0; floor instead of truncating division of the evaluation time; whole-second `time()`; dropping the 400-year rule; leap day counted in January and February; hour from the Unix seconds instead of Go's absolute seconds; no leap February in `days_in_month`; leap year taken from the March-based year; keeping `__name__`; mapping only the first chunk; regex without the `(?s:...)` group or without anchors; setting an empty label instead of removing it; skipping absent `label_join` sources; skipping the merge; skipping the duplicate check; comparing empty label values as present. - `replacements_expand_like_go`: 16 templates with expectations taken from Go, including `$1²`, where Go's Unicode classes end the name before `²`. - `merge_combines_disjoint_fields_at_one_point`: different fields at one point merge, one field present twice fails. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
6485e9c810 | chore: merge narthex into feat/narthex-map-label-time | ||
|
dennis zhuang
|
7677e5159e |
feat(promql): evaluate topk, bottomk, scalar, absent and instant sorts (#9680)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test reference; review only `feat(promql): evaluate topk, bottomk, scalar, absent and instant sorts over evaluated series`. ## What's changed and what's your intention? ### Problem The native path has no operators that depend on the whole set of series at each point: `topk`/`bottomk`, `scalar`, `absent` and the instant-result sorts. ### Change `select` module, following Prometheus (`aggregationK`, `funcScalar`, `funcAbsent`, `createLabelsForAbsentFunction`, `funcSort*`): - `topk(k, top, grouping, schema, series)`: at every point each `by`/`without` group keeps the `k` largest (or smallest) values, NaN ranking last for both. `k` is truncated and capped at the number of input series; below 1 there is no result. Selected samples keep their series' labels and values. - `scalar(schema, series)`: the value of the only series present at each point, Go's NaN otherwise; the result is a label-less series with a value at every point. A scalar in this representation is already `vector(scalar)`. - `absent(schema, series, labels)`: 1 with `labels` at every point where no series is present, in every field. `absent_over_time(m[r])` is `absent` over the `present_over_time` kernel of #9674. - `absent_labels(matchers)`: the result labels from the selector's `LabelMatcher`s (`name`, `MatchOp`, `value`, `matches_empty`). - `sort_instant(order, schema, series)`: `sort`, `sort_desc` (NaN last), `sort_by_label` and `sort_by_label_desc` (Go `natsort` order, then the label sets) for single-point results. Multi-point results are returned unchanged, as Prometheus does not sort matrices. - `go_sort::sort_by`: a port of Go's pdqsort (`sort.Sort` and `slices.SortFunc`). The instant sorts use it with Prometheus' comparators, so ties, ±0, several NaNs and `natsort`'s inconsistent order come out in Go's order, and an inconsistent comparator cannot panic. `topk`, `scalar` and the value sorts take one field and reject others with `NotImplemented`; histogram samples are rejected the same way. Review focus: - `topk`: NaN ranking and the `k` conversion. - `absent_labels`: the matcher order (see Constraints). - `natural_less` and `chunks` against `facette/natsort.Compare`. - `topk`/`bottomk` return nothing for `k` below 1 and reject a NaN `k` and one of 9223372036854774784 (Prometheus' `maxInt64`) or more with Prometheus' errors. - `absent` marks each field absent on its own. ### Non-goals - Non-constant `k`, `limitk`, `count_values`, planner wiring. ### Constraints - Ties in `topk`/`bottomk` go to the series whose label set sorts first. Prometheus resolves them by the order its storage returns series in (posting order), which is not a property of the series. - `absent_labels` applies Prometheus' rule (the first equality matcher of a name sets it, any other matcher of the name removes it) to the matchers in the order Prometheus actually reads them: TSDB's `PostingsForMatchers` stably moves narrowing matchers before subtracting ones, in place, before `absent` builds its labels. Over an empty storage Prometheus keeps the query order instead. The two orders differ only when one name has both an equality and another matcher. ### Verification - `cargo nextest run -p promql`: 315 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `select_operators_match_prometheus`: 180 cases from the Prometheus v3.15.0 engine built for amd64 (60 each of `topk`/`bottomk` with `by`/`without` and several `k`, `scalar`, and `absent` with equality, inequality and repeated matchers), over series with holes, special values and NaN, sampled on the grid with a 1ms lookback. Result label sets and value bits match. The fixture is `select/prometheus_cases.txt`. - `matches_go_sort_on_comparators_that_are_not_total_orders`: 208 orders produced by Go's `sort.Sort` and `slices.SortFunc` (`go_sort_cases.txt`), with consistent, NaN-like, tie-heavy and inconsistent comparators, up to 257 elements. - `sort_instant` and `natural_less` checks for the cases the fixture cannot carry (instant order, Go `natsort` behavior including digit runs beyond `i64`). - Mutation check: NaN ranked first, `topk`/`bottomk` swapped, `k` rounded up, `__name__` kept by `without`, `scalar` with several series, repeated matchers kept, query-order matchers, and `absent` ignoring presence each fail a test. - `topk_validates_k_like_prometheus` and `absent_evaluates_fields_independently` fail without these changes. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
6b94b067fd |
chore: merge narthex into feat/narthex-select
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
819c0d0065 |
feat(promql): match evaluated vectors for binary and set operators (#9679)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test reference; review only the two `vector_match` commits. ## What's changed and what's your intention? ### Problem The native path has no vector–vector operations: arithmetic, comparisons, `group_left`/`group_right` and `and`/`or`/`unless` between two sets of evaluated series. ### Change `vector_match::binary_vectors` and `vector_match::set_vectors` take both operands as evaluated series on one grid and return trimmed grid series. Semantics follow Prometheus `VectorBinop`, `resultMetric`, `signatureFunc`, `VectorAnd`, `VectorOr` and `VectorUnless` at every grid point: - Match groups: `on(...)` keeps the listed labels (`__name__` when listed); `ignoring(...)` and the default keep every other label except `__name__`. Empty label values count as absent. - Cardinality is checked per point, never over a series' whole lifetime: - The "one" side (right, or left with `group_right`) may hold one series per match group at a point where both operands are non-empty. - One-to-one matching may match a group once per point, filtered comparisons included. - `group_x` results must have unique label sets per point. Series of one group that are present at disjoint points never conflict. - Result labels: arithmetic and `bool` comparisons drop `__name__`, `__type__` and `__unit__`; one-to-one keeps the `on` labels or drops the `ignoring` ones; `group_x` labels are copied from the "one" side, and removed when it has no value for them. - Values: comparisons without `bool` keep the left value where they hold; `bool` returns 1 or 0. `^` and `atan2` use the standard library. - Errors: the error of the earliest failing grid point is returned. At one point a one-side duplicate is reported before a one-to-one or `group_x` violation, as Prometheus checks the one side before matching. - Fields are paired by position; the output carries the shorter side's field count, named after the left side. Each field is matched on its own, as if the operands held only that field, and the per-field results are joined by label set. Set operators require equal field counts and return `NotImplemented` otherwise. - `or` requires the same field count on both sides and rejects others with `NotImplemented`. Histogram samples are rejected with `NotImplemented`. Series are processed whole over their own grid ranges: the "one" side is indexed by match group, and per-point checks run only over the points a series covers. Review focus: - `binary_vectors`: the one-side duplicate check is limited to points where both operands are non-empty, as Prometheus short-circuits empty steps before checking. - `binary_vectors`: the one-to-one check counts a match even when the comparison filters it out. - `Results::sort_rows`: duplicate rows of one result label set at a point are the `group_x` uniqueness error; the earliest point across all label sets is returned. - `binary_vectors` error selection: earliest point wins; ties go to the one-side duplicate. - Results of different match groups that share a label set once `__type__` is dropped fail as duplicate label sets after the operator, as Prometheus does; two results within one group fail the `group_x` uniqueness check. ### Non-goals - Scalar operands (MAP-ARITH), `fill` modifiers, histogram operands (MATCH-HIST), planner wiring. ### Constraints Results agree with Prometheus within promql-compat's relative tolerance (1e-10), not bit for bit. The ordinary planner deviates from Prometheus in ways this kernel does not reproduce: - An operand whose plan has no tag columns is joined on time only and broadcast to every series of the other side (`only_join_time_index`). Prometheus matches it by label signature, so `sum(m) / m` returns nothing in Prometheus. - Labels with an empty value and absent labels join as different values (`NullEquality::NullEqualsNull` over nullable tag columns). Prometheus treats them as equal. ### Verification - `cargo nextest run -p promql`: 317 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `binary_vectors_match_prometheus_reference`: 3,000 random cases against a per-point transcription of Prometheus `VectorBinop`/`resultMetric`/`signatureFunc` that keeps labels in a map with `__name__` as an ordinary label. The cases cover all 13 operators, `bool`, random `on`/`ignoring` subsets including `__name__`, all three cardinalities with random `group_x` labels, 1–2 fields per side, and series with holes and disjoint lifetimes. Over 1,000 cases succeed and over 100 hit a cardinality error; results compare label sets, absolute grid ranges and value bits, and errors compare their class. - `set_vectors_match_prometheus_reference`: 3,000 random cases for `and`/`or`/`unless`. - `vector_operations_match_prometheus`: 180 fixtures (120 binary, 60 set) produced by the Prometheus v3.15.0 engine (`vector_match/prometheus_cases.txt`); results compare label sets and grid points exactly and values within the tolerance, errors compare their class. - `pow_and_atan2_match_go`: 452 Go `math.Pow`/`math.Atan2` results (`vector_match/go_math_cases.txt`) agree with the standard library within the tolerance. - `reports_the_earliest_failing_point`: a `group_left` violation at the first point wins over a one-side duplicate at the second. - Mutation check: dropping `__name__` for comparisons, ignoring the `ignoring` list, checking one-side duplicates at points where an operand is empty, removing the one-to-one match check, keeping a `group_x` label the one side lacks, not filtering comparisons, keeping matched rows in `or`, swapping `and`/`unless`, keeping all labels under `on`, accepting duplicate result rows, and taking the wrong "many" side for `group_right` each fail a test. So do replacing the Go ports with `powf`/`atan2`, misplacing the fractional-exponent split in `pow`, returning the first one-side or one-to-one violation found instead of the earliest, always or never preferring the one-side duplicate on ties, returning the first duplicate label set instead of the earliest, and skipping the row sort. - `colliding_results_fail_by_where_they_collide`: a collision across match groups reports duplicate label sets, one within a group the grouping error; the random reference now includes `__type__`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
c52039ca47 |
feat(promql): evaluate experimental first, mad and ts_of range functions (#9687)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Targets `narthex`. Stacked on #9677 for `RangeEvaluator` with `@` and subquery input; review only `feat(promql): evaluate experimental first, mad and ts_of range functions`. ## What's changed and what's your intention? ### Problem The native range evaluator has none of Prometheus' experimental range functions `first_over_time`, `mad_over_time` and `ts_of_first/last/min/max_over_time`. ### Change New `RangeFunction` variants with kernels ported from Prometheus: - `FirstOverTime`: the first sample of the window (`funcFirstOverTime`). - `MadOverTime`: the median of the absolute deviations from the median, through the existing `quantile`; NaN when any sample is NaN (`funcMadOverTime`). - `TsOfFirstOverTime`, `TsOfLastOverTime`: the timestamp in seconds of the first and last sample. - `TsOfMinOverTime`, `TsOfMaxOverTime`: the timestamp of the last sample holding the minimum or maximum; a NaN so far is replaced by any later sample (`compareOverTime` with `<=`/`>=`). Window timestamps are shifted by the offset (or the `@` fold offset) onto the evaluation timeline. The `ts_of_*` kernels subtract it to report the sample's own timestamp, as Prometheus does. All six are step-invariant, so `@` replays one result. Review focus: - `ts_of_*` in `RangeFunction::kernel`: the subtracted value is the fold offset passed to `kernel`, which equals the selector offset without `@`. - `timestamp_of_extremum`: `>=`/`<=` keep the last extremum, and a NaN best is always replaced. - Results drop the metadata labels except for `first_over_time`, as Prometheus does. - `ts_of_last_over_time` reports the max of the last timestamp and 0, as Prometheus 3.15 does by taking the max with a histogram timestamp that defaults to 0. ### Non-goals - Gating these functions behind an experimental flag, which belongs with planner wiring. - Histogram samples (`first_over_time` and `ts_of_first/last_over_time` also accept them in Prometheus). ### Constraints None. ### Verification - `cargo nextest run -p promql`: 318 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --check`, `hawkeye check`: clean. - `range_functions_match_prometheus` now also reads `range_eval/experimental_cases.txt`. It holds 240 cases (40 per function) from the Prometheus v3.15.0 engine built for amd64, 125 of them with `@` and most with an offset. Values compare bit for bit, NaN as any NaN. - `ts_of_extremum_moves_past_a_leading_nan` covers a window starting with NaN. - Mutation check, each of these fails a test: - `ts_of_first/max` without subtracting the offset; - strict `>`/`<` in `ts_of_max/min`, keeping a NaN best; - no NaN check in `mad`, returning the median instead of the deviation median; - `first` returning the last sample, `ts_of_last` returning the first; - truncating the timestamp to whole seconds. - `ts_of_last_over_time_reports_zero_for_negative_timestamps` and the `first_over_time` case of `range_functions_drop_metadata_labels`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |