refactor(promql): let the legacy histogram path read the native kernel (#10420)

* refactor(promql): read the legacy histogram estimators from the native kernel

`prom_native_histogram_quantile` and `prom_native_histogram_fraction` estimated from
`common_query`'s `NativeHistogram` bucket walk: exponential bucket bounds computed as `2^x`, and a
forward walk from the lowest bucket only. The native kernel in `histogram_eval` walks the same
histogram the way Prometheus does -- the bound table in `bounds`, and a walk from the highest bucket
down once a finite-sum quantile reaches the median -- so one histogram had two answers that differ
in their last places.

Read the `NativeHistogram` through the borrowed `HistogramView` both representations share and
return the kernel's answer, mapping its outcome back onto the NaN-observation notes the legacy UDFs
record. Their signatures and the notes are unchanged; only the arithmetic behind them moves.

The `2^x` estimator in `common_query::native_histogram` stays for that crate's public API and is
annotated as such: no PromQL path reaches it any more.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): delegate the legacy histogram arithmetic to the native kernel

`prom_native_histogram_add`, `_sub`, `_mul_scalar`, `_scalar_mul`, `_div_scalar`, `_neg`, the
sum/avg accumulators, `delta`/`idelta`/`rate`/`increase`, and the weighted mean that merges partial
states all scaled and combined `NativeHistogram`s with that type's own reconciliation: its own
zero-bucket widening, schema reduction and custom-bound intersection, over `2^x` bounds.
`FloatHistogram::add`/`sub`/`mul`/`div` already implement Prometheus's rules (`addBuckets`,
`reconcileZeroBuckets`, `CopyToSchema`, `intersectCustomBucketBounds`) and the native kernel's
golden tests cover them, so this keeps one implementation of the semantics.

`AddOutcome` folds into the existing semantics rather than being dropped: `custom_bounds_reconciled`
is the intersection the kernel keeps in `custom_values`, which the callers already announce from the
operands, and `counter_reset_collision` is the contradiction the kernel answers with the unknown
hint the legacy arithmetic derived from two hints that disagree; both are asserted in debug builds.
The public `NativeHistogram` signatures are unchanged, the sums are compacted as before so
`promql_eq` and the serialized spans keep their answers, and unary minus goes through the kernel's
multiplication like the other scalings.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the legacy delegation's observable answers

Comparing the bridge against the kernel it calls only shows they agree today; it cannot show
what either answer means. These tests pin the answers themselves.

- quantile with a NaN sum over several buckets answers 32.0 with NaNSkew, because the NaN walk
  leaves the last bucket for the interpolation; below the rank it answers NaN with NaNResult.
- an exact tie answers 1.0 through the legacy walk and 2.0 through the kernel, on a fixture with
  an empty bucket between the two, and neither records a note.
- assert_same_payload now compares every field, including the start timestamp policy, custom
  values, the complete span segmentation and the bucket bits.
- round-tripping preserves reset hints, custom values, zero-length spans and gaps.
- add/sub reconcile the reset hint and the custom bounds, compact cancelled buckets, and keep
  promql_eq's expanded-index equality.
- scale/divide_by keep the field-by-field fallback for an unreadable layout -- mismatched counts,
  a backwards span, an overflowing index, a negative length -- and preserve its start timestamp;
  add/sub answer None for the same layouts. Zero, negative and NaN factors are pinned.
- rate/increase/delta/idelta/irate clear the start timestamp of a derived histogram, while a
  single-sample sum aggregate keeps its stored one.

A layout whose later span steps backwards is refused before the kernel sees it: the legacy
bucket_indices rejected it, while side_from_view would have accepted repeated indices.

Signed-off-by: discord9 <discord9@163.com>

* test(http): pin the legacy histogram route's rate, NaN quantile and derived windows

The delegation changes what three kinds of well-formed data answer, so assert it where a user
sees it. Every request runs without the native hint and asserts !executes_fused_series on the
observed plan, which proves the legacy route rather than assuming it from the missing hint.

- rate and increase over a raw sample with a start timestamp inside the window still extrapolate
  from the synthetic zero: count 8/3 and sum 10/3 for a rate, 8 and 10 for an increase, with no
  infos or warnings.
- histogram_quantile(0.5) over schema 0, buckets [8, 1], count 10 and a NaN sum answers 32.0 and
  reports the NaN-observation info, matching Prometheus's walk that leaves the last bucket for
  the interpolation. The previous estimator answered 2^-0.375.
- rate over a raw window answers one series while the same window over h*1 answers none: the
  derived sample carries no start timestamp, so it has no synthetic zero to extrapolate from.
  Both phases are covered, memtable and flushed SST.

The native-histogram test now installs the route interceptor the way its sibling tests do, so the
plan can be observed; all of its original assertions are unchanged.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
discord9
2026-11-16 16:26:52 +08:00
committed by GitHub
parent 50b6b5bf82
commit 5517c4b76f
8 changed files with 1801 additions and 32 deletions
+117
View File
@@ -0,0 +1,117 @@
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Follow-up to #10086 / #10178 / #10199, which made the native route read, fold and average stored native histograms.
- Tracking issue: #8887 (native histogram stabilization).
## What's changed and what's your intention?
The legacy PromQL path still evaluates histogram UDFs through `common_query::NativeHistogram`, a second
complete implementation of the same arithmetic. The two disagree — `NativeHistogram` computes exponential
bucket bounds as `2^x`, the native kernel uses the bound table; `NativeHistogram` walks quantiles only from
the lowest bucket, the kernel walks from the highest once a finite-sum quantile reaches the median. So one
histogram has two answers that differ in their last places, and a bug fixed in one is still present in the
other.
This PR makes the legacy path read the native kernel instead of re-deriving the answer. It does not change
`FloatHistogram`, does not touch `servers`/`mito2`/`operator`, and keeps every legacy signature.
### What moves
- `histogram_eval/legacy_bridge.rs` (new): borrows a `NativeHistogram` as the `HistogramView` both
representations share, calls `histogram_eval::native::{quantile, fraction}`, and maps the kernel's outcome
back onto the NaN-observation notes the legacy UDFs record.
- The same bridge carries `add` / `sub` / `scale` / `divide_by` onto `FloatHistogram::{add, sub, mul, div}`,
folding `AddOutcome` into the existing semantics. That reaches `neg`, the sum/avg accumulators,
`delta`/`idelta`/`rate`/`increase`, and the arithmetic UDFs.
- `FloatHistogram::spans()` widens to `pub(crate)` (no semantic change) so the bridge can read a view's span
layout.
- `common_query::NativeHistogram` keeps its own estimator for the API surface that still calls it:
`promql_string`, `estimated_stdvar` and `to_prometheus_buckets` share `all_buckets`/`get_bound`, and
`quantile_from_parts` is reached only through `quantile_with_info` (now delegating) and the public
`quantile_histogram_view`. Removing any of it is a separate `common-query` cleanup: that crate cannot
depend on `promql`, and retiring `quantile`/`fraction`/`negated` and `quantile_histogram_view` means
auditing their public API and remaining tests first.
### Behaviour changes that are not last-place differences
Delegating means the legacy answers are now the kernel's answers. Six differences are observable, and all
six are cases where the legacy code did something the kernel does not. Three of them change what ordinary,
well-formed data answers and need an explicit decision; the other three are only reachable through input that
violates Prometheus's invariants, so reviewing them is a compatibility call rather than a correctness one.
This list is what two review passes found; it is not claimed to be exhaustive.
**Well-formed data (decision required)**
1. **`start_timestamp` is cleared when a histogram is derived from another.** Legacy `add`/`sub`/`scale`/
`divide_by` kept the input's field; `FloatHistogram` clears it on every operation that derives a value.
That is this module's own policy, documented at `float_histogram.rs`: Prometheus's `FloatHistogram` has no
such field, so nothing here is a Prometheus rule being newly enforced. `rate`, `delta`, `increase`,
`idelta`, the arithmetic UDFs and `-h` therefore answer `start_timestamp = None`. The sum/avg aggregates
clear it only once they combine **two or more** samples: the first sample is passed through untouched, so
a single-row aggregate keeps its stored start timestamp.
2. **`sum = NaN` with more than one bucket.** A NaN sum marks NaN observations, which is legal data. The
kernel reproduces Prometheus's branch that walks the remaining buckets and leaves the last one for the
interpolation, so the answer comes from the top bucket rather than the bucket the rank fell into. For
schema 0, `span(0, 2)`, counts `[8, 1]`, `count = 10`, `q = 0.5`: legacy answered `2^-0.375 ~ 0.7711`
(the bucket the rank lands in), the kernel answers `2^5 = 32`.
`quantile_with_nan_sum_uses_the_last_bucket` pins `32.0`, and `quantile_with_info_is_the_native_quantile`
pins agreement with the kernel across every fixture.
3. **Quantile ties at an exact bucket boundary.** Legacy always walked upward; Prometheus walks from the
highest bucket once a finite-sum quantile reaches 0.5. At an exact tie the two pick different buckets whose
bounds are both defensible; the kernel's answer now wins.
**Input that violates Prometheus's invariants (reviewer sign-off, not a data-path regression)**
4. **Zero-bucket widening clamps differently.** Both representations merge every bucket inside the raised
threshold into `zero_count` and remove it from the spans, so the ordinary case agrees. They differ when a
populated bucket straddles the new threshold: the kernel widens the threshold to that bucket's upper bound
instead of splitting it, while legacy decides each bucket by its own upper bound and keeps the straddler.
Legacy also does nothing at all when the threshold is unchanged.
5. **`count` larger than the buckets' total with a finite `sum`.** Legacy returned `NaN`; Prometheus clamps
and interpolates, which the kernel now does.
6. **Custom buckets carrying a positive `zero_count`.** The kernel refuses ("histograms with custom buckets
have no zero bucket") on `zero_count > 0` in the quantile path and `zero_count != 0` in the fraction path,
so legacy now answers `NaN` where it used to estimate.
Two details that are not semantics: `add`/`sub` still `compact()` the sum so `promql_eq` and the serialized
spans keep their previous answers, and `scale`/`divide_by` fall back to the field-by-field multiplication for
a layout a view cannot express (spans disagreeing with the counts, or a bucket index leaving `i32`) — verified
to still scale such a histogram rather than dropping it, while `add`/`sub` return `None` for it exactly as
before.
## Verification
- `cargo nextest run -p promql -p common-query -p datatypes` — **1808 passed, 0 skipped, 0 failed**.
- `cargo clippy -p promql -p common-query --all-targets -- -D warnings` — clean.
- `cargo fmt --all -- --check` — clean.
- Both commits were rebuilt independently and checked: commit 1 alone passes fmt, clippy and
`nextest -p promql -p common-query` (1384 passed).
Bit-equality tests in `histogram_eval/legacy_bridge.rs` (all passing):
| test | what it pins |
| --- | --- |
| `quantile_with_info_is_the_native_quantile` | six histograms (custom, gapped exponential, NaN sum, both sides + zero bucket, zero-bucket-only, empty) × 11 quantiles (`-inf, -0.1, 0, 0.001, 0.25, 0.5, 0.75, 1, 1.1, +inf, NaN`), compared with `to_bits()` |
| `fraction_with_info_is_the_native_fraction` | the same shape over eight bound pairs |
| `add_is_the_native_add` / `sub_is_the_native_sub` | bucket counts by index (`to_bits()`) plus `schema`/`zero_threshold`/`count`/`sum`/`zero_count` over five pairs (same layout, schema reduction, zero-threshold growth, mismatched custom bounds, NaN sum) |
| `scale_and_divide_by_are_the_native_mul_and_div` | the same comparison over factors `2, 0.5, -1, 0, +inf` |
| `nan_notes_follow_the_kernel_outcome` | locks the annotation mapping (NaNSkew vs NaNResult vs none) |
## Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [ ] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`).
### Notes
- `NativeHistogram::{add,sub,scale,divide_by,quantile_with_info,fraction_with_info}` remain in `common_query`
(public API, still covered by that crate's own tests) but no PromQL path reaches them any more. Deleting them
is a follow-up in `common_query` alone.
- The next step in this direction is moving the legacy aggregate accumulator onto the native
`HistogramAccumulator` fold/merge, so the sum/avg partial state has one implementation too.
+17
View File
@@ -984,6 +984,14 @@ fn add_reset_hint(left: CounterResetHint, right: CounterResetHint) -> CounterRes
}
/// Estimates a quantile from histogram metadata and a lazily materialized bucket sequence.
///
/// This is the estimator [`NativeHistogram::quantile_with_info`] and [`quantile_histogram_view`]
/// expose, so it stays for callers of this crate. The PromQL `histogram_quantile` does not use it:
/// that function reads the same histogram through the native kernel in the `promql` crate, which
/// walks the buckets the way Prometheus does -- from the highest bucket down for a quantile above
/// the median -- so one histogram has one answer instead of two that differ in their last places.
/// The exponential boundary below interpolates with `2^x`, which likewise stays only for this
/// estimator; the kernel interpolates with `exp2` over Prometheus's bound table.
fn quantile_from_parts(
q: f64,
schema: i32,
@@ -1391,6 +1399,15 @@ pub fn exponential_overflow_bucket_index(schema: i32) -> Option<i32> {
i32::try_from(last_finite.checked_add(1)?).ok()
}
/// The bound of bucket `idx`: a custom bound, an infinity sentinel, or `2^(idx / 2^schema)`.
///
/// The exponential case computes its bound instead of reading one, while Prometheus reads
/// `ExponentialBounds`: those entries are not all correctly rounded, because a bound becomes a zero
/// threshold that is compared exactly, and a computed one would not compare as the sender's did.
/// So the computed bound here agrees with the table the native histogram kernel uses only to within
/// an ulp. This function keeps serving the renderers and the estimators of this crate's public
/// `NativeHistogram` API (`promql_string`, `estimated_stdvar`, the legacy quantile above, the
/// zero-threshold reconciliation in `add`); the PromQL estimators read the kernel's `bound`.
fn get_bound(idx: i32, schema: i32, custom_values: &[f64]) -> Option<f64> {
if schema == CUSTOM_BUCKETS_SCHEMA {
return match idx {
@@ -6082,7 +6082,7 @@ async fn decode_state_batches(batches: Vec<RecordBatch>) -> Vec<StateFrameRow> {
assert_eq!(FrameContent::AggregateState, declaration.content());
let series = Arc::new(declaration.series().clone());
let state = Arc::new(declaration.aggregate_state().unwrap().to_owned());
let decoded = unpack_aggregate_state(
unpack_aggregate_state(
Box::pin(RecordBatchStreamAdapter::new(
schema,
futures::stream::iter(batches.into_iter().map(Ok)),
@@ -6093,8 +6093,7 @@ async fn decode_state_batches(batches: Vec<RecordBatch>) -> Vec<StateFrameRow> {
.unwrap()
.try_collect::<Vec<_>>()
.await
.unwrap();
decoded
.unwrap()
}
/// State is keyed by the cut aggregate's grouping, not by the tags its scan was bound with.
+97 -23
View File
@@ -39,6 +39,7 @@ use crate::functions::{
MinOverTime, PredictLinear, QuantileOverTime, Rate, StddevOverTime, StdvarOverTime,
SumOverTime, extract_array, extract_range_dict,
};
use crate::histogram_eval::legacy_bridge;
use crate::range_array::{RangeArray, unpack};
fn extract_histogram_array(value: &ColumnarValue, func_name: &str) -> DfResult<ArrayRef> {
@@ -568,11 +569,11 @@ impl NativeHistogramAdd {
}
pub fn scalar_udf() -> ScalarUDF {
histogram_pair_udf(Self::name(), NativeHistogram::add)
histogram_pair_udf(Self::name(), legacy_bridge::add)
}
pub fn scalar_udf_with_collector(collector: Option<PromqlAnnotationCollector>) -> ScalarUDF {
histogram_pair_udf_with_collector(Self::name(), NativeHistogram::add, collector)
histogram_pair_udf_with_collector(Self::name(), legacy_bridge::add, collector)
}
}
@@ -584,11 +585,11 @@ impl NativeHistogramSub {
}
pub fn scalar_udf() -> ScalarUDF {
histogram_pair_udf(Self::name(), NativeHistogram::sub)
histogram_pair_udf(Self::name(), legacy_bridge::sub)
}
pub fn scalar_udf_with_collector(collector: Option<PromqlAnnotationCollector>) -> ScalarUDF {
histogram_pair_udf_with_collector(Self::name(), NativeHistogram::sub, collector)
histogram_pair_udf_with_collector(Self::name(), legacy_bridge::sub, collector)
}
}
@@ -605,7 +606,7 @@ impl NativeHistogramMulScalar {
vec![native_histogram_arrow_type(), DataType::Float64],
0,
1,
|histogram, scalar| Some(histogram.scale(scalar)),
|histogram, scalar| Some(legacy_bridge::scale(&histogram, scalar)),
)
}
}
@@ -623,7 +624,7 @@ impl NativeHistogramScalarMul {
vec![DataType::Float64, native_histogram_arrow_type()],
1,
0,
|histogram, scalar| Some(histogram.scale(scalar)),
|histogram, scalar| Some(legacy_bridge::scale(&histogram, scalar)),
)
}
}
@@ -641,7 +642,7 @@ impl NativeHistogramDivScalar {
vec![native_histogram_arrow_type(), DataType::Float64],
0,
1,
|histogram, scalar| Some(histogram.divide_by(scalar)),
|histogram, scalar| Some(legacy_bridge::divide_by(&histogram, scalar)),
)
}
}
@@ -654,7 +655,9 @@ impl NativeHistogramNeg {
}
pub fn scalar_udf() -> ScalarUDF {
histogram_transform_udf(Self::name(), NativeHistogram::negated)
histogram_transform_udf(Self::name(), |histogram| {
legacy_bridge::scale(&histogram, -1.0)
})
}
}
@@ -814,7 +817,7 @@ impl NativeHistogramQuantile {
vec![DataType::Float64],
move |histogram, input, row, len, name| {
let q = read_scalar_f64_arg(&input[1], row, len, name)?;
let (value, info) = histogram.quantile_with_info(q);
let (value, info) = legacy_bridge::quantile_with_info(histogram, q);
if let Some(info) = info {
let message = match info {
NativeHistogramQuantileInfo::NaNSkew => {
@@ -850,7 +853,8 @@ impl NativeHistogramFraction {
move |histogram, input, row, len, name| {
let lower = read_scalar_f64_arg(&input[1], row, len, name)?;
let upper = read_scalar_f64_arg(&input[2], row, len, name)?;
let (value, excluded_nans) = histogram.fraction_with_info(lower, upper);
let (value, excluded_nans) =
legacy_bridge::fraction_with_info(histogram, lower, upper);
if excluded_nans {
record_info(
&collector,
@@ -987,7 +991,7 @@ impl NativeHistogramAggregateAccumulator {
Some(value) => {
record_custom_reconciliation(&self.collector, self.kind.name(), &value, &histogram);
let combined = match self.kind {
NativeHistogramAggregateKind::Sum => value.add(&histogram),
NativeHistogramAggregateKind::Sum => legacy_bridge::add(&value, &histogram),
NativeHistogramAggregateKind::Avg => {
weighted_histogram_mean(value, self.count, histogram, count, combined_count)
}
@@ -1035,8 +1039,10 @@ fn weighted_histogram_mean(
total_count: u64,
) -> Option<NativeHistogram> {
let total_count = total_count as f64;
left.scale(left_count as f64 / total_count)
.add(&right.scale(right_count as f64 / total_count))
legacy_bridge::add(
&legacy_bridge::scale(&left, left_count as f64 / total_count),
&legacy_bridge::scale(&right, right_count as f64 / total_count),
)
}
fn range_fold_histograms(
@@ -1063,7 +1069,7 @@ fn range_fold_histograms(
record_custom_reconciliation(collector, name, &value, &histogram);
let next_count = count.checked_add(1)?;
let combined = match kind {
NativeHistogramAggregateKind::Sum => value.add(&histogram),
NativeHistogramAggregateKind::Sum => legacy_bridge::add(&value, &histogram),
NativeHistogramAggregateKind::Avg => {
weighted_histogram_mean(value, count, histogram, 1, next_count)
}
@@ -2419,9 +2425,7 @@ fn histogram_delta(
}
if !is_counter {
return samples
.last()?
.sub(samples.first()?)
return legacy_bridge::sub(samples.last()?, samples.first()?)
.map(NativeHistogram::into_gauge);
}
@@ -2433,14 +2437,14 @@ fn histogram_delta(
} else {
(samples[0].clone(), 1)
};
let mut result = samples.last()?.sub(&initial)?;
let mut result = legacy_bridge::sub(samples.last()?, &initial)?;
for index in reset_scan_start..samples.len() {
if samples[index].detect_counter_reset(
&samples[index - 1],
timestamps[index - 1],
timestamps[index],
) {
result = result.add(&samples[index - 1])?;
result = legacy_bridge::add(&result, &samples[index - 1])?;
}
}
Some(result.into_gauge())
@@ -2461,11 +2465,11 @@ fn idelta_value(
let result = if is_rate && current.detect_counter_reset(previous, previous_ts, current_ts) {
current.clone()
} else {
current.sub(previous)?
legacy_bridge::sub(current, previous)?
};
Some(
if is_rate {
result.scale(1.0 / sampled_interval_secs)
legacy_bridge::scale(&result, 1.0 / sampled_interval_secs)
} else {
result
}
@@ -2599,7 +2603,7 @@ fn native_extrapolated_rate<const IS_COUNTER: bool, const IS_RATE: bool>(
continue;
};
if synthetic_zero_start && length > 1 {
let Some(with_synthetic_zero) = histogram.add(&samples[0]) else {
let Some(with_synthetic_zero) = legacy_bridge::add(&histogram, &samples[0]) else {
record_warning(
&collector,
format!(
@@ -2656,7 +2660,7 @@ fn native_extrapolated_rate<const IS_COUNTER: bool, const IS_RATE: bool>(
if IS_RATE {
factor /= range_length_secs;
}
histogram = histogram.scale(factor).into_gauge();
histogram = legacy_bridge::scale(&histogram, factor).into_gauge();
result.push(Some(histogram));
}
@@ -3359,6 +3363,41 @@ mod tests {
assert!(histogram.quantile(0.5) < 2.0);
}
#[test]
fn sum_aggregate_keeps_the_start_timestamp_of_a_single_sample() {
let mut histogram = sample_histogram(2.0, 3.0, vec![2.0]);
histogram.start_timestamp = Some(1_234);
let mut accumulator =
NativeHistogramAggregateAccumulator::new(NativeHistogramAggregateKind::Sum, None);
accumulator.push_histogram(histogram, 1).unwrap();
assert_eq!(
accumulator.value.unwrap().start_timestamp,
Some(1_234),
"the first sample is passed through untouched, so its start timestamp survives"
);
}
#[test]
fn sum_aggregate_clears_the_start_timestamp_once_it_combines_two_samples() {
let mut first = sample_histogram(2.0, 3.0, vec![2.0]);
first.start_timestamp = Some(1_234);
let mut second = sample_histogram(2.0, 3.0, vec![2.0]);
second.start_timestamp = Some(5_678);
let mut accumulator =
NativeHistogramAggregateAccumulator::new(NativeHistogramAggregateKind::Sum, None);
accumulator.push_histogram(first, 1).unwrap();
accumulator.push_histogram(second, 1).unwrap();
assert_eq!(
accumulator.value.unwrap().start_timestamp,
None,
"a derived sum carries no start timestamp of its own"
);
}
#[test]
fn comparison_observes_explicit_sparse_zero_buckets() {
let mut left = sample_histogram(1.0, 1.0, vec![1.0, 0.0]);
@@ -3955,6 +3994,41 @@ mod tests {
assert_eq!(increase.sum, 4.0);
}
/// The raw range functions report a derived difference or quotient, so none of them keeps the
/// start timestamp of the samples it read: the field names the start of the window one sample
/// belongs to, and a result that spans several samples has no single such window.
#[test]
fn raw_range_functions_clear_the_start_timestamp() {
let mut first = sample_histogram(2.0, 2.0, vec![2.0]);
first.start_timestamp = Some(1_000);
let mut last = sample_histogram(4.0, 4.0, vec![4.0]);
last.start_timestamp = Some(2_000);
for udf in [
NativeHistogramRate::scalar_udf(),
NativeHistogramIncrease::scalar_udf(),
NativeHistogramDelta::scalar_udf(),
] {
let name = udf.name().to_string();
let result = run_extrapolated_histogram_udf(udf, vec![first.clone(), last.clone()]);
assert_eq!(
result.start_timestamp, None,
"{name}: the range functions derive a value of their own"
);
}
for udf in [
NativeHistogramIDelta::scalar_udf(),
NativeHistogramIRate::scalar_udf(),
] {
let name = udf.name().to_string();
let result = run_histogram_range_udf(udf, vec![first.clone(), last.clone()]);
assert_eq!(
result.start_timestamp, None,
"{name}: the range functions derive a value of their own"
);
}
}
#[test]
fn extrapolated_rate_requires_strictly_in_range_start_timestamp() {
for (start_timestamp, range_end, range_length) in [
+1
View File
@@ -24,6 +24,7 @@ mod classic;
// functions and binary operators, which build on other branches.
#[cfg_attr(not(test), allow(dead_code))]
pub(crate) mod float_histogram;
pub(crate) mod legacy_bridge;
mod native;
#[cfg(test)]
mod tests;
@@ -123,8 +123,13 @@ impl FloatHistogram {
.saturating_add(layout)
}
/// The spans of each side.
fn spans(&self) -> [Vec<(i32, i32)>; 2] {
/// The spans of each side: the layout a view was read with, or the compacted layout of a
/// histogram this crate derived.
///
/// [`crate::histogram_eval::legacy_bridge`] reads them to build the legacy
/// `NativeHistogram`, whose spans are explicit where this type keeps them in the bucket
/// indices.
pub(crate) fn spans(&self) -> [Vec<(i32, i32)>; 2] {
self.layout
.clone()
.unwrap_or_else(|| [&self.positive, &self.negative].map(canonical_spans))
File diff suppressed because it is too large Load Diff
+219 -4
View File
@@ -5931,6 +5931,43 @@ pub async fn test_remote_write_v2_histogram_native_composition(store_type: Stora
tests_integration::test_util::execute_sql(instance.fe_instance(), sql).await;
}
// A counter histogram with a start timestamp strictly inside `(0, 3000]`, its window's only
// sample: `rate` can answer that window through the start timestamp alone.
let start_request = remote_write_v2::request_with_labels_and_histograms(
vec![
(prom_store::METRIC_NAME_LABEL, "native_comp_start"),
("job", "api"),
],
vec![Histogram {
count: Some(Count::CountInt(8)),
sum: 10.0,
schema: 1,
positive_spans: vec![BucketSpan {
offset: 0,
length: 2,
}],
positive_deltas: vec![4, 0],
reset_hint: 2,
start_timestamp: 1_500,
timestamp: 3_000,
..Default::default()
}],
);
let payload = prom_store::snappy_compress(&start_request.encode_to_vec()).unwrap();
let written = client
.post("/v1/prometheus/write")
.header("Content-Encoding", "snappy")
.header(
"Content-Type",
"application/x-protobuf;proto=io.prometheus.write.v2.Request",
)
.body(payload)
.send()
.await;
let status = written.status();
let body = written.text().await;
assert_eq!(status, StatusCode::NO_CONTENT, "write rejected: {body}");
// The first sample has two equal buckets of 5; h2 has two equal buckets of 2.
// Repeat the same public requests after flushing so the source can be memtable or SST.
for phase in ["memtable", "flushed"] {
@@ -5940,6 +5977,7 @@ pub async fn test_remote_write_v2_histogram_native_composition(store_type: Stora
"native_comp_h2",
"native_comp_dups",
"native_comp_f",
"native_comp_start",
] {
tests_integration::test_util::execute_sql(
instance.fe_instance(),
@@ -6217,6 +6255,56 @@ pub async fn test_remote_write_v2_histogram_native_composition(store_type: Stora
}
assert!(body.get("infos").is_none(), "{phase}: {query}: {body}");
assert!(body.get("warnings").is_none(), "{phase}: {query}: {body}");
// The same two windows asked without a hint, so the legacy route answers them and the
// observer proves it did. This is where the delegated kernel has to agree with the native
// route: the raw sample's start timestamp is what lets `rate` measure the window, while
// `* 1` gives that timestamp up, so the derived window has nothing to measure against.
for (query, expected_count) in [
("rate(native_comp_start[3s])", Some(8.0 / 3.0)),
("rate((native_comp_start * 1)[3s:1s])", None),
] {
let res = client
.get(&format!(
"/v1/prometheus/api/v1/query?query={}&time=3",
encode(query)
))
.send()
.await;
let status = res.status();
let body: Value = res.json().await;
assert_eq!(status, StatusCode::OK, "{phase}: {query}: {body}");
let results = body["data"]["result"]
.as_array()
.unwrap_or_else(|| panic!("{phase}: {query}: no result in {body}"));
match expected_count {
Some(count) => {
assert_eq!(results.len(), 1, "{phase}: {query}: {body}");
let reported: f64 = body["data"]["result"][0]["histogram"][1]["count"]
.as_str()
.unwrap_or_else(|| panic!("{phase}: {query}: no count in {body}"))
.parse()
.unwrap();
assert!(
(reported - count).abs() < 1e-12,
"{phase}: {query}: expected count {count}, got {body}"
);
}
None => assert!(
results.is_empty(),
"{phase}: {query}: `* 1` clears the start timestamp, so the derived window \
has nothing to measure: {body}"
),
}
assert!(
body.get("infos").is_none() && body.get("warnings").is_none(),
"{phase}: {query}: a counter histogram has nothing to report: {body}"
);
assert!(
!executes_fused_series(&route.take_plan()),
"{phase}: {query} did not run on the legacy route"
);
}
}
}
@@ -6755,12 +6843,22 @@ pub async fn test_remote_write_v2_histogram_count_and_group_native_route(store_t
pub async fn test_prometheus_remote_write_v2_native_histogram(store_type: StorageType) {
common_telemetry::init_default_ut_logging();
let (app, mut guard) = setup_test_prom_app_with_frontend(
store_type,
// The histogram answers at the end of this test are legacy ones, so the route observer is what
// proves the route they ran on: the engine-wide option stays off and no request carries a hint.
let route = Arc::new(PromRouteObserver::default());
let plugins = Plugins::default();
plugins.insert::<PromQueryInterceptorRef<servers::error::Error>>(route.clone());
let mut instance = tests_integration::standalone::GreptimeDbStandaloneBuilder::new(
"prometheus_remote_write_v2_native_histogram",
)
.with_default_store_type(store_type)
.with_plugin(plugins)
.build()
.await;
let client = TestClient::new(app).await;
let server = build_test_prom_server(instance.fe_instance().clone(), false, None)
.with_greptime_config_options(instance.opts.datanode_options().to_toml().unwrap())
.build();
let client = TestClient::new(server.build(server.make_app()).unwrap()).await;
let mut write_request = remote_write_v2::request_with_labels_and_samples(
vec![
@@ -6914,6 +7012,123 @@ pub async fn test_prometheus_remote_write_v2_native_histogram(store_type: Storag
)
.await;
// The raw sample carries a start timestamp of 1500, strictly inside the window `(0, 3000]` of
// the requests below, and it is the window's only sample. A legacy `rate`/`increase` can only
// answer such a window through that start timestamp: the whole sample is what the increase
// since the synthetic zero measures, and `rate` is that over the window's seconds.
for (query, count, sum) in [
(
"rate(remote_write_v2_latency_seconds[3s])",
8.0 / 3.0,
10.0 / 3.0,
),
("increase(remote_write_v2_latency_seconds[3s])", 8.0, 10.0),
] {
let res = client
.get(&format!(
"/v1/prometheus/api/v1/query?query={}&time=3",
encode(query)
))
.send()
.await;
let status = res.status();
let body: Value = res.json().await;
assert_eq!(status, StatusCode::OK, "{query}: {body}");
assert_eq!(body["data"]["resultType"], "vector", "{query}: {body}");
assert_eq!(
body["data"]["result"].as_array().map(Vec::len),
Some(1),
"{query}: {body}"
);
let histogram = &body["data"]["result"][0]["histogram"][1];
let reported = |field: &str| -> f64 {
histogram[field]
.as_str()
.unwrap_or_else(|| panic!("no histogram {field} in {body}"))
.parse()
.unwrap()
};
assert!(
(reported("count") - count).abs() < 1e-12 && (reported("sum") - sum).abs() < 1e-12,
"{query}: expected count {count} and sum {sum}, got {body}"
);
assert!(
body.get("infos").is_none() && body.get("warnings").is_none(),
"{query}: a counter histogram with a start timestamp has nothing to report: {body}"
);
assert!(
!executes_fused_series(&route.take_plan()),
"{query} did not run on the legacy route"
);
}
// A NaN observation: the buckets hold nine of the count's ten observations and the sum is NaN,
// which is how Prometheus marks the rest. Prometheus's NaN branch walks the buckets forward and
// leaves the top bucket for the interpolation, so the half quantile reads 2^5 rather than the
// 0.7711 the buckets' own rank gives - and the response has to say why.
let nan_request = remote_write_v2::request_with_labels_and_histograms(
vec![
(
prom_store::METRIC_NAME_LABEL,
"remote_write_v2_nan_quantile",
),
("job", "api"),
],
vec![Histogram {
count: Some(Count::CountInt(10)),
sum: f64::NAN,
// A bucket's upper bound is 2^(index * 2^-schema), so schema 0 makes the two buckets
// (0.5, 1] and (1, 2]; the deltas hold 8 and then 1.
schema: 0,
positive_spans: vec![BucketSpan {
offset: 0,
length: 2,
}],
positive_deltas: vec![8, -7],
reset_hint: 2,
timestamp: 3_000,
..Default::default()
}],
);
let payload = prom_store::snappy_compress(&nan_request.encode_to_vec()).unwrap();
let written = client
.post("/v1/prometheus/write")
.header("Content-Encoding", "snappy")
.header(
"Content-Type",
"application/x-protobuf;proto=io.prometheus.write.v2.Request",
)
.body(payload)
.send()
.await;
let status = written.status();
let body = written.text().await;
assert_eq!(status, StatusCode::NO_CONTENT, "write rejected: {body}");
let res = client
.get(&format!(
"/v1/prometheus/api/v1/query?query={}&time=3",
encode("histogram_quantile(0.5, remote_write_v2_nan_quantile)")
))
.send()
.await;
assert_eq!(res.status(), StatusCode::OK);
let body: Value = res.json().await;
assert_eq!(body["data"]["resultType"], "vector", "{body}");
assert_eq!(
body["data"]["result"][0]["value"][1], "32.0",
"the NaN walk leaves the top bucket: {body}"
);
assert_eq!(
body["infos"],
json!(["input to histogram_quantile has NaN observations, result is skewed higher"]),
"{body}"
);
assert!(
!executes_fused_series(&route.take_plan()),
"`histogram_quantile` did not run on the legacy route"
);
for flushed in [false, true] {
if flushed {
flush_table(&client, "remote_write_v2_latency_seconds").await;
@@ -7209,7 +7424,7 @@ pub async fn test_prometheus_remote_write_v2_native_histogram(store_type: Storag
)
.await;
guard.remove_all().await;
instance.guard.remove_all().await;
}
#[apply(both_deployment_cases)]