refactor(promql): let the legacy histogram path read the native kernel (#10420)

* refactor(promql): read the legacy histogram estimators from the native kernel

`prom_native_histogram_quantile` and `prom_native_histogram_fraction` estimated from
`common_query`'s `NativeHistogram` bucket walk: exponential bucket bounds computed as `2^x`, and a
forward walk from the lowest bucket only. The native kernel in `histogram_eval` walks the same
histogram the way Prometheus does -- the bound table in `bounds`, and a walk from the highest bucket
down once a finite-sum quantile reaches the median -- so one histogram had two answers that differ
in their last places.

Read the `NativeHistogram` through the borrowed `HistogramView` both representations share and
return the kernel's answer, mapping its outcome back onto the NaN-observation notes the legacy UDFs
record. Their signatures and the notes are unchanged; only the arithmetic behind them moves.

The `2^x` estimator in `common_query::native_histogram` stays for that crate's public API and is
annotated as such: no PromQL path reaches it any more.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): delegate the legacy histogram arithmetic to the native kernel

`prom_native_histogram_add`, `_sub`, `_mul_scalar`, `_scalar_mul`, `_div_scalar`, `_neg`, the
sum/avg accumulators, `delta`/`idelta`/`rate`/`increase`, and the weighted mean that merges partial
states all scaled and combined `NativeHistogram`s with that type's own reconciliation: its own
zero-bucket widening, schema reduction and custom-bound intersection, over `2^x` bounds.
`FloatHistogram::add`/`sub`/`mul`/`div` already implement Prometheus's rules (`addBuckets`,
`reconcileZeroBuckets`, `CopyToSchema`, `intersectCustomBucketBounds`) and the native kernel's
golden tests cover them, so this keeps one implementation of the semantics.

`AddOutcome` folds into the existing semantics rather than being dropped: `custom_bounds_reconciled`
is the intersection the kernel keeps in `custom_values`, which the callers already announce from the
operands, and `counter_reset_collision` is the contradiction the kernel answers with the unknown
hint the legacy arithmetic derived from two hints that disagree; both are asserted in debug builds.
The public `NativeHistogram` signatures are unchanged, the sums are compacted as before so
`promql_eq` and the serialized spans keep their answers, and unary minus goes through the kernel's
multiplication like the other scalings.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the legacy delegation's observable answers

Comparing the bridge against the kernel it calls only shows they agree today; it cannot show
what either answer means. These tests pin the answers themselves.

- quantile with a NaN sum over several buckets answers 32.0 with NaNSkew, because the NaN walk
  leaves the last bucket for the interpolation; below the rank it answers NaN with NaNResult.
- an exact tie answers 1.0 through the legacy walk and 2.0 through the kernel, on a fixture with
  an empty bucket between the two, and neither records a note.
- assert_same_payload now compares every field, including the start timestamp policy, custom
  values, the complete span segmentation and the bucket bits.
- round-tripping preserves reset hints, custom values, zero-length spans and gaps.
- add/sub reconcile the reset hint and the custom bounds, compact cancelled buckets, and keep
  promql_eq's expanded-index equality.
- scale/divide_by keep the field-by-field fallback for an unreadable layout -- mismatched counts,
  a backwards span, an overflowing index, a negative length -- and preserve its start timestamp;
  add/sub answer None for the same layouts. Zero, negative and NaN factors are pinned.
- rate/increase/delta/idelta/irate clear the start timestamp of a derived histogram, while a
  single-sample sum aggregate keeps its stored one.

A layout whose later span steps backwards is refused before the kernel sees it: the legacy
bucket_indices rejected it, while side_from_view would have accepted repeated indices.

Signed-off-by: discord9 <discord9@163.com>

* test(http): pin the legacy histogram route's rate, NaN quantile and derived windows

The delegation changes what three kinds of well-formed data answer, so assert it where a user
sees it. Every request runs without the native hint and asserts !executes_fused_series on the
observed plan, which proves the legacy route rather than assuming it from the missing hint.

- rate and increase over a raw sample with a start timestamp inside the window still extrapolate
  from the synthetic zero: count 8/3 and sum 10/3 for a rate, 8 and 10 for an increase, with no
  infos or warnings.
- histogram_quantile(0.5) over schema 0, buckets [8, 1], count 10 and a NaN sum answers 32.0 and
  reports the NaN-observation info, matching Prometheus's walk that leaves the last bucket for
  the interpolation. The previous estimator answered 2^-0.375.
- rate over a raw window answers one series while the same window over h*1 answers none: the
  derived sample carries no start timestamp, so it has no synthetic zero to extrapolate from.
  Both phases are covered, memtable and flushed SST.

The native-histogram test now installs the route interceptor the way its sibling tests do, so the
plan can be observed; all of its original assertions are unchanged.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
This commit is contained in:
discord9
2026-11-16 16:26:52 +08:00
committed by GitHub
parent 50b6b5bf82
commit 5517c4b76f
8 changed files with 1801 additions and 32 deletions
+17
View File
@@ -984,6 +984,14 @@ fn add_reset_hint(left: CounterResetHint, right: CounterResetHint) -> CounterRes
}
/// Estimates a quantile from histogram metadata and a lazily materialized bucket sequence.
///
/// This is the estimator [`NativeHistogram::quantile_with_info`] and [`quantile_histogram_view`]
/// expose, so it stays for callers of this crate. The PromQL `histogram_quantile` does not use it:
/// that function reads the same histogram through the native kernel in the `promql` crate, which
/// walks the buckets the way Prometheus does -- from the highest bucket down for a quantile above
/// the median -- so one histogram has one answer instead of two that differ in their last places.
/// The exponential boundary below interpolates with `2^x`, which likewise stays only for this
/// estimator; the kernel interpolates with `exp2` over Prometheus's bound table.
fn quantile_from_parts(
q: f64,
schema: i32,
@@ -1391,6 +1399,15 @@ pub fn exponential_overflow_bucket_index(schema: i32) -> Option<i32> {
i32::try_from(last_finite.checked_add(1)?).ok()
}
/// The bound of bucket `idx`: a custom bound, an infinity sentinel, or `2^(idx / 2^schema)`.
///
/// The exponential case computes its bound instead of reading one, while Prometheus reads
/// `ExponentialBounds`: those entries are not all correctly rounded, because a bound becomes a zero
/// threshold that is compared exactly, and a computed one would not compare as the sender's did.
/// So the computed bound here agrees with the table the native histogram kernel uses only to within
/// an ulp. This function keeps serving the renderers and the estimators of this crate's public
/// `NativeHistogram` API (`promql_string`, `estimated_stdvar`, the legacy quantile above, the
/// zero-threshold reconciliation in `add`); the PromQL estimators read the kernel's `bound`.
fn get_bound(idx: i32, schema: i32, custom_values: &[f64]) -> Option<f64> {
if schema == CUSTOM_BUCKETS_SCHEMA {
return match idx {
@@ -6082,7 +6082,7 @@ async fn decode_state_batches(batches: Vec<RecordBatch>) -> Vec<StateFrameRow> {
assert_eq!(FrameContent::AggregateState, declaration.content());
let series = Arc::new(declaration.series().clone());
let state = Arc::new(declaration.aggregate_state().unwrap().to_owned());
let decoded = unpack_aggregate_state(
unpack_aggregate_state(
Box::pin(RecordBatchStreamAdapter::new(
schema,
futures::stream::iter(batches.into_iter().map(Ok)),
@@ -6093,8 +6093,7 @@ async fn decode_state_batches(batches: Vec<RecordBatch>) -> Vec<StateFrameRow> {
.unwrap()
.try_collect::<Vec<_>>()
.await
.unwrap();
decoded
.unwrap()
}
/// State is keyed by the cut aggregate's grouping, not by the tags its scan was bound with.
+97 -23
View File
@@ -39,6 +39,7 @@ use crate::functions::{
MinOverTime, PredictLinear, QuantileOverTime, Rate, StddevOverTime, StdvarOverTime,
SumOverTime, extract_array, extract_range_dict,
};
use crate::histogram_eval::legacy_bridge;
use crate::range_array::{RangeArray, unpack};
fn extract_histogram_array(value: &ColumnarValue, func_name: &str) -> DfResult<ArrayRef> {
@@ -568,11 +569,11 @@ impl NativeHistogramAdd {
}
pub fn scalar_udf() -> ScalarUDF {
histogram_pair_udf(Self::name(), NativeHistogram::add)
histogram_pair_udf(Self::name(), legacy_bridge::add)
}
pub fn scalar_udf_with_collector(collector: Option<PromqlAnnotationCollector>) -> ScalarUDF {
histogram_pair_udf_with_collector(Self::name(), NativeHistogram::add, collector)
histogram_pair_udf_with_collector(Self::name(), legacy_bridge::add, collector)
}
}
@@ -584,11 +585,11 @@ impl NativeHistogramSub {
}
pub fn scalar_udf() -> ScalarUDF {
histogram_pair_udf(Self::name(), NativeHistogram::sub)
histogram_pair_udf(Self::name(), legacy_bridge::sub)
}
pub fn scalar_udf_with_collector(collector: Option<PromqlAnnotationCollector>) -> ScalarUDF {
histogram_pair_udf_with_collector(Self::name(), NativeHistogram::sub, collector)
histogram_pair_udf_with_collector(Self::name(), legacy_bridge::sub, collector)
}
}
@@ -605,7 +606,7 @@ impl NativeHistogramMulScalar {
vec![native_histogram_arrow_type(), DataType::Float64],
0,
1,
|histogram, scalar| Some(histogram.scale(scalar)),
|histogram, scalar| Some(legacy_bridge::scale(&histogram, scalar)),
)
}
}
@@ -623,7 +624,7 @@ impl NativeHistogramScalarMul {
vec![DataType::Float64, native_histogram_arrow_type()],
1,
0,
|histogram, scalar| Some(histogram.scale(scalar)),
|histogram, scalar| Some(legacy_bridge::scale(&histogram, scalar)),
)
}
}
@@ -641,7 +642,7 @@ impl NativeHistogramDivScalar {
vec![native_histogram_arrow_type(), DataType::Float64],
0,
1,
|histogram, scalar| Some(histogram.divide_by(scalar)),
|histogram, scalar| Some(legacy_bridge::divide_by(&histogram, scalar)),
)
}
}
@@ -654,7 +655,9 @@ impl NativeHistogramNeg {
}
pub fn scalar_udf() -> ScalarUDF {
histogram_transform_udf(Self::name(), NativeHistogram::negated)
histogram_transform_udf(Self::name(), |histogram| {
legacy_bridge::scale(&histogram, -1.0)
})
}
}
@@ -814,7 +817,7 @@ impl NativeHistogramQuantile {
vec![DataType::Float64],
move |histogram, input, row, len, name| {
let q = read_scalar_f64_arg(&input[1], row, len, name)?;
let (value, info) = histogram.quantile_with_info(q);
let (value, info) = legacy_bridge::quantile_with_info(histogram, q);
if let Some(info) = info {
let message = match info {
NativeHistogramQuantileInfo::NaNSkew => {
@@ -850,7 +853,8 @@ impl NativeHistogramFraction {
move |histogram, input, row, len, name| {
let lower = read_scalar_f64_arg(&input[1], row, len, name)?;
let upper = read_scalar_f64_arg(&input[2], row, len, name)?;
let (value, excluded_nans) = histogram.fraction_with_info(lower, upper);
let (value, excluded_nans) =
legacy_bridge::fraction_with_info(histogram, lower, upper);
if excluded_nans {
record_info(
&collector,
@@ -987,7 +991,7 @@ impl NativeHistogramAggregateAccumulator {
Some(value) => {
record_custom_reconciliation(&self.collector, self.kind.name(), &value, &histogram);
let combined = match self.kind {
NativeHistogramAggregateKind::Sum => value.add(&histogram),
NativeHistogramAggregateKind::Sum => legacy_bridge::add(&value, &histogram),
NativeHistogramAggregateKind::Avg => {
weighted_histogram_mean(value, self.count, histogram, count, combined_count)
}
@@ -1035,8 +1039,10 @@ fn weighted_histogram_mean(
total_count: u64,
) -> Option<NativeHistogram> {
let total_count = total_count as f64;
left.scale(left_count as f64 / total_count)
.add(&right.scale(right_count as f64 / total_count))
legacy_bridge::add(
&legacy_bridge::scale(&left, left_count as f64 / total_count),
&legacy_bridge::scale(&right, right_count as f64 / total_count),
)
}
fn range_fold_histograms(
@@ -1063,7 +1069,7 @@ fn range_fold_histograms(
record_custom_reconciliation(collector, name, &value, &histogram);
let next_count = count.checked_add(1)?;
let combined = match kind {
NativeHistogramAggregateKind::Sum => value.add(&histogram),
NativeHistogramAggregateKind::Sum => legacy_bridge::add(&value, &histogram),
NativeHistogramAggregateKind::Avg => {
weighted_histogram_mean(value, count, histogram, 1, next_count)
}
@@ -2419,9 +2425,7 @@ fn histogram_delta(
}
if !is_counter {
return samples
.last()?
.sub(samples.first()?)
return legacy_bridge::sub(samples.last()?, samples.first()?)
.map(NativeHistogram::into_gauge);
}
@@ -2433,14 +2437,14 @@ fn histogram_delta(
} else {
(samples[0].clone(), 1)
};
let mut result = samples.last()?.sub(&initial)?;
let mut result = legacy_bridge::sub(samples.last()?, &initial)?;
for index in reset_scan_start..samples.len() {
if samples[index].detect_counter_reset(
&samples[index - 1],
timestamps[index - 1],
timestamps[index],
) {
result = result.add(&samples[index - 1])?;
result = legacy_bridge::add(&result, &samples[index - 1])?;
}
}
Some(result.into_gauge())
@@ -2461,11 +2465,11 @@ fn idelta_value(
let result = if is_rate && current.detect_counter_reset(previous, previous_ts, current_ts) {
current.clone()
} else {
current.sub(previous)?
legacy_bridge::sub(current, previous)?
};
Some(
if is_rate {
result.scale(1.0 / sampled_interval_secs)
legacy_bridge::scale(&result, 1.0 / sampled_interval_secs)
} else {
result
}
@@ -2599,7 +2603,7 @@ fn native_extrapolated_rate<const IS_COUNTER: bool, const IS_RATE: bool>(
continue;
};
if synthetic_zero_start && length > 1 {
let Some(with_synthetic_zero) = histogram.add(&samples[0]) else {
let Some(with_synthetic_zero) = legacy_bridge::add(&histogram, &samples[0]) else {
record_warning(
&collector,
format!(
@@ -2656,7 +2660,7 @@ fn native_extrapolated_rate<const IS_COUNTER: bool, const IS_RATE: bool>(
if IS_RATE {
factor /= range_length_secs;
}
histogram = histogram.scale(factor).into_gauge();
histogram = legacy_bridge::scale(&histogram, factor).into_gauge();
result.push(Some(histogram));
}
@@ -3359,6 +3363,41 @@ mod tests {
assert!(histogram.quantile(0.5) < 2.0);
}
#[test]
fn sum_aggregate_keeps_the_start_timestamp_of_a_single_sample() {
let mut histogram = sample_histogram(2.0, 3.0, vec![2.0]);
histogram.start_timestamp = Some(1_234);
let mut accumulator =
NativeHistogramAggregateAccumulator::new(NativeHistogramAggregateKind::Sum, None);
accumulator.push_histogram(histogram, 1).unwrap();
assert_eq!(
accumulator.value.unwrap().start_timestamp,
Some(1_234),
"the first sample is passed through untouched, so its start timestamp survives"
);
}
#[test]
fn sum_aggregate_clears_the_start_timestamp_once_it_combines_two_samples() {
let mut first = sample_histogram(2.0, 3.0, vec![2.0]);
first.start_timestamp = Some(1_234);
let mut second = sample_histogram(2.0, 3.0, vec![2.0]);
second.start_timestamp = Some(5_678);
let mut accumulator =
NativeHistogramAggregateAccumulator::new(NativeHistogramAggregateKind::Sum, None);
accumulator.push_histogram(first, 1).unwrap();
accumulator.push_histogram(second, 1).unwrap();
assert_eq!(
accumulator.value.unwrap().start_timestamp,
None,
"a derived sum carries no start timestamp of its own"
);
}
#[test]
fn comparison_observes_explicit_sparse_zero_buckets() {
let mut left = sample_histogram(1.0, 1.0, vec![1.0, 0.0]);
@@ -3955,6 +3994,41 @@ mod tests {
assert_eq!(increase.sum, 4.0);
}
/// The raw range functions report a derived difference or quotient, so none of them keeps the
/// start timestamp of the samples it read: the field names the start of the window one sample
/// belongs to, and a result that spans several samples has no single such window.
#[test]
fn raw_range_functions_clear_the_start_timestamp() {
let mut first = sample_histogram(2.0, 2.0, vec![2.0]);
first.start_timestamp = Some(1_000);
let mut last = sample_histogram(4.0, 4.0, vec![4.0]);
last.start_timestamp = Some(2_000);
for udf in [
NativeHistogramRate::scalar_udf(),
NativeHistogramIncrease::scalar_udf(),
NativeHistogramDelta::scalar_udf(),
] {
let name = udf.name().to_string();
let result = run_extrapolated_histogram_udf(udf, vec![first.clone(), last.clone()]);
assert_eq!(
result.start_timestamp, None,
"{name}: the range functions derive a value of their own"
);
}
for udf in [
NativeHistogramIDelta::scalar_udf(),
NativeHistogramIRate::scalar_udf(),
] {
let name = udf.name().to_string();
let result = run_histogram_range_udf(udf, vec![first.clone(), last.clone()]);
assert_eq!(
result.start_timestamp, None,
"{name}: the range functions derive a value of their own"
);
}
}
#[test]
fn extrapolated_rate_requires_strictly_in_range_start_timestamp() {
for (start_timestamp, range_end, range_length) in [
+1
View File
@@ -24,6 +24,7 @@ mod classic;
// functions and binary operators, which build on other branches.
#[cfg_attr(not(test), allow(dead_code))]
pub(crate) mod float_histogram;
pub(crate) mod legacy_bridge;
mod native;
#[cfg(test)]
mod tests;
@@ -123,8 +123,13 @@ impl FloatHistogram {
.saturating_add(layout)
}
/// The spans of each side.
fn spans(&self) -> [Vec<(i32, i32)>; 2] {
/// The spans of each side: the layout a view was read with, or the compacted layout of a
/// histogram this crate derived.
///
/// [`crate::histogram_eval::legacy_bridge`] reads them to build the legacy
/// `NativeHistogram`, whose spans are explicit where this type keeps them in the bucket
/// indices.
pub(crate) fn spans(&self) -> [Vec<(i32, i32)>; 2] {
self.layout
.clone()
.unwrap_or_else(|| [&self.positive, &self.negative].map(canonical_spans))
File diff suppressed because it is too large Load Diff