Files
discord9 25263af9d0 feat(promql): fold and average native histograms in the native aggregate (#10199)
* feat(promql): carry a native histogram in aggregate state

`sum` and `avg` over native histograms need a partial state whose payload is
variable width, which the numeric state family cannot express. Add two kinds,
`histogram_sum` and `histogram_avg`, whose components are a live histogram, a
layout-carrying compensation, and the evidence a merge records.

A position's payload is one `StatePayload`, either the numeric components a
float kind carries or a boxed histogram state, so absence stays a single
`None` and a field never holds both families. The numeric encoding is byte for
byte what it was, and a histogram component nests the same struct a histogram
sample frame already uses.

The compensation must carry the live value's bucket layout: the kernel indexes
it by the value's positions, so a placeholder without bucket indices would be
an out-of-bounds read rather than a loss of precision. Both the producer and
the decoder check the schema, the zero threshold, the custom bounds and the
expanded logical bucket indices, which accepts two spellings of the same
buckets and refuses two different layouts.

State accounting charges the payload's real width instead of a fixed slot, so
a wide bucket layout is paid for. `HistogramSum` cannot declare
`incremental_mean`, which is checked structurally: a sum has no mean mode.

No admission, split or runtime wiring yet; nothing produces these states until
the aggregate kernels land.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): fold native histograms into a `sum` aggregate

`sum` over a native histogram now folds in the native aggregate: the first
sample at a grid position establishes the accumulator from a layout-carrying
zero compensation, and every later one is added with a compensated add whose
receiver is the live value.

The evidence the final answer needs is recorded as it appears: set when a merge
reconciled custom bucket bounds, and both reset-hint bits when samples reported
a reset and reported that none happened. Two layouts that cannot be added -
exponential and custom buckets in one position - poison the point, which the
final answer drops instead of answering a partial sum.

A numeric overflow is not a fault and does not poison anything: `sum` answers
`Inf` when its cumulative sum saturates, exactly as the kernel's own sum does.

`avg` keeps refusing a histogram: its mean mode needs the merge rules, which
are not part of this change. A group that kept histogram payloads now refuses
to export a partial numeric state rather than exporting an empty one.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): average native histograms in the native aggregate

`avg` over a native histogram folds the same way the v3.15.0 kernel averages a
range: sum with compensation while the sum fits, then switch to a running mean
once the next add would overflow.

The switch keeps the answer finite where the sum would not be: two histograms
whose scalar sum is `1e308` each average to `1e308`, not to `Inf`. It starts the
mean from the sum accumulated so far divided by the samples that produced it,
and weights every later sample by the fraction of samples it represents, so the
compensation is scaled by the same weight as the value it belongs to.

The complete answer divides both the sum and its compensation by the sample
count, or answers the mean with its compensation. `start_timestamp` is cleared
and the payload compacted, as every other derived histogram is.

No operation refuses a histogram sample any more, so the refusal variant is
gone rather than left unused: what a fold accepts is decided by its arithmetic,
not by the sample's type.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): merge native histogram state across a split

A split aggregate exports its partial state and folds it somewhere else, so a
histogram fold has to travel. The binding now says which layout a node carries:
`aggregate_state_binding(kind, field_kind)` picks the histogram family for a
histogram input and the numeric family otherwise, and `BoundAggregate` records
the kind its direct input answers with, decided where the node is bound.

The partial half exports the live histogram, the compensation that carries its
bucket layout, and the evidence the merge needs; a sum cannot claim a mean mode
because its kind has no such component. The final half merges two partials the
way the v3.15.0 kernel merges contributions: two compensated adds for a sum, and
a mean path that converts each side, turns its count into a weight, and scales
before adding so that a pair of `1e308` histograms still averages to `1e308`. A
sum/sum pair first runs the same trial the complete fold runs, and only keeps it
when it does not overflow.

An unreconcilable layout poisons the point in either partial: the merge keeps it
poisoned and never revives it, and the answer drops that point. Evidence is
merged bit by bit, including from a poisoned side, so which annotation fires
does not depend on the merge order.

The node that finishes the answer is the only one that reports: `Complete` and
`Final` speak after their output is queued, `Partial` stays silent because its
evidence travels in the state. A poisoned point suppresses its reconciliation
info, which is what makes the message set order-independent.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): admit a native `sum` and `avg` over histograms

A payload-folding aggregate is now admitted exactly when it binds histogram
state: `sum` and `avg` over a native histogram fold and merge their own state,
so the gate that refused every such input now asks what the node will build
rather than what its input carries. Their answer column is the canonical
histogram struct; `count`, `group`, `min`, `max`, `stddev` and `stdvar` keep
answering floats whatever they read.

The split may carry a histogram answer when the cut aggregate exports histogram
state. That is also what makes the state schema's value column count follow the
kind, which a histogram split otherwise failed against.

A histogram fold the split declines is not segmented. Segmentation answers a
group by folding each segment and appending the pieces, so a point whose
contributions are spread over partitions would be folded once per segment - a
sum would merge histograms that never met, an average would average averages.
The whole statement therefore keeps the plan it had, which also covers an
aggregate above an aggregate and a whole-vector ranking above a fold: neither
can cross the split boundary, and cutting them would answer a piece no segment
owns.

Multi-input histograms and a histogram behind a canonical merge keep their
refusals: the state crossing a canonical boundary has no execution coverage.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): read a native histogram sum and average over HTTP

Three cases write native histograms through Remote Write 2.0 and read the
answer back through the Prometheus query API. Each one asserts the native route
really executed: the observer records the split's two fused producers, and the
one publishing a histogram column is the final consumer, so a statement that
fell back to the existing plan cannot satisfy the test.

`sum` and `avg` answer the values the v3.15.0 kernels produce, and both are
compared against the same query without the native hint. That comparison uses a
relative tolerance for `avg` and an exact one for `sum`, because the existing
plan averages by weighting each sample (`12 * (1/3)` is one ulp below `12 / 3`)
while the native fold divides at the end - the difference this change documents
rather than hides.

The partitioned case flushes after every write so the scan really splits, then
asserts the partial producer reported output in more than one partition and that
the same query at parallelism one answers the same JSON.

Two layouts in one metric remove the group and report it: the answer is empty
and the response's warnings name the incompatible layouts, asserted after the
native nodes are known to have run.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): decide a reset collision per point and pin the avg merge weights

The contradictory-reset warning was decided from the two evidence bits OR-ed across every
group and every point, so sum by (job) (h) with CounterReset only in job a and
NotCounterReset only in job b warned, as did one series crossing its reset over a range.
Prometheus checks counterResetSeen && notCounterResetSeen per group per step, so the bit is
now decided per accumulator and only the decision is OR-ed across the answer. The state
layout is untouched: it still carries the two raw bits, and the collision is settled when the
answer is.

Nothing caught a swapped pair of avg merge weights: the existing overflowing-pair test gives
both partitions one sample and the same mean, so both weightings answer 1e308. The new case
merges a two-sample partition that switched to a mean with a six-sample one that still sums,
weights 2/8 and 6/8, and asserts 7 * 2^1019; swapping the weights answers 13 * 2^1019 and
fails.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): merge histogram partials through the accumulator, not a row encoder

The eight histogram merge tests drove their partials through a test-side encoder that writes
numeric state rows, so they panicked the moment a state carried a histogram payload. That
encoder never produced a histogram row for anyone: the merge they want to check is the one
HistogramAccumulator::merge_state performs, and the row path is the executor's.

They now merge the payloads directly -- from_payload for the first, merge_state for the rest
-- and read the answer through finished_histogram and the evidence through the production
summary. Every assertion keeps its value and strength: the unequal-count case still asserts
7 * 2^1019, so swapping the merge weights answers 13 * 2^1019 and fails, and the evidence
case still fails when the merge order decides the result.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): drive histogram state rows through merge_row

The cross-partition merge reads each partition's state frame as a StateRow, and
merge_histogram_row is the code that decodes a histogram payload out of one. No test reached
it: the numeric row encoder panics on a histogram, and the histogram merge cases go through
the accumulator instead, so a bug in that decode -- a swapped component, a poison bit lost, a
field index off by one -- would have left every test green.

These construct StateFieldsBatch rows directly and merge them. Sum and the average's mean
path assert the same bits the accumulator answers for the same payloads, so the two paths
cannot drift; a poisoned row is asserted to poison the point; a row missing a component and a
row whose compensation carries another bucket layout are asserted to be refused rather than
answer something. Dropping the poison bit, not decoding the mean mode, and reversing the
field index each fail one of them.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): bind an aggregate's state from the statement's own field kind

The rebase replayed the compiler call with a literal Some(FieldKind::Float) for the
statement's source kind, so every aggregate bound float state: sum over a native-histogram
column asked the gate whether it folds a payload, answered yes, and the statement fell back
to the legacy plan with prom_native_histogram_agg_sum in it.

The call now passes statement_kind, the kind the prepared scans report for the statement's
fields. The duplicate guard on the segmentation path -- which refused every histogram fold
rather than only the ones the split declined -- is removed, and the mutation scripts the
earlier commits picked up are dropped.

Signed-off-by: discord9 <discord9@163.com>

* test(http): register the three native histogram aggregate cases

Splicing the range test and this branch's cases together during the rebase kept the function
bodies but dropped their http_tests! entries, so the three cases existed without ever running.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): identify the histogram column by type, not by name

Native result columns are named after their expressions now, so the three admission tests
that looked for a column called greptime_native_histogram found none in a plan that answered
exactly the histogram they meant to check. They look for the histogram value type instead,
which is what the assertions were about.

Signed-off-by: discord9 <discord9@163.com>

* test(http): find the histogram answer by type and allow an empty scan partition

`answer_consumer` looked for a column named `greptime_native_histogram`, which the answer is no
longer called now that result columns take their expression's name (#10231). It reads the histogram
value type instead, and only the schema's own fields: a state frame keeps its histogram inside
`fields`, so a top-level type check still finds the consumer and not the producer.

The partition case also required every scan partition to hold rows. Routing hashes each series'
label set, so eight series over four partitions can leave one empty; #10469 stopped asking for
non-empty partitions elsewhere and `test_remote_write_v2_histogram_native_range_function` already
reads the same fixture this way. The producer still has to run on more than one partition, and the
answer still has to count every series once.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): carry the histogram fold onto #10433's bound aggregate

Rebasing onto #10433 left three things behind. `BoundAggregate` gained
`field_kind`, which the quantile tests construct without; the value is `Float`
because those fixtures hand the fold float samples, and the one call to
`aggregate_state_binding` now names the kind it asks about.

`native_composed_binary_admits_complex_operands` also expected
`hist * on(tag_0) sum by (tag_0) (hist)` to fall back, for a reason this branch
made false: it said `sum` has no histogram state to fold into. It has one now,
so the pair composes. `histogram * histogram` is not an error in Prometheus -
only `+` and `-` apply and any other operator removes the element with an info
annotation - so the composed pair is admitted and keeps the legacy float layout
for the answer no sample can fill. The test asserts that shape instead, and the
refusal it still checks (`sum(sum(hist))`) is named for what refuses it: the
split cannot carry an aggregate above a split aggregate.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): carry the histogram fold onto #10202's plan-line display

`BoundAggregate.field_kind` again: the display test's `sum(rate(...))` builds one without
the field, and its input is a histogram, so the field says so.

`native_fused_quantile_is_never_folded_per_partition` looked for the statement's text in the
display to show the fold is still named by the quantile it wrote. #10202 renders the tree as plan
lines instead, so the quantile is named by its own line - `Aggregate: quantile 0.5` - and that is
what the assertion reads now.

Signed-off-by: discord9 <discord9@163.com>

* style(http): keep the answer-column doc off a list item

The first line of the histogram sentence ended on a hyphen, so the next line began with one and
clippy read it as an unindented list item.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-17 18:29:58 +08:00
..