mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-21 02:04:02 +00:00
* feat(promql): carry a native histogram in aggregate state `sum` and `avg` over native histograms need a partial state whose payload is variable width, which the numeric state family cannot express. Add two kinds, `histogram_sum` and `histogram_avg`, whose components are a live histogram, a layout-carrying compensation, and the evidence a merge records. A position's payload is one `StatePayload`, either the numeric components a float kind carries or a boxed histogram state, so absence stays a single `None` and a field never holds both families. The numeric encoding is byte for byte what it was, and a histogram component nests the same struct a histogram sample frame already uses. The compensation must carry the live value's bucket layout: the kernel indexes it by the value's positions, so a placeholder without bucket indices would be an out-of-bounds read rather than a loss of precision. Both the producer and the decoder check the schema, the zero threshold, the custom bounds and the expanded logical bucket indices, which accepts two spellings of the same buckets and refuses two different layouts. State accounting charges the payload's real width instead of a fixed slot, so a wide bucket layout is paid for. `HistogramSum` cannot declare `incremental_mean`, which is checked structurally: a sum has no mean mode. No admission, split or runtime wiring yet; nothing produces these states until the aggregate kernels land. Signed-off-by: discord9 <discord9@163.com> * feat(promql): fold native histograms into a `sum` aggregate `sum` over a native histogram now folds in the native aggregate: the first sample at a grid position establishes the accumulator from a layout-carrying zero compensation, and every later one is added with a compensated add whose receiver is the live value. The evidence the final answer needs is recorded as it appears: set when a merge reconciled custom bucket bounds, and both reset-hint bits when samples reported a reset and reported that none happened. Two layouts that cannot be added - exponential and custom buckets in one position - poison the point, which the final answer drops instead of answering a partial sum. A numeric overflow is not a fault and does not poison anything: `sum` answers `Inf` when its cumulative sum saturates, exactly as the kernel's own sum does. `avg` keeps refusing a histogram: its mean mode needs the merge rules, which are not part of this change. A group that kept histogram payloads now refuses to export a partial numeric state rather than exporting an empty one. Signed-off-by: discord9 <discord9@163.com> * feat(promql): average native histograms in the native aggregate `avg` over a native histogram folds the same way the v3.15.0 kernel averages a range: sum with compensation while the sum fits, then switch to a running mean once the next add would overflow. The switch keeps the answer finite where the sum would not be: two histograms whose scalar sum is `1e308` each average to `1e308`, not to `Inf`. It starts the mean from the sum accumulated so far divided by the samples that produced it, and weights every later sample by the fraction of samples it represents, so the compensation is scaled by the same weight as the value it belongs to. The complete answer divides both the sum and its compensation by the sample count, or answers the mean with its compensation. `start_timestamp` is cleared and the payload compacted, as every other derived histogram is. No operation refuses a histogram sample any more, so the refusal variant is gone rather than left unused: what a fold accepts is decided by its arithmetic, not by the sample's type. Signed-off-by: discord9 <discord9@163.com> * feat(promql): merge native histogram state across a split A split aggregate exports its partial state and folds it somewhere else, so a histogram fold has to travel. The binding now says which layout a node carries: `aggregate_state_binding(kind, field_kind)` picks the histogram family for a histogram input and the numeric family otherwise, and `BoundAggregate` records the kind its direct input answers with, decided where the node is bound. The partial half exports the live histogram, the compensation that carries its bucket layout, and the evidence the merge needs; a sum cannot claim a mean mode because its kind has no such component. The final half merges two partials the way the v3.15.0 kernel merges contributions: two compensated adds for a sum, and a mean path that converts each side, turns its count into a weight, and scales before adding so that a pair of `1e308` histograms still averages to `1e308`. A sum/sum pair first runs the same trial the complete fold runs, and only keeps it when it does not overflow. An unreconcilable layout poisons the point in either partial: the merge keeps it poisoned and never revives it, and the answer drops that point. Evidence is merged bit by bit, including from a poisoned side, so which annotation fires does not depend on the merge order. The node that finishes the answer is the only one that reports: `Complete` and `Final` speak after their output is queued, `Partial` stays silent because its evidence travels in the state. A poisoned point suppresses its reconciliation info, which is what makes the message set order-independent. Signed-off-by: discord9 <discord9@163.com> * feat(promql): admit a native `sum` and `avg` over histograms A payload-folding aggregate is now admitted exactly when it binds histogram state: `sum` and `avg` over a native histogram fold and merge their own state, so the gate that refused every such input now asks what the node will build rather than what its input carries. Their answer column is the canonical histogram struct; `count`, `group`, `min`, `max`, `stddev` and `stdvar` keep answering floats whatever they read. The split may carry a histogram answer when the cut aggregate exports histogram state. That is also what makes the state schema's value column count follow the kind, which a histogram split otherwise failed against. A histogram fold the split declines is not segmented. Segmentation answers a group by folding each segment and appending the pieces, so a point whose contributions are spread over partitions would be folded once per segment - a sum would merge histograms that never met, an average would average averages. The whole statement therefore keeps the plan it had, which also covers an aggregate above an aggregate and a whole-vector ranking above a fold: neither can cross the split boundary, and cutting them would answer a piece no segment owns. Multi-input histograms and a histogram behind a canonical merge keep their refusals: the state crossing a canonical boundary has no execution coverage. Signed-off-by: discord9 <discord9@163.com> * test(promql): read a native histogram sum and average over HTTP Three cases write native histograms through Remote Write 2.0 and read the answer back through the Prometheus query API. Each one asserts the native route really executed: the observer records the split's two fused producers, and the one publishing a histogram column is the final consumer, so a statement that fell back to the existing plan cannot satisfy the test. `sum` and `avg` answer the values the v3.15.0 kernels produce, and both are compared against the same query without the native hint. That comparison uses a relative tolerance for `avg` and an exact one for `sum`, because the existing plan averages by weighting each sample (`12 * (1/3)` is one ulp below `12 / 3`) while the native fold divides at the end - the difference this change documents rather than hides. The partitioned case flushes after every write so the scan really splits, then asserts the partial producer reported output in more than one partition and that the same query at parallelism one answers the same JSON. Two layouts in one metric remove the group and report it: the answer is empty and the response's warnings name the incompatible layouts, asserted after the native nodes are known to have run. Signed-off-by: discord9 <discord9@163.com> * fix(promql): decide a reset collision per point and pin the avg merge weights The contradictory-reset warning was decided from the two evidence bits OR-ed across every group and every point, so sum by (job) (h) with CounterReset only in job a and NotCounterReset only in job b warned, as did one series crossing its reset over a range. Prometheus checks counterResetSeen && notCounterResetSeen per group per step, so the bit is now decided per accumulator and only the decision is OR-ed across the answer. The state layout is untouched: it still carries the two raw bits, and the collision is settled when the answer is. Nothing caught a swapped pair of avg merge weights: the existing overflowing-pair test gives both partitions one sample and the same mean, so both weightings answer 1e308. The new case merges a two-sample partition that switched to a mean with a six-sample one that still sums, weights 2/8 and 6/8, and asserts 7 * 2^1019; swapping the weights answers 13 * 2^1019 and fails. Signed-off-by: discord9 <discord9@163.com> * test(promql): merge histogram partials through the accumulator, not a row encoder The eight histogram merge tests drove their partials through a test-side encoder that writes numeric state rows, so they panicked the moment a state carried a histogram payload. That encoder never produced a histogram row for anyone: the merge they want to check is the one HistogramAccumulator::merge_state performs, and the row path is the executor's. They now merge the payloads directly -- from_payload for the first, merge_state for the rest -- and read the answer through finished_histogram and the evidence through the production summary. Every assertion keeps its value and strength: the unequal-count case still asserts 7 * 2^1019, so swapping the merge weights answers 13 * 2^1019 and fails, and the evidence case still fails when the merge order decides the result. Signed-off-by: discord9 <discord9@163.com> * test(promql): drive histogram state rows through merge_row The cross-partition merge reads each partition's state frame as a StateRow, and merge_histogram_row is the code that decodes a histogram payload out of one. No test reached it: the numeric row encoder panics on a histogram, and the histogram merge cases go through the accumulator instead, so a bug in that decode -- a swapped component, a poison bit lost, a field index off by one -- would have left every test green. These construct StateFieldsBatch rows directly and merge them. Sum and the average's mean path assert the same bits the accumulator answers for the same payloads, so the two paths cannot drift; a poisoned row is asserted to poison the point; a row missing a component and a row whose compensation carries another bucket layout are asserted to be refused rather than answer something. Dropping the poison bit, not decoding the mean mode, and reversing the field index each fail one of them. Signed-off-by: discord9 <discord9@163.com> * fix(promql): bind an aggregate's state from the statement's own field kind The rebase replayed the compiler call with a literal Some(FieldKind::Float) for the statement's source kind, so every aggregate bound float state: sum over a native-histogram column asked the gate whether it folds a payload, answered yes, and the statement fell back to the legacy plan with prom_native_histogram_agg_sum in it. The call now passes statement_kind, the kind the prepared scans report for the statement's fields. The duplicate guard on the segmentation path -- which refused every histogram fold rather than only the ones the split declined -- is removed, and the mutation scripts the earlier commits picked up are dropped. Signed-off-by: discord9 <discord9@163.com> * test(http): register the three native histogram aggregate cases Splicing the range test and this branch's cases together during the rebase kept the function bodies but dropped their http_tests! entries, so the three cases existed without ever running. Signed-off-by: discord9 <discord9@163.com> * test(promql): identify the histogram column by type, not by name Native result columns are named after their expressions now, so the three admission tests that looked for a column called greptime_native_histogram found none in a plan that answered exactly the histogram they meant to check. They look for the histogram value type instead, which is what the assertions were about. Signed-off-by: discord9 <discord9@163.com> * test(http): find the histogram answer by type and allow an empty scan partition `answer_consumer` looked for a column named `greptime_native_histogram`, which the answer is no longer called now that result columns take their expression's name (#10231). It reads the histogram value type instead, and only the schema's own fields: a state frame keeps its histogram inside `fields`, so a top-level type check still finds the consumer and not the producer. The partition case also required every scan partition to hold rows. Routing hashes each series' label set, so eight series over four partitions can leave one empty; #10469 stopped asking for non-empty partitions elsewhere and `test_remote_write_v2_histogram_native_range_function` already reads the same fixture this way. The producer still has to run on more than one partition, and the answer still has to count every series once. Signed-off-by: discord9 <discord9@163.com> * fix(promql): carry the histogram fold onto #10433's bound aggregate Rebasing onto #10433 left three things behind. `BoundAggregate` gained `field_kind`, which the quantile tests construct without; the value is `Float` because those fixtures hand the fold float samples, and the one call to `aggregate_state_binding` now names the kind it asks about. `native_composed_binary_admits_complex_operands` also expected `hist * on(tag_0) sum by (tag_0) (hist)` to fall back, for a reason this branch made false: it said `sum` has no histogram state to fold into. It has one now, so the pair composes. `histogram * histogram` is not an error in Prometheus - only `+` and `-` apply and any other operator removes the element with an info annotation - so the composed pair is admitted and keeps the legacy float layout for the answer no sample can fill. The test asserts that shape instead, and the refusal it still checks (`sum(sum(hist))`) is named for what refuses it: the split cannot carry an aggregate above a split aggregate. Signed-off-by: discord9 <discord9@163.com> * fix(promql): carry the histogram fold onto #10202's plan-line display `BoundAggregate.field_kind` again: the display test's `sum(rate(...))` builds one without the field, and its input is a histogram, so the field says so. `native_fused_quantile_is_never_folded_per_partition` looked for the statement's text in the display to show the fold is still named by the quantile it wrote. #10202 renders the tree as plan lines instead, so the quantile is named by its own line - `Aggregate: quantile 0.5` - and that is what the assertion reads now. Signed-off-by: discord9 <discord9@163.com> * style(http): keep the answer-column doc off a list item The first line of the histogram sentence ended on a hyphen, so the next line began with one and clippy read it as an unindented list item. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com>