Commit Graph
222 Commits
Author SHA1 Message Date
Ning Sun 7f0f592b6b chore: update rust toolchain to 1.98.1 (#9429) 2026-10-08 09:00:10 +00:00
Yingwen 97913b4b3a fix: coalesce equal histogram quantile bucket boundaries (#9451)
* fix: coalesce equal histogram quantile bucket boundaries

Merge numerically equivalent bucket boundaries before monotonicity repair, preserving the allocation-free path for unique boundaries. Share coalescing with histogram_fraction and cover duplicate boundaries with unit and SQL regressions.

Fixes #9443

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: keep duplicate infinity buckets in one histogram group

Confirm classic histogram completion at a series/timestamp boundary or EOF instead of the first positive infinity. Retain lookahead across batches and reject incomplete optimistic groups so equivalent infinity labels reach numeric coalescing together.

Add stream and SQL regressions for duplicate infinity labels, including established bucket counts and EOF. Correct older fixtures to use distinct keys for distinct histogram groups.

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: avoid buffering histogram groups with invalid trailing bounds

Discover candidate bucket counts at the first group boundary and let optimistic validation fall back to safe mode for malformed bounds. Document discovery and cover trailing invalid and null labels with a focused streaming regression.

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-10-07 07:18:25 +00:00
dennis zhuang 78a7b93292 fix(promql): align timestamp(), label_join and label_replace with Prometheus semantics (#9385)
* fix(promql): align timestamp() and label_join with Prometheus semantics

- timestamp() over a selector reports the selected sample's timestamp, without adding the offset.
- timestamp() over any other expression reports the evaluation time instead of the input value.
- label_join that overwrites an existing label rejects duplicate label sets at runtime.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): align label_replace and label_join edge cases with Prometheus

- label_replace leaves non-matching series unchanged instead of copying the source value into the destination label.
- label_replace may overwrite an existing label; duplicate label sets are rejected at runtime like label_join.
- label_join with no source labels removes the destination label, and an empty source label name is rejected.
- Empty label values produced by these functions are NULL, the same as an absent label.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): join absent labels as empty strings in label_join

concat_ws skips NULL arguments together with their separator, so a label that is absent (NULL) dropped its separator, e.g. label_join(label_join(vector(1), "a", "", "missing"), "b", "-", "a", "a") lost b="-". Read every source label as coalesce(label, '') like label_replace does.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): skip rows without a sample in timestamp()

timestamp() replaced the input fields with the timestamp before the empty-value filter ran, so a row whose fields were all NULL got a timestamp, e.g. timestamp(-m) for a NULL sample. Filter such rows before the projection, for both the selector and the evaluation-time paths.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-30 01:42:11 +00:00
dennis zhuang 218000e21b fix(promql): resolve dotted column names as unqualified columns (#9391)
* fix(promql): resolve dotted column names as unqualified columns

col(), From<&str>/From<String> for Column and string join keys go through
Column::from_qualified_name, which splits `service.name` into relation
`service` and column `name` and lowercases unquoted identifiers. Build
PromQL column references with Column::from_name / ident() instead.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(servers): resolve remote read matcher labels as unqualified columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover same-name columns differing in case and without()

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-29 08:13:28 +00:00
discord9 28e415a103 feat(query): implement PromQL @ modifier on vector and matrix selectors (#9224)
* feat(query): implement PromQL @ modifier on vector and matrix selectors

The planner previously dropped the `at` field of selectors (`at: _`),
so `some_metric @ 300` silently returned step-following values instead
of the samples anchored at the fixed timestamp.

Anchoring follows Prometheus's `setOffsetForAtModifier` + `refetch`
semantics: resolve the anchor (`@ <ts>`, `@ start()`, `@ end()`), apply
`offset` to the anchor, rewrite the selector offset to
`eval_start - anchor`, scan only the anchored window, then report the
same window at every evaluation step via a grid-wide-replay
`InstantManipulate`. `@ start()` / `@ end()` resolve against the
statement's evaluation range (`stmt_start` / `stmt_end`), which subquery
planning does not rewrite.

Timestamps before the Unix epoch are accepted; unrepresentable ones are
rejected with `AtModifierTimestampOutOfRange` instead of wrapping.
Selectors without `@` are planned exactly as before.

Report: .e-agent/greptimedb_promql_compatibility_report_2026-09-16.md P0-2
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): center predict_linear on the evaluation timestamp

predict_linear_impl used the window's last sample time as the evaluation
timestamp, and the UDF took only (ts_range, value_range, t) with no channel
for the current instant. After a range selector is folded for @ (or with an
offset) the same window is replayed at every step, so the prediction stayed
constant at the anchor's answer; even a plain window ended before the step
produced a stale value.

Give prom_predict_linear a 4th argument carrying the step's evaluation
instant (ms timestamp), derived in the planner from the row's time index
plus the offset the window was folded with (at_offset for @, offset_ms
otherwise, recorded on PromPlannerContext). The regression is centered on
that instant, matching Prometheus' use of enh.Ts. The mixed
float/native-histogram path forwards the same argument.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(query): tighten @ modifier planner and predict_linear

- range_fold_offset is always a concrete millisecond offset, not an
  Option; the single-step replay helper no longer wraps an infallible
  plan in Result.
- predict_linear's eval timestamp is always cast to Timestamp(ms) at the
  call site, so the UDF drops its dead Int64 branch and the extra
  func_name argument.
- Drop two redundant comparison cases from the at_modifier sqlness test
  and fix the lookback window comment to the half-open (0s, 300s].

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): accept parse-time rejection of @ on Windows

`@ 1e16` is 10^19 milliseconds, beyond i64::MAX. A Unix SystemTime
holds it and the planner rejects the anchor it cannot represent, but a
Windows SystemTime tops out near 1.8e12 seconds, so the parser's
checked_add fails first and the same literal is rejected while parsing.
Accept either rejection path so the test passes on both platforms.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): avoid replaying label_join over rewritten series keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): narrow anchored range call promotion

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): address @ modifier review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): satisfy super import format check

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs(query): address at modifier review follow-ups

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-28 09:38:12 +00:00
dennis zhuang 88197f4019 fix(promql): derive vector matching result labels and reject ambiguous matchings (#9306)
* fix(promql): derive vector matching result labels and reject ambiguous matchings

A vector-vector binary operation projected one operand's whole tag set and
inner-joined without any cardinality check, so `on()`/`ignoring()` did not
reduce the result labels, `group_left`/`group_right` changed nothing, and a
non-unique match group produced a cross product that PromQL cannot represent.

Result labels now follow Prometheus `resultMetric`: `on(...)` keeps the
matching labels, `ignoring(...)` drops them, and a group modifier keeps the
many side's labels plus the `group_x(...)` labels taken from the one side.
A label the one side does not carry is deleted from the result. The reduced
label set no longer identifies the operand series, so `__tsid` is dropped
from the context on this path.

Cardinality is enforced with a `count(1) OVER (PARTITION BY match keys, ts)`
window and a scalar UDF that fails the query on a repeated group: on the one
side before the join, and on the result labels after it, matching where
Prometheus raises each of its three errors. Series are unique by their whole
tag set, so the window is only planted when the match keys drop a tag; plain
arithmetic and `on(<all tags>)` plan exactly as before.

Closes #9207, closes #9208, closes #9209.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* perf(promql): keep __tsid when the result labels are an operand's whole tag set

Deriving the result labels dropped `__tsid` from the context unconditionally,
so an enclosing operation fell back to joining on the tag columns even where
the column still identified the result series.

Keep it when every result label comes from one operand and covers that
operand's whole tag set: no other operand value reaches the labels, and the
matching gives each of its rows a single partner, so its `__tsid` is still one
per result series. That is the common `on(<all tags>)` and bare `group_left`
shape; a matching that actually drops a tag still clears it.

The column is re-qualified as the result's own, which is how the enclosing
expression and the context look it up.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep the match group count column unambiguous

The cardinality check aliased its row count to a fixed `__promql_match_group_count`.
An operand carrying a label of that name made the window output two fields with
the same name, and planning failed with "Schema contains qualified field name
collide_right.__promql_match_group_count and unqualified field name
__promql_match_group_count which would be ambiguous".

Pick a name the operand does not already have, the way the `or` operator
allocates its match key columns.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): cover a match group spread over several regions

The cardinality check runs above the merge of the region scans, so it counts a
match group globally. Nothing pinned that: every table in these cases holds a
single region, and a check evaluated per region would pass them all.

Partition the operand on a column outside the match keys, which puts the two
series of one match group in different regions, and assert both the pre-join
and the post-join check still reject it. The case runs in the distributed
environment too, where the regions sit on different datanodes.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): group an outer aggregate on the labels the operand kept

`by`/`without` planning topped up a missing grouping column by walking down to
the table scan and re-projecting it. That is right for a column the scan pruned
for efficiency, but the labels a matching modifier deletes are also absent from
the operand's output, and they were restored the same way:

  sum without(host) (a / on(host) b)

`on(host)` leaves the operand with `host` alone, so the sum covers everything
and Prometheus answers `{} 10`. Instead `device` came back from the scan under
`a` and split the result into `{device="d1"} 5` and `{device="d2"} 5`. Same for
`sum by(device)` of that operand, which has no `device` to group on at all.

Take the grouping labels from the operand's own label set rather than from the
row keys of the scan beneath it. A label pruned from the plan is still in that
set and still gets restored; a label the operand dropped is not.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(promql): drop the unreachable aggregation tag top-up

`by`/`without` planning could restore a grouping column that the plan no longer
carried by rewriting the scan underneath it. Once the grouping labels come from
the operand's own label set, there is nothing left for it to restore: a scan
projects every label of `ctx.tag_columns` (`scan_tag_columns` only ever adds
matcher columns to that set), so a label in the set is always in the schema.

Stubbing the rewriter to a no-op passed the whole sqlness suite, in both the
standalone and the distributed environment.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): drop cases that guarded the removed tag top-up

Three plain selector aggregates were there to show that restoring a pruned
grouping column still worked. With the restore gone they only repeat what the
aggregate cases already cover. Also fix a comment that still said the metric
engine scan prunes tag columns: it projects every label of the operand.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): note the duplicate a propagated matcher hides

A matcher copied onto the one-side operand removes groups without a partner
before the cardinality check sees them, so a duplicate in such a group is not
reported. Prometheus checks every group of the one side and fails the query.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-23 11:08:18 +00:00
discord9 e9beb62eef perf(promql): avoid concatenating constant series tags (#9108)
* perf(promql): experiment with constant-tag series concat

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): verify logical constant-tag concat equivalence

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): qualify constant-tag series concat

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover fragmented millisecond series concat

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): compact constant dictionary tags at construction

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(promql): construct constant string dictionaries directly

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): benchmark ordinary TQL queries for constant tags

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): address constant-tag review and cardinality coverage

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): scope concat optimization to string dictionaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): move high-cardinality constant-tag cases to the heavy set

The 10k and 100k constant-tag direct-SST cases repeatedly kill the
self-hosted query-regression runner (lost communication during the run),
while the default-cardinality case passes. Move them out of the default
'all' set into the heavy set so they only run on demand (case=heavy or
the heavy-regression label), and qualify them locally instead.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-17 10:08:54 +00:00
discord9andDennis Zhuang 528ceb7733 perf(promql): reuse sliding min and max candidates (#9099)
* perf(promql): reuse sliding min and max candidates

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): simplify extrema benchmark parameters

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): record baseline sliding extrema SQL results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(promql): rescan windows that barely overlap

Reusing candidates loses to a plain scan when consecutive windows overlap
little: the deque bookkeeping then costs more than the rescan it replaces.
A local Criterion run on 4096 samples at width 240 / step 240 measured
10.79 -> 22.14 us for min and 12.83 -> 19.85 us for max.

Pick the evaluator once per batch from the first two windows. RangeManipulate
emits one window length and one step per batch, so that sample decides for all
of them, and both evaluators return identical bits, so a wrong pick costs time
only. Batches that do not qualify fold each window on its own.

Move the incremental state into SlidingExtrema so tests can drive it directly:
the exhaustive four-sample differential test cannot reach it through a UDF
call, because such a batch never qualifies for reuse.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* fix(promql): select the extrema evaluator from batch averages

Reading the window shape off the first two windows misreads the batch.
RangeManipulate starts a series at max(query start, first aligned sample),
so a series that begins inside the query range gets a first window covering
roughly one step, and a window covering no sample at all is emitted as
(0, 0). Either one closed the gate for the whole batch, including the
one-hour window at a 15s step that candidate reuse was written for.

Compare the batch averages instead: at least 32 samples per window, and a
step advancing at most a quarter of that. Uniform batches select exactly as
before, so the thresholds keep the meaning they were measured with.

The 32-sample rule had also moved most of the benchmark and query-regression
shapes onto the rescan, including the case built to measure reset and
rebuild. Widen those windows to 40 samples, add a step at the selection
boundary, and add an end-to-end case with 40-sample windows advancing 5.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* fix(promql): ignore empty windows when measuring batch advance

The advance was read from the first and last window offsets, but a window
covering no sample is emitted as (0, 0). A query whose last evaluation lands
exactly one window past the last sample ends on such a window, and its zero
offset made a batch of disjoint windows look like one that never moved, which
selected the evaluator built for overlap. Results stayed correct; the cost was
deque bookkeeping on the shape the scan fallback exists for.

Take the offset span over the windows that cover a sample. Empty windows stay
in the window count, where they only make both conditions stricter.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
Co-authored-by: Dennis Zhuang <xzhuang@greptime.com>
2026-09-17 03:18:21 +00:00
discord9 5e57b00a95 fix(promql): skip empty batches in SeriesDivide stream (#9178)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-16 07:37:15 +00:00
discord9andNing Sun 94d7e2c7fc feat!: upgrade DataFusion to 55 (#8555)
* feat!: upgrade DataFusion dependencies to 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: migrate DataFusion 55 APIs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve table function planning behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve PostgreSQL query compatibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve distributed execution plan behavior

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover DataFusion 55 behavior regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update DataFusion 55 SQLness expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DataFusion 55 test API migration

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address DataFusion 55 CI regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: address remaining DataFusion 55 regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: adapt latest base code to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: normalize environment-specific DataFusion 55 plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align final DataFusion 55 expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: isolate DataFusion 55 regression cases

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve empty result schema in timestamp widening

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: preserve JSON source column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* chore: use released DataFusion 55 integrations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: adapt latest execution plan mock to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: pin DataFusion recursive schema and date repairs

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): align dictionary temporality match keys

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: retain Greptime DataFusion fork behaviors on version 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore ordinary function error expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh distributed count compatibility plan

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): adapt last-row cast hint to DataFusion 55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh instant last-row empty results for Arrow 59

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* style: simplify DataFusion expression visitor imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore sorting and PostgreSQL column-order assertions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): restore primitive numeric coercion signatures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(function): share geo integer signature types

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover timestamp widening overflow boundaries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: fix decimal coercion regression imports

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(function): preserve scalar count_hash NULL state semantics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: simplify decimal clamp case type inference

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: retain historical count_hash wrapper result

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restore timestamp widening equality and IN pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream aggregate dynamic filter correctness fix

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: carry upstream null and predicate simplification fixes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore baseline JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: restore histogram JSON ordering expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh empty PromQL range result schemas

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: align native timestamp plan with DF55 decimal display

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh native timestamp SQLness results for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regenerate NULL sample empty result headers for DF55

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: use DF55 child replacement API in timestamp regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: expose pushed scan dynamic filters to DF55 producers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: encode string-backed PostgreSQL OID aliases in binary results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify REGPROC binary and text over PostgreSQL protocol

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: register real PostgreSQL catalogs in server fixtures

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: complete DF55 expression inventories for custom query plans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: correct RangeSelect expression fixture and column identities

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* ci: wait for Kafka WAL helper deployment rollout

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update custom storage empty result headers

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: require exact row counts in scan statistics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: suppress deprecated partition_statistics warning in test

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-09-15 11:42:38 +00:00
dennis zhuang 7152ca9264 fix(promql): skip NULL samples and fix counter extrapolation order (#9118)
* fix(promql): treat NULL field values as absent samples in range functions

Range functions read the value column through `Float64Array::values()`,
which returns the raw buffer and ignores the null bitmap. A NULL field
value means the series has no sample at that timestamp, so the padding
under a null slot (0.0 in practice) was counted as a real sample.

`rate`, `increase`, `delta`, `changes`, `resets`, `idelta`, `irate`,
`quantile_over_time` and `avg_over_time` now work on samples instead of
slots. `stddev_over_time` and `stdvar_over_time` used to panic on a NULL
slot, and tokio swallowed the panic so the query returned success with
that series missing.

Null handling is gated on `null_count() == 0` over the whole backing
array, checked once per batch, so tables without NULLs keep the existing
code path. `deriv` and `predict_linear` already had this guard through
`linear_regression_slices` and are untouched.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): keep other fields when one has no sample in the window

Review follow-up. Two things surfaced once range functions started
returning NULL for a window without samples.

The filter after a function call required every field column to be
non-NULL, so on a multi-field table one field with no samples in a
window would drop the other fields' results with it. It now keeps a row
when any field has a sample, which is the shape a selector already
emits. On a single field column the two predicates are identical.

`quantile_over_time` returned NaN rather than NULL for a window without
samples, so the row survived that filter. Prometheus returns an empty
vector there, so the emptiness check now sits in the range UDF; the
shared quantile kernel still yields NaN for an empty slice, matching
upstream's `quantile()` helper that `quantile_aggr` depends on.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(promql): correct two comments on the null handling

The planner one did not say why `preserve_any_value` is hardcoded at
that call site, which is the question a reader arrives with. The
`quantile_over_time` one described the empty-window behaviour while
sitting on the `has_nulls` line, and that behaviour had moved into
`window_quantile`.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(promql): clamp extrapolation before snapping a counter to zero

Prometheus clamps `durationToStart` to half an average interval once the
first sample is past the extrapolation threshold, and only then lets the
counter zero-snap shorten it further, so the snap can never lengthen the
leading extrapolation. Running the snap first let it rescue a duration
the clamp should have cut, and `rate` and `increase` over-extrapolated
to the left. For samples 1@0s and 2@1s in a 4s window at 1s, upstream
gives 0.375 and this returned 0.5.

`extrapolation_matches_prometheus_on_seeded_windows` carries a
line-by-line port of upstream `extrapolatedRate` as an oracle and diffs
it against the UDF over seeded windows, so the order stays pinned.
`factor` also picked up upstream's guard against a zero sampled
interval, which previously divided by zero.

Two sqlness results move, both verified against the upstream algorithm.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(promql): fold range_presence_null into null_samples

#9104 landed its own NULL-sample case whose data is the same series this
one already used: one host with interior NULLs, one host with nothing
but NULLs. Keeping both means two files asserting the same semantics on
the same rows.

The merged case keeps every query from both, so the presence functions
still cover the trailing-NULL window, the count of two, the empty
left-open window at t=7, and the all-NULL windows.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-09-14 06:33:36 +00:00
discord9 7f949f48c0 fix(promql): preserve native timestamps through sample selection (#9070)
* fix(promql): preserve native timestamps through sample selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): retain column indices in instant plan ordering

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): update native precision plan expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): preserve selector output column order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): verify preserved selector output order

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): refresh native timestamp explain expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: apply PromQL offsets without native timestamp overflow

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: cover negative PromQL offsets at native timestamp bounds

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* feat: allow native precision instant LastRow selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: discard unused bounds for empty range intersections

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: preserve native time bounds independently for LastRow

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: verify native LastRow predicates and overflow through SQL

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs: explain native PromQL selection and scan invariants

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): address timestamp helper and stream review feedback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor(promql): pass selector offsets explicitly from planner

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): retain explicit offset in payload overflow regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): record inner-offset subquery SQL results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-14 03:47:57 +00:00
discord9 555485c40e perf(promql): avoid per-window allocations in simple range functions (#9104)
* perf(promql): avoid per-window allocations in simple range functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* perf(promql): avoid copying smoothing window values

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover smoothing copy removal across window layouts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): skip null samples in simple range functions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-11 11:59:45 +00:00
dennis zhuang 0f625a7e92 fix(promql): correct counter reset accumulation in rate windows (#9089)
* fix(promql): correct counter reset accumulation in rate windows

`prom_rate` and `prom_increase` reused the previous window's counter-reset
correction when the next window slid forward by exactly one sample, adding
the entering reset and subtracting the leaving one. Running a sum through
addition and subtraction does not restore the earlier terms in f64: a large
reset absorbs the smaller ones that must survive it, and an expired infinity
leaves a NaN that no later window can clear. `prom_delta` shares the code but
is not a counter function, so it never took that path.

Index the reset positions of the value array once instead, and reduce each
window over the resets it contains, in sample order. The result is
bit-identical to scanning the window directly, so windows keep the direct
reduction when they request fewer sample pairs than the input has.

Also sweep the query step in the rate benchmarks: the cost of the reset
correction depends on how much the windows overlap, which no existing case
varied.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* test(promql): cover counter reset precision over adjacent rate windows

The unit tests build the range windows directly, so they do not show that a
plain PromQL range query produces the window layout that lost the correction.
This case does: with a query step equal to the sample interval, `increase`
over the second window returns 1.333 before the fix and 2.667 after it.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* perf(promql): advance the counter reset bounds instead of searching

Locating a window's resets with two binary searches costs more than the
reduction it replaces once a series resets often enough for the searches to
get deep: on a 20k-sample counter resetting every 37 samples, stepping the
windows by one sample was 2.7x slower than the previous code, against 1.2x
for a counter that never resets.

Windows normally advance, so walk the bounds forward from the previous
window and only search when they move back. The cost then no longer depends
on the reset density.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* perf(promql): cut the per-window cost of the counter reset index

Two costs the index added showed up on a one-sample query step, where the
removed fast path used to answer each window with two comparisons.

Cache the two reset positions that bound the active slice. A window that only
advanced and reached neither of them covers the same resets as the previous
one, so the common case is four integer comparisons and no lookup at all.

Stop summing the requested sample pairs once they exceed one pass over the
values. The sum only decides which side of that comparison the input falls on,
and a query with a short lookback and a long step settles it after a few
windows instead of after every key.

Together these take the one-sample step from 25-32% slower than the previous
code down to 6-11%, measured as before / after / before to bound drift. No
other step value regresses, and a ten-sample step stays about 88% faster.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* fix(promql): accumulate counter resets into the running result

`prom_rate` and `prom_increase` summed a window's counter-reset corrections
on their own and added that sum to `last - first`. Prometheus folds each
reset into the running result instead, and so did this code before #7880.
The two are not interchangeable in f64: over samples `[1e16, 1, 0, 1]` the
isolated sum rounds `1e16 + 1.0` back to `1e16`, which then cancels against
the first sample and reports no increase at all, where folding the resets in
one at a time keeps the 1.0.

Restore the original order. The reset index accumulates into the result the
same way, so it still matches a direct scan of the window bit for bit, but a
window's contribution can no longer be cached as a standalone value and is
re-added from its own difference each time. The bounds are still cached, so
a window that did not cross a reset skips the lookup, and one that holds no
resets returns without touching the index at all.

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

* test(promql): note which reset boundaries the stride of one walks

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>

---------

Signed-off-by: Dennis Zhuang <xzhuang@greptime.com>
2026-09-10 07:11:39 +00:00
discord9 da5cb1a190 perf(promql): push down last row for instant queries (#9034)
* perf(promql): push down last row for instant queries

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: guard instant last row correctness

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: update instant query explain results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: apply last row after source deduplication

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: scope post-merge last row selection

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(perf): cover instant PromQL last row

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(perf): sort generated SST rows before writing

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): cover instant last row selection in sqlness

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(promql): avoid last row hints for lossy timestamp casts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): preserve stale marker semantics across flushes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): decode dictionary labels in stale regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): avoid reserved column name in stale fixture

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(promql): exercise LastRow hints and filtered results

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: keep after-merge mode in LastRow selector

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: reject instant LastRow across residual filters

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: expect after-merge selector in instant vector guards

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* docs: explain instant LastRow filter eligibility

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix: restrict instant LastRow to safe selector nodes

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* refactor: show LastRow merge mode directly in diagnostics

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: refresh LastRow display in explain expectations

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-09-09 12:38:52 +00:00
shuiyisong b86da3d35f feat: support raw OTLP delta metrics (#8970)
* feat: support raw OTLP delta metrics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: fmt

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test(promql): update sqlness results for normalized label matching

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: derive temporality label from default column prefix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test(promql): add analyze coverage for delta temporality

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): scope label alignment to temporality marker

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: handle count-only histograms and vector broadcasts

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: exclude temporality marker from entity descriptions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: use a fixed label for OTLP aggregation temporality

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): preserve mixed-range semantics for raw delta

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-09-03 09:28:11 +00:00
shuiyisong 82444635f5 feat: support quantile and fraction queries on mixed histograms (#8874)
* fix: test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove try_build_float_literal

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: remove NonCommutative

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: NaN

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test(query): cover frontend-only histogram fold planning

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): ignore unparseable histogram bucket labels

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: typo

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-08-24 12:45:35 +00:00
shuiyisong 6493435bee feat(promql): support native histogram aggregations (#8848)
* feat(promql): support native histogram aggregations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): correct mixed native histogram aggregations

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): format mixed count_values labels consistently

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(promql): preserve reset hint warnings for incompatible histograms

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-08-12 07:50:54 +00:00
Ning Sun 3510ef7d4c feat!: update native histogram unsigned int types (#8824)
* feat(native-histogram): store counts and span lengths as signed integers

Native histograms are unreleased, so the on-disk integer payload columns
are switched from unsigned to signed types without backward-compat:

  - count_u64 / zero_count_u64: uint64 -> int64
  - positive_span_lengths / negative_span_lengths: list(uint32) -> list(int32)
  - Span.length (query-time model): u32 -> i32

The Prometheus remote-write v2 source carries these as uint64/uint32, so
the unsigned->signed conversion at the ingestion boundary is overflow
checked: an integer count >= 2^63 or a span length >= 2^31 is rejected
with an explicit error rather than silently wrapping to a negative value.
read_spans additionally rejects negative stored lengths to keep the
non-negative invariant sound for downstream `as usize` casts.

The UDAF accumulator's own observation counter (transient aggregation
state, not part of the persisted histogram value) is intentionally left
as uint64.

Signed-off-by: Ning Sun <sunning@greptime.com>

* refactor(native-histogram): rename count/zero_count fields to _i64

Now that the integer payload columns are stored as int64, rename the
field constants and persisted names to match:

  COUNT_U64_FIELD  ("count_u64")      -> COUNT_I64_FIELD  ("count_i64")
  ZERO_COUNT_U64_FIELD ("zero_count_u64") -> ZERO_COUNT_I64_FIELD ("zero_count_i64")

The local builder variables and the docs/JSON snapshot are updated to
match. No backward-compat (unreleased feature).

Signed-off-by: Ning Sun <sunning@greptime.com>

* test(native-histogram): refresh planner plan snapshot for signed types

The mixed native-histogram range test embeds the full histogram Struct
type in its expected plan string, which still carried the pre-rename
unsigned fields. Update the snapshot to match the signed schema:

  positive/negative_span_lengths: List(UInt32) -> List(Int32)
  count_u64/zero_count_u64: UInt64            -> count_i64/zero_count_i64: Int64

Signed-off-by: Ning Sun <sunning@greptime.com>

---------

Signed-off-by: Ning Sun <sunning@greptime.com>
2026-08-11 08:55:27 +00:00
shuiyisong 97cbf79fb6 feat(promql): support native histogram vector operators (#8798)
* feat(promql): support native histogram vector operators

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-08-11 08:01:06 +00:00
shuiyisong 4ce49ec33e feat(promql): support mixed sample ranges (#8784)
* feat(promql): support mixed sample ranges

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add comments

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: CR issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: return error than panic

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-08-10 09:06:20 +00:00
shuiyisong a28a0466b5 feat(promql): define native histogram semantics (#8758)
* feat(promql): define native histogram semantics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: change author of the rfc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update rfc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update rfc

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: update comments and tests

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fix issues

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: add test

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-08-06 06:59:59 +00:00
discord9 a7590f8174 perf(promql): avoid repeated scans in sliding range evaluation (#8646)
* perf(promql): use two pointers for sliding range boundaries

Replace the stale cursor heuristic in RangeManipulateStream::calculate_range
with monotonic left/right cursors. The old path rescanned each evaluation
window (O(E x samples-per-window)) and could lose valid samples after sparse
gaps or trailing empty windows. The two pointers keep strict monotonic
progress, reducing boundary generation to O(N + E) while preserving
(curr-range, curr] semantics, start/end shortening, and empty-window output.

Controlled release benchmarks (fixed CPU, ABBA):
- Public RangeManipulate wall time: ~28% faster at 1m/15s, ~66% at 5m/15s,
  ~96% at 1h/15s.
- Warmed distributed TQL ANALYZE 1h queries: ~17-21% faster end to end;
  shorter windows stayed within run-order noise.

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): specialize changes/resets with adaptive edge counting

The generic range_fn macro slices, downcasts, and rescans every overlapping
window for changes() and resets(). Replace the macro path for these two
functions with hand-written UDF wrappers backed by a shared private
edge-count kernel: direct raw-offset scans when requested edges are few,
otherwise one global u64 edge prefix so each window is answered by a prefix
difference.

Behavior is preserved bit-for-bit, including raw null-buffer values, NaN
semantics, signed zero, infinities, empty/singleton windows, independent
timestamp/value offsets, arbitrary window layouts, and exact DataFusion
error messages. The shared proc macro, planner, serializer, and other range
functions are untouched.

Controlled release benchmarks (fixed CPU, ABBA):
- Dense sliding windows (k=4/20/240): 91.7-95.6% less public UDF wall time.
- Low-coverage fallback (N=4096, 8 windows): 73.9-74.4% faster.
- Warmed distributed TQL ANALYZE 5m/1h changes/resets: 12.1-19.7% client
  and 12.0-20.9% server latency improvement; controls stayed within drift.

Signed-off-by: discord9 <discord9@163.com>

* ci(query-regression): include PromQL range boundary case in defaults

An audit of historical query-regression runs found zero range-query
coverage: all 208 PromQL ANALYZE samples were bare selectors, so range
evaluation could regress without CI noticing. Wire the
promql_range_boundary case (introduced in #8646) into DEFAULT_CASES so
label-triggered runs measure the range path. The case is cheap: a ~0.3s
synthetic fixture and about a minute of query execution per base/candidate
pass.

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): address sliding range review nits

Move test-only imports into their test modules and remove the unused
pre-specialization changes and resets helpers.

Signed-off-by: discord9 <discord9@163.com>

* style(promql): apply pinned rustfmt

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover sparse range results

Share the changes and resets test scaffolding while keeping their behavior
oracles independent. Add an end-to-end sqlness regression for sparse samples,
empty intermediate windows, and a valid trailing sample.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-08-03 06:53:21 +00:00
shuiyisong 08537b5eaf feat(promql): add native histogram functions (#8664)
* feat(promql): add native histogram functions

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: rebase main

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: cr issue

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-07-28 14:08:42 +00:00
discord9 054d4046d5 fix(promql): preserve query-aligned range tail (#8650)
Align the shortened trailing evaluation bound relative to the query start so non-epoch-aligned queries retain their final valid window.

Signed-off-by: discord9 <discord9@163.com>
2026-07-27 11:23:40 +00:00
Ruihang Xia 0479c05092 perf: preserve dictionary-encoded query labels (#8541)
* perf: preserve dictionary-encoded query labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(client): skip dictionary Flight batches

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): decode dictionary labels in HTTP output

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* test(mito2): support dictionary tags in series scans

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(servers): preserve dictionary child nulls in SQL

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(promql): compare dictionary tags by logical nulls

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): keep dictionary tags within query paths

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refactor(query): scope PK dictionary encoding to reads

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve pushdown for dictionary labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): handle dictionary labels in query operators

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): complete label type matching

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): align dictionary query schemas

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): decode incompatible OR labels

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix(query): preserve dictionary partition pruning

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: handle dictionary query edge cases

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix: import dictionary downcast macro

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-07-17 13:57:10 +00:00
discord9 a598690bf7 fix(promql): handle missing labels in or matching (#8504)
* fix(promql): handle missing labels in or matching

Signed-off-by: discord9 <discord9@163.com>

* test(promql): streamline or matching coverage

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): preserve unmatched rhs series in or

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): stream union distinct inputs

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-16 02:29:43 +00:00
discord9 2123108db0 fix(promql): preserve ordinary NaN samples (#8494)
* fix(promql): distinguish stale markers from NaN

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): preserve ordinary NaN samples

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-07-15 01:54:49 +00:00
Yingwen a25152664b fix: qualify HistogramFold schema (#8157)
* test: add regression test for binary op on histogram_quantile (#8144)

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix: preserve column qualifiers in HistogramFold output schema (#8144)

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-05-25 07:40:48 +00:00
Ruihang Xia aa0ff3cce7 feat: some optimistic paths for instant manipulate (#7812)
* feat: some optimistic paths for instant manipulate

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* use tsid in manipulate plan, resolve_tag_columns walks whole plan

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix tsid reuse

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test assertions

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* cap max points

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-05-14 02:36:25 +00:00
Ruihang Xia 8ad77ce649 perf: optimize extrapolated rate op family (#7880)
* perf(promql): optimize extrapolated rate hot path

* more ut

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix gauge rate case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* adjust comments

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-04-13 19:11:56 +00:00
Ruihang Xia 09b368c00a feat: tune constants (#7851)
* feat: tune constants

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* cap output batch size

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* handle empty input

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* one more ut for cr

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-04-08 23:34:13 +00:00
shuiyisong ba32c5fe9e chore: remove unused deps using udeps (#7906)
* chore: remove unused deps using udeps

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* chore: fmt toml

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-04-02 06:49:27 +00:00
Ning Sun e14404c677 chore: update rust toolchain to 2026-03-21 (#7849)
* chore: update rust toolchain to 2026-03-21

* chore: new format

* fix: lint

* chore: resolve lint issues

* chore: remove as_millis_f64

* chore: deps up
2026-03-30 12:13:14 +00:00
Ruihang Xia fe45ae446c perf: optimize promql range functions (#7878)
* bench(promql): add range-function benchmark suite

* perf(promql): use flat buffers in range function hot loops

* perf(promql): reuse quantile scratch buffers
2026-03-27 23:36:13 +00:00
Ruihang Xia 39140058d0 feat: adapt new name of holt winters fn (#7700)
* feat: adapt new name of holt winters fn

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update parser

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* alias old fn name

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-03-04 23:32:37 +00:00
b2074e3863 chore: upgrade DataFusion family, again (#7578)
* chore: upgrade DataFusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: switch to released version of datafusion-pg-catalog

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
Co-authored-by: Ning Sun <sunng@protonmail.com>
2026-03-03 07:36:39 +00:00
Ruihang Xia 4ac73a7e9e feat: adapt prometheus 3.x selector and lookback behavior (#7688)
* feat: adapt prometheus 3.x selector and lookback behavior

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove commit hash from comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-02-10 04:00:45 +00:00
Ruihang Xia a10d40e552 feat: adapt prometheus 3.x matrix selector behavior change (#7671)
feat: adapt prometheus 3.x matrix selector behavior

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-02-06 03:06:33 +00:00
Ruihang Xia f9030a84c8 fix: incorrect column statistics of promql plans (#7662)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-02-04 04:58:38 +00:00
discord9 38e4a94956 feat: ser/de for HistogramFold&UnionDistinctOn (#7601)
* feat: ser/de for histogramFold&UnionDistinctOn

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

* fix: skip find time window when in tql

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-01-21 15:02:20 +00:00
Ruihang Xia 8566bf1409 feat: support for types of series identifier in series divide plan (#7583)
Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-16 08:41:34 +00:00
LFC e64c31e59a chore: upgrade DataFusion family (#7558)
* chore: upgrade DataFusion family

Signed-off-by: luofucong <luofc@foxmail.com>

* use main proto

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2026-01-14 14:02:31 +00:00
Ruihang Xia 170f94fc08 feat: enable pruning for manipulate plans (#7565)
* feat: enable pruning for manipulate plans

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* apply to other plans and add sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* fix scalar manipulate and histogram fold for missing some columns

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* don't drop every columns

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* remove unrelated part

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2026-01-14 08:32:51 +00:00
Ruihang Xia bd3ad60910 fix: promql offset direction (#7392)
* fix: promql offset direction

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sort sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* commit forgotten file

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-12-12 07:51:35 +00:00
Ruihang Xia cbfdeca64c fix: promql histogram with aggregation (#7393)
* fix: promql histogram with aggregation

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update test constructors

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sqlness tests

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* update sqlness result

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* redact partition number

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-12-12 07:32:04 +00:00
Ruihang Xia 60f752d306 feat: run histogram quantile in safe mode for incomplete data (#7297)
* initial impl

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* sqlness test and fix

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* correct sqlness case

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* simplification

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

* refine code and comment

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>

---------

Signed-off-by: Ruihang Xia <waynestxia@gmail.com>
2025-12-05 09:19:21 +00:00
LFC 4a7c16586b refactor: remove Vectors from RecordBatch completely (#7184)
* refactor: remove `Vector`s from `RecordBatch` completely

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
2025-11-21 08:53:35 +00:00
LFCandNing Sun 2f637a262e chore: update datafusion to 50 (#7076)
* chore: update datafusion to 50

Signed-off-by: luofucong <luofc@foxmail.com>

* fix ci

Signed-off-by: luofucong <luofc@foxmail.com>

* fix: update datafusion_pg_catalog import

* chore: fix toml format

* chore: fix toml format again

* fix nextest

Signed-off-by: luofucong <luofc@foxmail.com>

* fix sqlness

Signed-off-by: luofucong <luofc@foxmail.com>

* chore: switch datafusion-orc to upstream tag

* fix sqlness

Signed-off-by: luofucong <luofc@foxmail.com>

* resolve PR comments

Signed-off-by: luofucong <luofc@foxmail.com>

---------

Signed-off-by: luofucong <luofc@foxmail.com>
Co-authored-by: Ning Sun <sunning@greptime.com>
2025-10-23 07:18:36 +00:00
discord9 a9a3e0b121 fix: prom ql logical plan use column index not name (#7109)
* feat: use index not col name

Signed-off-by: discord9 <discord9@163.com>

* fix: use name without qualifier&output schema fix

Signed-off-by: discord9 <discord9@163.com>

* proto

Signed-off-by: discord9 <discord9@163.com>

* refactor: resolve column name/index

Signed-off-by: discord9 <discord9@163.com>

* pcr

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

* chore: update proto

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2025-10-22 09:04:09 +00:00