mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
cc6aa640fd5f914d9ae0505dfc65920261cc676f
6595
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dennis Zhuang
|
cc6aa640fd |
feat(promql): add stddev, stdvar, quantile and count_values aggregates
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
 discord9
|
3ad5f96328 |
feat(promql): compile a parameterised range function (#9743)
* feat(promql): compile a parameterised range function `quantile_over_time(q, x[r])` now compiles: the quantile is bound as a number literal and the range selector is bound by the same path a function without parameters takes. The two shapes share one binder, so the range, the offset, the anchor and the grid cannot drift between them. The quantile is not range-checked. Prometheus binds any value and the kernel answers `-Infinity` below zero and `+Infinity` above one, so refusing it here would be this compiler inventing a rule the engine does not have. A parameter in parentheses binds the same quantile: the planner already looks through the parentheses when it reads a literal, so a query written `quantile_over_time((0.9), x[5m])` keeps working on this path too. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): cover a parenthesised selector of a parameterised function The unparen rule applies to the selector a parameterised function binds, not only to the one a plain range function binds: a query that writes its range selector in parentheses asks for the same tree. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): name the real run-time parameter, and reuse `unparen` The doc said `1 + 1` is computed at run time. It is not: `try_build_literal_expr` folds `Paren`, `Unary` and literal-only `Binary` in the existing planner, so `quantile_over_time(1 - 0.1, x[5m])` and `quantile_over_time(-(0.9), x[5m])` plan today and reach this compiler unfolded. Refusing them is the agreed behaviour for this stage, not a run-time-only case, and the doc now says so; `time()` is the real run-time example. A test pins both folded forms as refused, so implementing folding later is a deliberate change rather than a silent one. `scalar_parameter` open-coded the parenthesis loop `unparen` already implements, so the rule for transparent wrappers now lives in one place. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
44df7a2a1f |
feat(promql): compile a bare selector and an anchored range window (#9736)
* feat(promql): compile a bare selector and an anchored range window The compiler refused a bare instant selector and any `@` anchor, so the two shapes an ordinary query starts from could not reach the tree. It now binds a selector's lookback, offset and output grid into an instant node, and resolves `@ 1234`, `@ start()` and `@ end()` into the millisecond anchor a range window already carries, bound apart from the offset so a shift and an anchor are not folded into one number. A bare selector's own anchor stays refused: an instant node binds a lookback, an offset and a grid and has no field to anchor one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): tell `@ end()` apart from the last grid point The anchor test used a statement ending at 100s on a 5s step, so its last grid point was also 100s: binding `end()` to the last grid point instead of to the statement's own end would have passed. The end is now a parameter of the statement helper, and a statement ending at 103s pins the two apart, asserting both the anchor and that the grid still stops at 100s. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): write down what an anchored window reads, and cover a pre-epoch anchor An `@`-anchored window reads `(anchor - offset - range, anchor - offset]`, which can lie entirely outside the statement's own range, so whoever binds the scan has to derive it from the window rather than from `stmt.start` and `stmt.end`. A scan bound to the statement alone leaves such a window with no samples, and the query returns empty without an error. That contract was nowhere in the code. The anchor cases only covered non-negative anchors, so a helper that defaulted a pre-epoch anchor to zero would have kept the suite green. `rate(x[5m] @ -1.5)` now pins -1500, which is the `Err(earlier)` branch of the conversion. The `assert_ne!` at the end of the `end()` test was implied by the two equalities above it and could not fail on its own. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4c7cc8b10b |
feat(promql): compile a statement into a native tree and display it (#9732)
* feat(promql): compile a PromQL statement into a native tree Compile a range selector and its surrounding shape into the native plan nodes introduced earlier, and render the resulting tree through `Display` so an `EXPLAIN` can show it. The display carries each node's operation, its bound window and the fields it reports. Field layout is validated here rather than at execution: a node that declares a different number of fields than it consumes is a planning error, which is what the fixture tests pin. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): strip parentheses the native compiler does not read through Parentheses group; they say nothing about the expression inside them, so `(rate(x[5m]))` asks for exactly what `rate(x[5m])` asks for. The compiler matched the bare shape only and refused the parenthesised one as NotImplemented, rejecting a statement on how it was typed rather than on what it means. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an ordering a window cannot consume, and stop claiming an executor that does not exist `compile` recorded the scan's ordering without checking it, so `compile(rate(x[5m]), schema, Interleaved)` built a tree that `validate` accepted and the range evaluator was guaranteed to reject at run time: it reads one series to its last chunk before the next. The ordering is refused where it is declared now, and the tree's own window check is the backstop. The tree carries no metric name or matchers, so the contract on `input` decides correctness and was nowhere written: matchers, `__field__` among them, have to be applied before `input` is bound, and `input.fields` is the post-`__field__` set. The display marked a hand-written list of nodes "(no executor)". No executor exists at this point in the stack, so `Window` and `Map` read as runnable while `Instant` was marked as not, although `select_instants` runs in production. The marker is gone; every line reads as the node it is and claims nothing about whether it could run. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): render a grouping as the shape it was bound as The tree now binds an aggregate's grouping as `Grouping::{By, Without}` (#9723), so `label_list(&op.group)` no longer type-checks: the argument is a `Grouping`, and the function takes `&[Arc<str>]`. This is production code, not a test helper, so the merged stack would not build. Rendering on the `By`/ `Without` shape fixes more than the type error: both were printed as `by [... ]`, which would show `by(job)` and `without(job)` — two different queries over the same label list — as the same query. A `without` aggregate now reads `sum without [job] complete`, and `a_without_grouping_does_not_read_as_by` pins it. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f2f8043fdd |
feat(promql): add the bound native tree and the metric surface it reports through (#9723)
* feat(promql): add the bound tree and batch protocol for native execution A native PromQL evaluation needs a compiled form whose operations are already bound to kernels, fields and grids. Recognising an opportunity from a finished physical plan cannot provide that: the opportunity then depends on how the optimizer happened to arrange a subtree, and every new shape widens a fixed template. `NativeExpr` instead covers input, scalar, instant, window, map, aggregate and binary nodes, each carrying a bound operation and a declared output type. Legality becomes a question about types and data dependencies, which `NativeNode::validate` answers, so a new combination of supported operations is an ordinary tree rather than another matcher. Two properties are checked there rather than assumed: a window's declared input grid must be the grid its input produces, so a subquery's inner grid is consumed as it is instead of being resampled, and an aggregate's output type must match its stage, so intermediate state cannot pass for a query value. `EvaluatedChunk` is the batch a node hands to the next one: an ordinary Rust value, addressed on absolute grid positions, closing a series exactly once. An empty final chunk is still meaningful — it is how a closed prefix is reported, which a consumer needs in order to tell "nothing more here" from "no more input". Nodes are identified by their position in the tree rather than by an id. The position is derived by the one traversal both `DisplayAs` and run-time construction use, so what is shown and what is registered cannot disagree. It stays in the metric name because the collector aggregates by name first, which would otherwise merge two nodes of the same operation. This adds the tree and the batch protocol only. The evaluator's chunked interface, memory reservation and execution are not part of it. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): make the bound tree's type contracts match what each stage consumes Four review findings on the bound tree, all about checks that passed the wrong things. An aggregate's stage decided its output but not its input. Every stage first required its input to be evaluated, which aggregate state is not, so `Final(Partial(...))` was rejected and `Final(Window(...))` was accepted. Each stage now states its pair: `Complete` takes a vector and produces one on the same grid, `Partial` takes a vector and produces state on that grid with the bound grouping, `Final` takes state with the bound grouping and produces a vector on the state's grid. Window and instant selections accepted aggregate state. The cause was asking `is_evaluated()`, which is false both for raw samples and for state, since state has no grid: `Instant(Partial(...))` and a window bound to `input_grid: None` over `Partial(...)` both passed and would have read state as raw samples. Both now match on `NativeType::Samples` first and only then on the grid, so state is rejected by what it is rather than by lacking a grid. Scalars were modelled as a grid-less evaluated sample, and `BoundScalar`'s doc claimed a value varying per grid point is not a scalar. That is wrong for PromQL: `time()` and `scalar(v)` produce a scalar per evaluation time. A scalar is now its own `NativeType::Scalar(grid)`, so a binary operation can tell a broadcast from label matching -- `sum by(job)(rate(x[5m])) + time()` broadcasts rather than matching on the empty label set -- and `Map`, `Aggregate`, `Instant` and `Window` reject it by type. The `time()` kernel itself is not in scope. An input could bind only samples, so an exchange boundary could not bind the state a `Final` aggregate consumes. `BoundInput` holds a `NativeType`, and input nodes check that their output is the same binding rather than the same schema. A peak is the one figure the collector must not add up: it groups metrics by name, clears the partition and sums gauges, so partition peaks of 1024 and 2048 reported 3072, which is neither peak. `PEAK_RESERVED_BYTES` now carries the partition in its name (`native.p0.root.peak_reserved_bytes`), while counts and the other gauges still sum across partitions, where a sum is meaningful. The test that claimed the peak kept its partition only checked a name prefix; it now registers two partitions, runs the real `aggregate_by_name`, and asserts both peaks survive with their own values. The walkthrough's `+1` test was wrong twice over: it took a `Final` aggregate directly over a `Window` -- the case the first finding now rejects -- and applied `MathOp::Abs`, which is not `+1`. It now builds a real binary `+` with a scalar operand. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): use the evaluator's chunk type for native batches `native/batch.rs` defined its own `EvaluatedChunk` with the same two fields as `promql::evaluated::EvaluatedChunk`. Two definitions of one protocol drift: the evaluator's chunk could not be handed to a native node without translation, and a change to either would have to be mirrored. Use `crate::evaluated::EvaluatedChunk` instead. The type moves here with the evaluator's documentation, which states the same two rules this module did: a grid position is absolute, and `is_last` is the only thing that closes a series. The module keeps its own doc explaining the rules native nodes rely on and re-exports the type, so `native::batch::EvaluatedChunk` still names it. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reject a node that declares a different field count than it consumes A window and a map both report one value per input field, so they keep the layout of their input. Validation checked that both sides were samples and that the grids matched, but not that the field counts did, so a tree could declare two output fields over a one-field input and still pass `NativeTree::try_new`. The mismatch then surfaced at run time, against a schema the tree was supposed to have validated. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate the field layout of every node that declares one Field counts were checked for a window and a map and nothing else, so an instant selection, either aggregate stage and a binary operation could each declare a layout their kernel does not produce and still pass validation, surfacing at run time against a schema the tree was supposed to have validated. Each node is now checked against the count its own kernel produces, which for a binary over two vectors is the shorter side. Grids are compared by their specification, as the kernels compare them, instead of by allocation. A compiler that allocates one grid per window -- `rate(a[5m]) + rate(b[5m])` -- built a tree the runtime would run and validation rejected. Binding identity is still compared by allocation: an input has to report the one binding it was given, which is a different question from what two equal grids mean. A rejected tree is a compiler defect, so it is reported as an internal error rather than as a mistake in the client's statement, which `Plan` would have implied. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): check a window's input ordering and range, and drop the tests that only repeat another module `validate` accepted an interleaved input for a window, although the range evaluator reads one series to its last chunk before it starts the next, so the tree this layer builds was guaranteed to fail at run time. A window now rejects both an interleaved input and a range that is not positive. An instant selection reads one series at a time, so it keeps accepting an interleaved input and now says why. The grid comment claimed the input grid is checked to cover the window's read range. It is not: only a subquery feeds an evaluated input to a window, and this compiler does not accept subqueries yet. The binary operation's catch-all arm is unreachable at run time -- state has no grid and the grid checks return before the match -- but the match still has to cover the state pairs, so they are spelled out and say so instead of claiming a value arrived. Rejection branches had no tests, so deleting any of them kept the suite green. Each now has one. The four tests in `native/batch.rs` repeated the ones in `evaluated.rs` through the re-export, so they could not fail on their own; they are gone, and the coverage stays in `evaluated.rs`. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): bind a grouping as `by` or `without`, not as a label list A label list can only spell `by`: `sum without (instance) (rate(x[5m]))` has no list of group labels, because its key is every label the series has except the ones named. Spelling that as a list would group the query by labels it meant to remove, and the kernel takes both shapes already (`aggregate::Grouping::{By, Without}`), so the tree had no way to say what the engine can already do. State carries its grouping in the same terms now, so a `Final` also refuses state built for the other shape, not only state built from a different label list. The `groups` gauge counts per partition: a `Partial` aggregate sees one group in every partition it reaches, so the summed gauge is the total of per-partition entries rather than distinct groups. Both the metric's doc and the comment that claimed otherwise now say so. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@users.noreply.github.com> |
||
|
discord9
|
872753e4cd |
refactor(promql): read and write sparse evaluated series (#9750)
* refactor(promql): read and write sparse evaluated series The axis can now say a series holds values at only some grid positions. This makes PromQL use that: the builder stops padding its output, every operator reads positions and rows from the axis instead of deriving one from the other, and the streaming protocol stops treating a run boundary as a delivery boundary. `GridSeriesBuilder::finish` returns one `Series`, or `None` when no retained row holds a value, with the axis built from the runs the retained positions make up. It keeps one buffer per field and slices runs out of it by span, so a series with values at 10 and 900000 on a million-point grid costs two rows, not the distance between them. The queue that cut a series into one piece per run is gone: `take_buffered` hands out everything buffered so far as one `Series` and ends nothing, so a caller that wants a prefix still gets one while the later rows stay on the same logical series. The position cursor resets only in `begin_series`, which is what marks the next logical series; a flush must not rewind it, and a test pins that. Every reader follows. A place that took one start offset and computed a row as `position - start` now asks the axis for the row, or walks positions with a forward cursor per field, or intersects two axes with `intersect_runs` carrying one compact offset per side -- broadcast and vector matching used to compare envelope bounds and subtract starts, which is only correct while every axis is contiguous. Matches stay exhaustive: no wildcard arm was added. The six functions that had grown a `Vec` purely because a result could be segmented return `Option<Series>` again: `absent`, `map_native_histogram`, `map_date`, `map_compare`, `map_arithmetic_series` and `map_grid_scalar`. Each has one output identity per input -- one metric name, one field set, one axis -- and a hole in the result is a property of that axis, not a second identity. `None` means no row survived. Operators that genuinely fan one input into several identities keep returning a `Vec`: aggregation groups, topk, binary and set matching, `merge_fields`, `merge_same_labelset` and the label operators; those reuse one builder per output series and now call `begin_series` explicitly, since `finish` no longer resets. An evaluated series may now be closed by an empty delivery. `push_series` refused any empty input while the evaluator itself produces an empty closing chunk after a valued prefix and for an identity that never produced a value, so a pipeline of two evaluators rejected its own output. An empty chunk that is not the last still decides nothing and stays an error; the last one is a close, whatever it carries. `SeriesState::last_ts` becomes an `Option` so an identity with no position at all has no timestamp to invent. Three walks that were correct but quadratic are fixed here because the shapes changed under them: binary matching rebuilt its two sample cursors for every shared run, so each run re-read every row before it; set matching asked the filter for presence one point at a time and topk asked one member at a time, both through `Samples::sample`, which scans chunk descriptors from the start; and topk ranked every grid point instead of the union of the members' positions. The cost argument is from the code, not measured -- how many samples a walk reads is not observable through the public API -- while the tests pin that the results are unchanged under fragmentation. The per-operator suites asserted the old shapes, so the expectations that said "one piece per run" now say "one series, several runs". Two of them could not have failed before: histogram aggregation was compared through a helper that dropped every non-histogram row, so a padded range and a sparse one compared equal, and the sparse `map_date` case claimed to exercise the cutting path while holding only present samples. Both now assert the exact runs, row counts and alignment, so a return to padded output fails. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): pick a merged row's cursor by slot, not by search `merge_same_labelset` collects the rows of every member of a label group and then merges them position by position. Collecting stores which member each row came from, and the merge read that member's value through a per-member cursor, because a member of several runs must not re-scan its field chunks per row. Between the two it looked the member up again for every row: let member_index = members .iter() .position(|candidate| *candidate == member) .expect("the row names one of the members"); That is a linear search over the group per row, so a group of N members with one disjoint value each pays N(N+1)/2 comparisons just to find its cursors -- on the path a label function takes when it drops the label that kept the series apart, which is the ordinary way several identities become one. The previous shape indexed `series[member]` directly and had no such term. The rows already hold the member; make it hold the member's slot instead. `members` is built in ascending input order, so sorting by slot orders the merge exactly as sorting by member did, and the cursors stay forward-only. No lookup, no map, no change to the merge itself. The regression case is the one that exposes it: many members whose single values sit at distinct positions share a label set once the distinguishing label is dropped, so the collision check never short-circuits and the whole merge runs. It asserts the merged axis carries one row per member with each member's own value, which a wrong slot fails -- checked by forcing the slot to zero, where both this case and the existing disjoint-fields case fail. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): say what a chunk does and does not promise, and compare without folding `EvaluatedChunk` claimed the positions after the last row handed out had not been decided yet, and `range()` claimed it reported how far a series had been evaluated. Neither holds: a push can settle a trailing stretch as empty and hand out nothing for it. What holds is that a position no chunk carries either holds no value or has not been decided, and a consumer cannot tell the two apart, so only `is_last` says a series is complete. `assert_result` folded the series it was given by label set before comparing, so a producer that wrongly cut one label set into two series was folded back together and passed. It now refuses that outright; `fold_pieces` stays for the callers that genuinely fold a stream. `RangeEvaluator::pull` and the `finish` branch that checked it were dead: every push ends in an `emit` that drains the builder, so nothing is ever left to pull and seven assertions said a condition that could not be false. `skipped_and_all_missing_positions_are_the_same_series` pushed `&[None]` where it meant to skip, so both builders received the same calls and the test compared a builder with itself. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
82c7d145b7 |
feat(datatypes): support sparse evaluation-grid axes (#9762)
* feat(datatypes): support sparse evaluation-grid axes A series that holds a value at only some grid positions had no way to say so: its time axis was one contiguous range, and an explicit timestamp list is not the same thing -- it loses the evaluation grid the series is validated against, and cannot answer which grid position a row belongs to. `TimeAxis` already owns positions, length, timestamp access and slicing, and already has a chunked form that keeps several Arrow buffers unflattened. This gives grids the same treatment. `GridSpan` is one run of absolute positions; `GridRanges` is an ordered, non-overlapping list of spans on one shared `EvalGrid`; `TimeAxis::GridRanges` carries it beside the contiguous `TimeAxis::Grid(GridRange)` fast path, which keeps its contract including the zero-copy `timestamps()` slice. The coordinates are fixed by the axis and documented on it. `len()` counts carried rows, not the distance from the first position to the last; `timestamp`, `slice` and `row_of` speak compact rows; `positions()` walks `(absolute position, compact row)` pairs in one pass with one cursor per span; `intersect_runs` merges two run lists without forming their product; `union_positions` merges several axes. Spans are validated at construction -- ordered, non-overlapping, inside the grid, arithmetic checked -- and an axis with no rows still owns its grid, so nothing unwraps a first span to find it. `row_of` is a binary search and `slice` rebuilds through the checked constructor. `Samples` gains a forward `RowCursor` so a walk reads rows in order without scanning chunk descriptors per row. `Series::try_new` accepts the new form under the same rule as the contiguous one: its grid must match `SeriesSchema::eval_grid`, and an explicit axis still may not carry an evaluation grid. Nothing produces a fragmented axis yet. The type and its invariants land first, and every existing reader keeps its behaviour: the places that require a contiguous axis already return an error for anything else, so they reject the new form rather than misreading it. Wiring the readers changes what those places must do with their results, which is why it follows separately. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(datatypes): check the shared grid, keep one constructor, and stop a cursor from poisoning itself `intersect_runs` and `union_positions` walk two axes together but never said they must be addressed on one grid; on an explicit axis every position is 0, so the pairs were silent nonsense. Both now state the precondition and assert it in debug builds. `GridRanges::try_new` accepted a single span, which is not normalized into the one run a `TimeAxis` fast path expects, so a public caller could build a range that reports `is_contiguous()` and still has no grid range. It is now `pub(crate)`; `TimeAxis::from_grid_spans` stays the public way in and keeps normalizing a single span. `Samples::RowCursor::at` drained its iterator on an out-of-range row, so one bad call made every later call answer `None`. It now bound-checks first and leaves the cursor where it was. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
05659b1094 |
perf(promql): hand out evaluated series as chunks instead of one buffered series (#9719)
* perf(promql): hand out evaluated series as chunks instead of one buffered series `RangeEvaluator::push` evaluated every grid point a chunk completed, wrote the rows into its `GridSeriesBuilder`, and then returned `None` for a non-final chunk. A series therefore became visible only on its last chunk: the rows of every earlier chunk stayed buffered in the builder, and a consumer could not release or forward anything until the series ended. That is also why the fused node that evaluates raw samples could emit one batch per series and not per input chunk. `push` now returns the positions it finished as an `EvaluatedChunk`: the evaluated rows plus an `is_last` flag, addressed on the grid by an absolute `GridRange`. A consumer can forward a prefix while the series continues, and `is_last` is the only thing that closes the series. `evaluated.rs` holds that type and the rule it enforces, that a chunk is addressed on a grid. Two protocol points make streaming safe. Ranges of one series only increase, so concatenating the chunks of a series never reorders them. The closing chunk is handed out even when it holds no position, so a series whose remaining grid points have no value still reports that it ended rather than leaving a consumer waiting for input that will not come; previously those series returned `None` and ended silently. `push_series` likewise returns its one closing chunk. Tests fold the chunks of a series back into one series and compare that against the same brute-force reference and Prometheus UDF references as before, so the chunking is checked against the values, not against itself. The fold helper fails on a chunk after the closing chunk, and the golden range-function and mixed-histogram suites, which push one sample per chunk, go through it. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): check the label and position of the chunk that closes a series The fold helper checked the ordering of a chunk only through the rows holding a value, which the grid builder sees. A closing chunk has no rows, so an empty chunk at a wrong position passed. It now compares every chunk's range against the end of the previous one directly, and compares every chunk's metric, since a consumer keys series by their labels and a later chunk with different ones would look like a different series. `an_empty_closing_chunk_is_labelled_like_the_chunks_before_it` covers the label half for `rate`, which drops `__name__`: this also fails if the empty closing chunk stops applying the metadata removal. Also corrects the docs: the range spans the first and last position holding a value, so it is an envelope that may contain missing values, not a report of every position evaluated. `is_last`, not the empty endpoint, says whether the positions after the range are missing or not evaluated yet. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): hand out positions decided to be empty, not only positions holding a value `emit` finished the builder and returned `None` for a non-final chunk when the builder held no valid row. A grid point that was already decided to have no result therefore produced no chunk, and a consumer never learned it was settled: it could not advance its own windows past those points and stayed waiting for input that had already been shown to contribute nothing. The example that fails: inner `rate(x[1s])` on the grid 0..4s. A chunk with samples at -0.5s and 0s decides the point at 0s. A chunk with a sample at 2s decides that 1s has an empty window and 2s a single-sample window -- both missing -- and nothing later can change them, since a sample at 4s lies past both windows. The old code returned `None` there, so an outer `sum_over_time(...[3s:1s])` could not emit its own result for 1s and 2s until the series ended. To report progress, the evaluator now tracks how far it has decided, separately from how far it has values, and the chunk span covers that whole range: - `SeriesState::completed` is the grid index past the last decided position. A point counts as decided when it has been evaluated, when no retained sample reaches its window any more (the loop stops there instead of walking the grid, so the points behind it are reported too), or when the series ends. - `GridSeriesBuilder::cover_from` / `cover_through` widen the builder's span around positions that hold no value, so the finished series reports them as missing instead of trimming them away. The builder's own `finish` still trims to the outermost value for callers that have not decided anything. - `emit` covers `[emitted_end, completed)`, which is empty only when nothing was decided since the last chunk. A series that ends without deciding anything new still closes, on an empty chunk. Tests that pinned the old trimming are updated, and `a_chunk_of_decided_missing_positions_is_handed_out_before_the_series_ends` is the case above: with the sample at 4s not yet arrived, the second push must yield a non-final, all-missing chunk covering points 1 and 2. Folding the three chunks still reproduces the whole grid, so reporting the missing spans costs no values. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): hand out no chunk when a push completes no position `evaluate` opens the builder's span at the first position it has not handed out yet, even when the push settled nothing. `emit` then called `cover_from` and `cover_through` for the decided range, and `finish` built a zero-length series instead of returning `None`, so a consumer received a chunk whose span claimed progress the push had not made. With a grid of 10/20/30ms and a 10ms window, a first chunk holding only `(5ms, 1.0)` reported a zero-length chunk while the first grid point was still undecided. Return `None` when the decided range is empty and this is not the last chunk. The closing chunk is still handed out, empty when nothing is left to report, because `is_last` is what ends a series. `a_push_that_completes_nothing_hands_out_no_chunk` pins that single push, and then checks the closing push still reports every position of the grid. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): fold the batch-path chunk before comparing against the per-window series `a_batch_spanning_fields_and_the_flush_threshold_matches_the_per_window_path` compared the two paths as `Series`, but `push` hands out an `EvaluatedChunk`. Taking `.series` is not enough: a chunk's grid range reports the positions its push decided, which can include leading and trailing positions holding no value, while the reference assertion requires the grid range to be trimmed to the valid rows. The batch path reported `(0, 600)` against the expected `(1, 599)`. Fold the chunk back into one series first, which trims to the valid rows the same way, and compare that. The rebase also removed the `sum` helper along with the test `overlapping_window_spans_chunks`, so the now-unused `floats` helper goes with it. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * chore(promql): let an evaluated chunk be cloned `EvaluatedChunk` holds a `Series` and a bool, both cheap to copy behind their existing ownership, so deriving `Clone` costs nothing. A consumer that has to keep the chunk it matched on, or hand the same chunk to two stages, would otherwise have to destructure and rebuild it. Needed by the native tree, whose bound nodes receive chunks by value and reuse them. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): share the owned schema when folding a batch chunk Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 discord9anddiscord9
|
21e3e10b78 |
perf(promql): evaluate rate and increase over a batch of windows (#9710)
* perf(promql): evaluate rate and increase over a batch of windows `RangeEvaluator` called a boxed `FnMut(FloatWindow) -> Option<f64>` once per grid point per field, building a six-field window struct and appending a row for each. A series of thousands of points therefore made thousands of boxed calls and row appends. The counter functions now take a batch: `BatchKernel` is `FnMut(&FloatWindows, &mut Vec<Option<f64>>)`, selected once outside the loop, and `extrapolated_rate_batch` runs concretely typed per `rate` and `increase`. `FloatWindows` reuses the retained arrays and one sample range per window rather than a copy per window, because a counter function reads only a window's endpoints, sample count and resets. Other functions keep the per-window form and the evaluator falls back to it. This batches the computation only. The results still reach `GridSeriesBuilder` one row at a time, and `flush_batch` reuses the evaluator's row buffer rather than allocating per point. Writing a whole batch as columns needs a column entry point on the builder, which is follow-up work; this commit cannot be read as carrying the whole gain of the experimental branch. A window holding a native histogram sample is decided by the mixed-window rules of `histogram::evaluate`, which read both kinds of sample together and report nothing for a counter over a mixed window. The batch kernel reads the floats alone, so a point whose window holds a histogram flushes what is pending and runs on the per-window path. Without that, a mixed window would report a float rate instead of nothing, and a histogram-only window would lose its histogram result. Tests assert the batch path is the one that ran, since both dispatch shapes produce the same output; a mutation that drops the histogram admission check fails them. Window ranges stay absolute into the retained arrays across a batch, and each accumulated point keeps its own grid index, so a batch spanning a skipped grid point or a chunk boundary still writes the right rows. Signed-off-by: discord9 <discord9@users.noreply.github.com> * test(promql): cover several fields and a mid-evaluation flush in batch window evaluation The batch path was checked only by single-field inputs of at most 20 points, so two of its properties had no test: that each field's windows come from that field's own samples, and that a batch flushes when it fills up before the series ends. Mutating the first -- giving every field field 0's windows -- and `panic!`ing in the second both passed the whole suite. `a_batch_spanning_fields_and_the_flush_threshold_matches_the_per_window_path` pushes 600 points over three fields, each field resetting at a different point, growing at a different rate and missing at a different interval, and compares the batch result against the per-window path on the same input. The gaps are what make the property observable: with every field present at every point the three fields' windows coincide and field 0's spans would be correct by accident, which is how the first form of this test missed the mutation. The histogram admission check also stops scanning every remaining histogram per point and running for functions without a batch kernel: a field's histograms are retained in time order, so the first one ahead of the window head decides, and the check only matters when there is a batch kernel to fall back from. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: discord9 <discord9@users.noreply.github.com> |
||
|
discord9anddiscord9
|
c500f4697f |
perf(promql): append ordinary raw chunks to the evaluator in bulk (#9709)
* perf(promql): append ordinary raw chunks to the evaluator in bulk `ingest` copied every input sample into `FieldSamples` one at a time. A `RawSeriesChunk` hands the evaluator Arrow buffers `Samples::from_float` wrapped without copying, and `ingest` read them back through `Samples::iter` only to perform, per sample, an enum match, a stale-marker comparison, an overflow-checked offset addition and two `Vec::push` calls. A chunk of null-free float columns read with no offset now appends with `extend_from_slice`. Every other shape keeps the per-sample loop: a shifted timestamp reads different timestamps, a null has no sample, a field mixing native histograms reports `DenseSamplesRef::Mixed` and must keep its windows on the histogram path where the mixed-window rules decide the result, and a chunk whose values contain a stale marker drops its row and needs the per-row decision. The reset index stays per-sample inside the bulk append, so the recorded resets are the same ones the per-sample path records. Tests assert both that the values still match the brute-force reference and that the bulk path is the one that ran, since both paths produce identical output by construction; a mutation that lets a mixed field through the admission check fails them. Signed-off-by: discord9 <discord9@users.noreply.github.com> * refactor(promql): keep the ingest docs with ingest and drop untested guards Three review findings on the bulk-ingest path, none changing behaviour. The doc comment describing `ingest` sat on top of `ingest_in_bulk` after the bulk path was inserted above it, so `ingest` had lost its docs and the bulk function carried documentation for work it does not do. The comment moves back to `ingest`. `ingest_in_bulk` returned `Result<bool>` but never returned `Err`: the two length checks inside it are already done by `check_chunk` for raw input and by `push_series` for evaluated input. It now returns `bool`, and the length checks go with the `Result`. The stale-marker and offset "is not appended in bulk" tests guarded faults the behaviour tests already catch -- the differential and reference comparisons fail if either condition is admitted. The histogram one stays: it is the only test covering that admission rule. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Co-authored-by: discord9 <discord9@users.noreply.github.com> |
||
|
discord9
|
3fca17791c |
perf(promql): index counter resets per field instead of rescanning the window (#9708)
* perf(promql): index counter resets per field instead of rescanning the window The fused evaluator rescanned every window for counter resets, once per grid point. Record the resets of each field as the samples are ingested instead, keyed by an absolute sequence number, and read the ones a window covers with a binary search. Only `rate` and `increase` correct counters, so only they maintain the index. The binary search runs over the field's retained reset index rather than over the window's samples: its input is the remaining slice of resets, so its cost follows the number of retained resets, not the window length. A drop across an intervening native histogram sample is recorded as a reset of two samples that are not neighbours. That is unobservable: a window holding both floats also holds the histogram between them, so it is a mixed window, whose counter functions report nothing; and a window starting after the earlier float excludes a reset on its first sample. Verified with `cargo nextest run -p promql` (453 passed) and `cargo clippy -p promql --all-targets --all-features -D warnings`. Tests assert both that the indexed results match the brute-force scan and that the index is what ran. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop per-field clearing the evaluator never needs Two review findings, both about code that does nothing. `finish_series` cleared every field after `evaluate(None)` returned. That loop is redundant: `evaluate` ends with either every point evaluated -- so `head` moved to the end and `advance` dropped the samples -- or no field holding a sample, and either way the fields are empty. It also did not clear `histograms`, which made it read as though it should. Removing it keeps the suite green, and `push` already documents that an evaluator is not reused after an error. `FieldSamples::clear` existed only for that loop. `overlapping_window_spans_chunks` used a `sum` kernel and never touched resets. Windows spanning chunks are already covered by `evaluator_matches_brute_force_reference`, which chunks randomly, and no mutation was caught only by this test. Its `sum` helper went with it. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@users.noreply.github.com> |
||
|
discord9
|
04589c175f |
perf(promql): widen the shared extrapolation arithmetic to 64 bits (#9704)
* perf(promql): widen the shared extrapolation arithmetic to 64 bits `extrapolation_factor` is the one place both the float and the native-histogram range paths compute Prometheus' extrapolation. Its three millisecond differences were widened to `i128` and narrowed back with `as f64`; each of those narrowing conversions compiles to a call to the `__floattidf` builtin rather than an instruction, and the function runs once per grid point per series. Use saturating `i64` subtraction instead, which is exact here: a difference too large for `i64` is either a window start below every representable instant, or a distance far past the extrapolation threshold where both values take the same branch. Both callers now share the same arithmetic; neither path carries its own copy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): wrap the extrapolation arithmetic like Go instead of saturating it `extrapolation_factor` widened to `i128`, subtracted, and narrowed back with `as f64`; that compiles to a call to the `__floattidf` builtin once per grid point per series. Doing the subtraction in `i64` removes the calls, but the first form of it clamped with `saturating_sub`, and clamping is not what Go computes: with `eval_ts = i64::MIN + 10`, `range_ms = 100`, `first_t = i64::MIN + 1` and `last_t = i64::MIN + 9`, Prometheus returns 1.625 and the clamped subtraction returns 1.25. `wrapping_sub` is Go's `int64` arithmetic bit for bit, at the same cost. Real timestamps are nowhere near that range, so the two agree on every input a query can produce; the difference only shows up where the old code silently disagreed with Prometheus. Signed-off-by: discord9 <discord9@users.noreply.github.com> Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> Signed-off-by: discord9 <discord9@users.noreply.github.com> |
||
 Lei, HUANG
|
aa4abe554a |
feat(promql): add evaluated series arithmetic and basic aggregates (#9682)
* feat(promql): add evaluated series math kernels Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): add evaluated series scalar arithmetic Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): complete evaluated series math functions Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): add series comparisons and scalar results Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): add sparse evaluated series basic aggregates Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(promql): align time-varying scalar maps on evaluated grids Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop metric name in evaluated series math like Prometheus Prometheus' simpleFloatFunc drops __name__ together with __type__ and __unit__. Check abs, ceil, floor, sqrt and round against fixtures generated by the Prometheus v3.15.0 engine. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop empty labels with the metric name in series math Prometheus has no empty-valued labels, so a kept empty value would hide a same-labelset collision from the caller. Round shares its calculation with the UDF, so check it only against the Prometheus fixtures. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): align evaluated series map kernels with Prometheus Follow Prometheus v3.15.0 simpleFloatFunc, clamp, VectorscalarBinop and vectorElemBinop: drop __name__, __type__ and __unit__ for math, arithmetic and bool comparisons; use ports of Go math (including the amd64 Log and Exp assembly) for ^, %, atan2 and the transcendental functions; empty a clamp with max < min; keep -0 in sgn and Prometheus' operation order in deg and rad. Fields are compared independently and filtered output is trimmed with GridSeriesBuilder. Check the kernels against fixtures generated by the Prometheus engine and Go math results built for amd64. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): align evaluated series aggregation with Prometheus Follow Prometheus aggregation and generateGroupingLabels: no modifier is by (), without drops __name__, empty label values are absent, sum and avg use Neumaier compensation and avg switches to an incremental mean on overflow. Fields are aggregated independently, so a field without values at a row is null also for count and group. Output is trimmed with GridSeriesBuilder. Check against fixtures generated by the Prometheus engine and drop the main UDAF oracle and the common-function dependency. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): validate whole segments in series comparisons map_compare checked sample types one by one, so a histogram or mixed segment without a present sample passed. Validate segments up front as the other kernels do, check them before an empty clamp, share the empty-label rule with aggregation, and pin exact bits in the deg, rad and incremental avg tests. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: exclude promql golden fixtures from typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(promql): point series round at Prometheus funcRound Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): keep Go math ports only where std leaves the tolerance Series math now follows the relative-1e-10 comparison rule: functions use Rust's standard library unless Go's amd64 result differs beyond the tolerance or in overflow and sign. asin, atan, tanh, asinh, acosh and atanh use std; the Exp assembly port is reduced to its overflow edge, and sinh/cosh build on it above 21. Golden and Go math fixtures are compared with almost.Equal. The Go cases are evaluated through the kernels' own per-sample functions, and each retained port must have a case where std fails, so a port that is no longer needed, or a swap back to std, fails a test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(promql): validate aggregate input like series maps aggregate_float repeated map's schema, grid-axis and float-segment checks in a different order, so one bad series could fail with a different error category in each. Both now use validate_float_series, and map_math/map_negate/map_arithmetic run it before mapping instead of their own axis check. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin avg compensation to Prometheus bits With fixtures compared within the tolerance, dropping or misplacing the Kahan compensation in avg only moves the last bits. Three Prometheus cases where it does are compared exactly: compensation on the incremental mean, the separate division of sum and compensation, and rescaling the compensation when avg switches to the incremental mean. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin the zero sign of clamp and the Neumaier swap Under the fixture tolerance, +0 and -0 compare equal and no case had an addend larger than the running sum, so two branches lost their tests: Go's Max picking +0 over -0, whose sign a later 1/x turns into the sign of an infinity, and the Neumaier swap, without which sum(1, 1e100, 1, -1e100) is 1 instead of 2. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: allow the Unicode category Nd in typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): keep aggregate state in fixed-size grid blocks Aggregation kept one B-tree entry and one Vec allocation per populated evaluation point, plus a tree lookup per sample. State is now stored in 64-point blocks keyed by absolute grid index / 64, holding one state per field and point. A block is allocated only once a sample falls in it, so sparse input never expands to the full grid, and each field's run through a block costs one lookup. Points without samples are skipped on output, which also keeps the last block from reaching past the grid. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): drop reserved and empty labels in one rebuild drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin that aggregation keeps type and unit labels Prometheus generateGroupingLabels drops only the metric name, unlike math and arithmetic. Nothing failed when grouping dropped __type__ or __unit__. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): return aggregate groups in first-seen order Groups were collected in a BTreeMap and returned sorted by label. Prometheus' rangeEvalAgg appends groups in the order their first input appears, and a nested aggregation folds them in that order: for sum(sum by (g) (m)) over 1e308, -1e308, 1e308 with g = a, c, b, Prometheus gives 1e308 and label order overflows to +Inf. Look groups up by label key and keep them in a Vec. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(promql): pin which zero min and max keep Prometheus replaces the min/max accumulator only on a strict comparison, so of +0 and -0 the first is kept. The golden tolerance treats the zeros as equal, and a non-strict comparison passed every test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
Lei, HUANG
|
50002c6830 |
feat(promql): add evaluated series math kernels (#9672)
* feat(promql): add evaluated series math kernels Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop metric name in evaluated series math like Prometheus Prometheus' simpleFloatFunc drops __name__ together with __type__ and __unit__. Check abs, ceil, floor, sqrt and round against fixtures generated by the Prometheus v3.15.0 engine. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(promql): drop empty labels with the metric name in series math Prometheus has no empty-valued labels, so a kept empty value would hide a same-labelset collision from the caller. Round shares its calculation with the UDF, so check it only against the Prometheus fixtures. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: exclude promql golden fixtures from typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(promql): point series round at Prometheus funcRound Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * chore: allow the Unicode category Nd in typos Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(promql): drop reserved and empty labels in one rebuild drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
discord9
|
a1052c0c0f |
feat(query): add raw-series preflight and execution surface to query traits (#9661)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
cbc2a729a5 |
feat(common-meta): add raw-series preflight and execution surface to Datanode trait (#9659)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
5f0f344908 |
feat(table): scan-bound typed raw series production via SeriesScanExec (#9664)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9657** (raw request types). Also requires **#9651** (scanner preflight), **#9655** (frame ordering), **#9656** (recordbatch adapter). ## What's changed and what's your intention? Add scan-bound typed raw series production to `TableScanExec`: - When the scan binding has a `RawSeriesRequest` and preflight passes, produces `SeriesScanExec` that streams typed `RawSeries` frames. - Shares MVCC/projection/filter semantics with the existing RecordBatch path. - Strictly opt-in: absent or declined preflight returns the ordinary plan unchanged. **2 files, +475/-2 production + 1506 test lines.** Note: diff includes cherry-picked commits from #9651 and #9656. Once they merge, the diff will shrink. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
18e11944c9 |
feat(promql): add native instant manipulate execution over raw series (#9660)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9654** (source materializer). Also requires **#9649** (instant eval) and **#9655** (frame ordering). ## What's changed and what's your intention? Add native instant manipulate execution over raw series streams: - `NativeInstantMapping`: preflighted mapping from typed raw series to query output schema. - `NativeInstantManipulateExec`: ExecutionPlan that consumes `SendableRawSeriesStream`, materializes labels, selects instants, and emits aligned RecordBatches. - `InstantManipulate::try_to_native_execution_plan()`: validates the raw frame schema and constructs the native plan. **5 files, +1830/-0 lines.** No callers wired yet. Note: diff currently includes cherry-picked commits from #9649 and #9655. Once those merge into narthex and the base is updated, the diff will shrink to only the 5 native exec files. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p promql` → **357/357 passed** ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
b3707c28ac |
feat(promql): merge ordered raw series inputs (#9653)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Depends on #9650** (labels decoder). Includes `materialize_raw_labels` — if the source-only PR merges first, this rebases cleanly. ## What's changed and what's your intention? Add `promql::source::merge_ordered_raw_series`: merges multiple ordered raw-series streams into one globally ordered stream by series key and time. All inputs must share the same schema and label binding, be `KeyThenTime` ordered, and have undecoded labels. The merge recomputes per-series `is_last` globally. Errors are terminal; already-emitted prefixes are not rolled back. **8 files, +4118/-0 lines (622 production + 3489 tests + 7 config).** ### Verification - Compiled and tested with #9650 merged: `cargo check --all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D warnings` ✅ - Cannot compile standalone until #9650 merges. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 discord9
|
200ed3ce36 |
feat(table): scan-bound typed raw series production via SeriesScanExec
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f354e3423a |
feat(promql): add native instant manipulate execution over raw series
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
535923629b |
feat(promql): merge ordered raw series inputs
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
83a0d58980 |
feat(query): capture native raw-series routes and extend MergeScan for typed frames (#9663)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Stacked on #9662** (native config). Also requires **#9655** (frame ordering metadata). ## What's changed and what's your intention? Add native raw-series route capture and MergeScan typed-frame support: - `CaptureNativeRawRoute`: captures region routing during planning when `experimental_native_promql` is enabled. - `NativeRawSource`: borrows the captured route for native raw-series execution. - Schema identity validation for `greptime.series.` metadata. - Wires `experimental_native_promql` from `QueryEngineState` → `DfQueryPlanner` → `DistExtensionPlanner`. **4 files, +1746/-113 lines.** Note: diff includes cherry-picked commits from #9655. Once it merges, the diff will shrink. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
440ecb909b |
feat(common-query): add raw series request binding and preflight types (#9657)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add raw series request types for the named raw-series preflight contract: - `RawSeriesRequest` / `RawSeriesTag`: versioned binding for a single-region raw-series request. - `PreparedNamedRawSeries` + encode/decode: JSON body for the `PREPARE_RAW_SERIES_ACTION` Flight action. Strict serde (`deny_unknown_fields`), version 1 only. Includes wire compatibility fixtures. **7 files, +661 production + wire fixtures.** ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
e0af59f14c |
feat(promql): materialize complete raw source labels (#9654)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) **Depends on #9650** (labels decoder). Must merge after (or be rebased onto) #9650. ## What's changed and what's your intention? Add `promql::source::materialize_raw_labels`: a stream decorator that converts raw series chunks with primary-key-encoded labels into chunks with complete explicit labels. Uses `PrimaryKeyLabelDecoder` (from #9650) to decode dense/sparse primary keys against the scan-bound region metadata snapshot. Rejects schemas with eval grids or empty fields, and requires complete (not projected) label decoding. Output always advertises `RawLabelSource::Explicit`. **5 files, +1216/-0 lines (169 production + 1040 tests + 7 config).** The merge module is intentionally excluded. ### Verification - Compiled and tested with #9650 cherry-picked: `cargo check --all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D warnings` ✅ - Cannot compile standalone until #9650 merges. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
bd9f7dc40b |
docs(promql): say what draining the histograms after an early stop is for
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
328218776d |
fix(promql): follow up the post-merge review of native histogram operators
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
discord9
|
53ef80a516 |
feat(query): capture native raw-series routes and extend MergeScan for typed frames
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
3ef71b5702 |
feat(common-query): add raw series request binding and preflight types
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
7c6595d9f7 |
feat(promql): materialize complete raw source labels
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
f8054ab297 |
feat(promql): select instant samples from shared typed fields (#9649)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Builds on the shared series types already present on `narthex` (#9598, #9599). This PR targets `narthex`, not `main`, and has no dependency on another open PR. ## What's changed and what's your intention? Extract instant sample selection into `select_instants`, operating on the existing `RawTimestamps` and `SeriesFields`. It returns source row indices and evaluation timestamps without converting the payload into a new representation. Use it in the existing `InstantManipulate` path when timestamps are non-null, strictly increasing milliseconds and the configured fields are Float64. Other inputs retain the existing path, including duplicate timestamps and Histogram columns. Selection preserves lookback boundaries, offsets, stale markers, ordinary NaNs, null samples, and the existing output-point limit. This is one extracted slice of previously local work: **5 files, +732/-3 lines, including tests**. It does not add native planning, scanner/Flight integration, configuration, storage formats, or any new Histogram type. ### Verification - Passed: scoped Rust formatting, `hawkeye format`, `hawkeye check`, `git diff --check`, and normal commit hooks. - Running on this exact extracted branch; results not yet claimed: `cargo check --locked -p promql --all-targets` and `cargo nextest run --locked -p promql --retries 0`. - Independent source review is in progress. Kept as a draft until verification and review are complete. - Full-workspace tests, full-workspace Clippy, and SQLness have not been run for this extracted branch. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
88a1e684a0 |
feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors (#9656)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Recognize `SeriesScanExec` in query metrics and preserve typed errors through the record batch adapter: - `PlanMetrics`: recognizes `SeriesScanExec` nodes so region-scoped stats attribution works. - `RecordBatchStreamAdapter`: preserves the original typed error via `BoxedError` instead of wrapping into a generic execution error. This lets downstream recover the specific status code (e.g., `InvalidArguments` instead of `EngineExecuteQuery`). **2 files, +203/-9 production + 1 sqlness expected result.** ### Error code change The typed error preservation changes the error code for the `semantic_graph` upper-bound-only query from `3001(EngineExecuteQuery)` to `1004(InvalidArguments)`. The error message text is unchanged. The new code is correct: the original error was an argument validation error, not a generic execution failure. The sqlness expected result has been updated. ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-recordbatch` → **65/65 passed** ✅ - sqlness expected result updated for the typed error code change ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
5d93b8177c |
feat(common-query): adapt ordered record batches to raw series (#9652)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. Builds on shared series types from #9598/#9599. ## What's changed and what's your intention? Add `record_batches_to_raw_series` in `common-query`: converts an ordinary DataFusion `RecordBatch` stream into contiguous raw series chunks. The adapter validates column types at construction (time must be Timestamp(Millisecond, None), tags must be string/dictionary, fields must be Float64 or native histogram), encodes series identity via Arrow RowConverter, and produces `RawSeriesChunk`s with explicit labels and contiguous ordering. Float64 fields share source buffers (zero-copy); histograms reuse the existing `histogram_samples_from_arrow` bridge. Chunking merges equal-key runs within and across batches; a key change or end-of-input sets `is_last`. Timestamps must be strictly increasing within a series. Empty batches are skipped with a bounded poll loop. **3 files, +1666/-0 lines, including 20 tests.** No production callers yet — consumed by follow-up scan wiring PRs. ### Verification - `cargo check --locked -p common-query --all-targets` ✅ - `cargo nextest run --locked -p common-query --retries 0` ✅ - `cargo clippy --locked -p common-query --all-targets -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
fb9bda2744 |
feat(store-api): add raw-series scanner preflight and ordering contract (#9651)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add two default methods to `RegionScanner` for raw-series scan preflight: - `raw_series_schema()` — non-consuming check whether this scanner can provide raw series for a given label source. Returns `None` (unsupported) by default; overrides promise `scan_series_partition` can execute. - `raw_series_ordering()` — returns the ordering guarantee without consuming scanner state. Defaults to `Contiguous` (conservative for legacy sources). Both are additive-only trait defaults with no behavior change for existing implementors. **32 insertions, 0 deletions, 1 file.** ### Verification - `cargo check --locked -p store-api --all-targets` ✅ - `cargo nextest run --locked -p store-api --retries 0` ✅ - `cargo clippy --locked -p store-api --all-targets -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
e25fa2bd8d |
Merge remote-tracking branch 'upstream/narthex' into feat/narthex-instant-eval-pr
# Conflicts: # src/promql/src/lib.rs |
||
|
discord9
|
869a32fdcb |
feat(query): add experimental_native_promql config option (#9662)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Independent PR. No dependencies. ## What's changed and what's your intention? Add the `experimental_native_promql` configuration option (default: `false`): - `QueryOptions.experimental_native_promql`: new boolean field, default `false`. - Four example TOML configs updated with the commented-out option. - `config/config.md` regenerated with the new option. The option is stored in process configuration only. Session-level override and the actual gate in query planning are in follow-up PRs. **6 files, +24/-0 lines.** ### Verification - Compiles standalone: no unused fields or methods. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [x] This PR requires documentation updates. (config.md regenerated) - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches. |
||
|
discord9
|
b27ca7c7bd |
feat(common-query): declare raw series frame ordering in schema metadata (#9655)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) No dependencies on other open PRs. ## What's changed and what's your intention? Add ordering metadata to raw series frame schemas: - `raw_series_frame_schema_with_ordering()`: builds the canonical v1 frame schema and stamps the producer's ordering (`contiguous` or `read_key_then_time`) into Arrow schema metadata. Rejects `Interleaved` (no v1 wire representation). - `raw_series_frame_ordering()`: reads the declaration back, defaulting absent metadata to `Contiguous` (legacy frames), rejecting unknown values. - `pack_raw_series` now declares ordering on every yielded batch via `RecordBatch::with_schema`. Includes a wire compatibility fixture (`tests/compatibility/wire/raw_series_ordering/`) with 4 cases. **4 files, +277/-7 lines.** ### Verification - `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅, `cargo clippy -- -D warnings` ✅ ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
2d9b2a970f |
feat(mito-codec): decode scan-bound primary key labels (#9650)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Builds on `narthex` base types (#9598, #9599, #9602, #9603). No dependency on other open PRs. ## What's changed and what's your intention? Add `PrimaryKeyLabelDecoder` to `mito-codec`: given a scan-bound `RegionMetadata` snapshot and a primary key byte sequence, decode the complete metric name and label set (or a projected subset). Supports both dense and sparse primary key encodings. Sparse keys get a new `visit_label_ids` method that validates the full tail (rejecting duplicates, reserved IDs, and truncated framing) before reading values. Dense keys check that all fields are present and decode each value with exact type fidelity. This is one extracted slice of previously local work: **4 files, +1046/-13 lines, including tests**. It has no production call site yet — it will be consumed by the raw series scan wiring in follow-up PRs. ### Verification - `cargo check --locked -p mito-codec --all-targets` ✅ - `cargo nextest run --locked -p mito-codec --retries 0` → **62/62 passed** ✅ - `cargo clippy --locked -p mito-codec --all-targets -- -D warnings` ✅ - `hawkeye format`, `hawkeye check`, `git diff --check` ✅ - Full-workspace tests and SQLness not run for this extracted branch. ## PR Checklist - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
0ce358dde5 |
feat(promql): match vectors holding native histograms (#9696)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) MATCH-HIST in internal-docs#27. Builds on #9679 (`vector_match`) and #9686 (`FloatHistogram`). ## What's changed and what's your intention? ### Problem `binary_vectors` and `set_vectors` rejected native histogram samples with `NotImplemented`. ### Change - `vector_match::elem_binop` ports Prometheus `vectorElemBinop` for histograms: - histogram `+`/`-` histogram, compacted, with `-` marking the result as a gauge; - histogram `==`/`!=` histogram, layout-sensitive like Go's `Equals`, keeping the left histogram; - histogram `*` float, float `*` histogram, histogram `/` float, compacted. - Every other pairing that involves a histogram, and a `+`/`-` between exponential and custom buckets, is skipped: no result, `bool` included, and the pair does not count as a match. This follows Prometheus, which reports these as annotations before recording the match. - The one-to-one "multiple matches" check now counts the matches actually made per group and point, so skipped pairs do not count. - Set operators carry histogram values through; results with histograms are built with `GridSeriesBuilder::push_samples`. - Operand presence no longer reads values, and the helpers that could only fail on histograms are infallible now. Review focus: - `elem_binop` against `vectorElemBinop`: which pairings keep, filter or skip. - `binary_field`: the one-to-one match counting and the ordering of the skip against it. ### Non-goals - The experimental trim operators (`</`, `>/`). - Annotations Prometheus attaches to skipped pairs or counter-reset collisions. ### Constraints None. Public signatures are unchanged. ### Verification - `histogram_operators_match_prometheus`: 280 cases (20 per operator: `+ - * / % ^ atan2 == != > <= and or unless`) from the Prometheus v3.15.0 engine over vectors mixing floats and native histograms drawn from small per-case pools, with `on`/`ignoring`, `group_left`/`group_right` and `bool` (`vector_match/histogram_cases.txt`). Results, error classes, floats within 1e-10 and histograms bucket by bucket match. - Mutation check: all 10 faults fail a test (keeping incompatible pairs as filtered matches, not skipping mixed bucket kinds, no gauge hint, no compaction, `/` as `*`, `==` keeping the right side, `!=` as `==`, ignoring `group_right`'s swap, set operators dropping histograms, `float * histogram` unsupported). - `cargo nextest run -p promql`: 359 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
8b85a3c58f |
feat(promql): match vectors holding native histograms
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
cd3292f59e |
feat(promql): evaluate range functions over native histogram samples (#9695)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) EVAL-HIST in internal-docs#27. Builds on #9673/#9674/#9677 (`RangeEvaluator`) and #9686 (`FloatHistogram`). ## What's changed and what's your intention? ### Problem `RangeEvaluator` rejected native histogram samples with `NotImplemented`, so no range function could read a series holding native histograms, alone or mixed with floats. ### Change - `RangeEvaluator` keeps the histogram samples of each field apart from its floats, drops a histogram whose sum is the stale marker, and evaluates windows holding histograms in `range_eval::histogram` following the histogram branches of the Prometheus functions: - `min/max/stddev/stdvar/quantile/mad_over_time`, `deriv`, `predict_linear`, `double_exponential_smoothing`, `ts_of_min/max_over_time` read only the floats, as before; a window without floats has no result. - `count_over_time`, `present_over_time`, `ts_of_first/last_over_time` count both kinds; `first/last_over_time` return the earliest/latest sample, histogram or float. - `resets` and `changes` walk both kinds in time order: a change of kind is one edge, two histograms compare with `DetectReset` / `Equals`. - `sum/avg_over_time` and `rate/increase/delta` give nothing for a mixed window; over histograms alone they port the Kahan sum/mean and `histogramRate` with `extrapolatedRate`'s extrapolation. - `irate/idelta` take the last two samples of either kind; a float and a histogram give nothing. - An evaluator built from a bare kernel (`with_kernel`) still rejects histograms. - `GridSeriesBuilder::push_samples` writes rows holding histograms; fields become mixed samples when they hold any. - `histogram_eval::sum_or_avg` is the histogram sum/mean shared by `aggregate_histograms` and the range functions; the aggregation still compacts its result, the range functions do not, as in Prometheus. - `range_eval::function::extrapolation_factor` is the extrapolation step shared by the float and histogram rates. Review focus: - `range_eval/histogram.rs`: each arm against the Prometheus function it names; `histogram_rate` against `histogramRate` (nulling the first sample on a reset between the first two, schema reduction, adding back at resets). - `RangeEvaluator::evaluate`: retention and the skip to the earliest sample now cover histograms too. - `GridSeriesBuilder::push_samples`: histogram indices stay aligned with the float rows across gaps. ### Non-goals - Anchored and smoothed range selectors (NEW-EVAL-MOD); start-timestamp based resets, which Prometheus only uses behind a feature flag. - Annotations (warnings and infos) Prometheus attaches to mixed or incompatible windows. ### Constraints None. `RangeEvaluator`'s public interface is unchanged. ### Verification - `histogram_range_functions_match_prometheus`: 520 random cases (20 per function, 26 functions) plus 6 fixed counter histories, from the Prometheus v3.15.0 engine over one or two series mixing floats and native histograms, with ranges of 15s to 75s and offsets (`range_eval/histogram_cases.txt`). Floats compare within 1e-10 (1e-6 for `deriv`, `predict_linear`, `stddev`, `stdvar`), histograms bucket by bucket. - Mutation check: 16 of 17 faults fail a test (comparison order of first/last, counting only floats, mixed-edge counting, histogram resets as changes, mixed sums and rates, the counter zero point, adding back at resets, nulling the first sample, the gauge hint, irate's reset check, float-only functions answering for histograms, histogram retention, builder index alignment). The remaining one, skipping the schema reduction before the subtraction, only changes rounding: a later higher-schema sample is a reset whose addition reduces the schema anyway. - `histogram_samples_need_a_function_and_drop_stale_markers` pins the stale histogram drop and the bare-kernel rejection. - `cargo nextest run -p promql`: 358 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
17f30f594a |
feat(promql): evaluate range functions over native histogram samples
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
a78d4aa313 |
feat(promql): cover absent_over_time and vector(), sort quantile_over_time like Go (#9694)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Completes SEL-ABSENT (`absent_over_time`) and SEL-SCALAR (`vector()`) from internal-docs#27, and the `quantile_over_time` limitation left by #9674. ## What's changed and what's your intention? ### Problem - `quantile_over_time` sorted windows with a stable sort. Go's `sort.Sort` is unstable for more than 12 elements, so in a long window holding both +0 and -0 the zero a rank lands on, and the sign of the result, could differ from Prometheus. - `absent_over_time` and `vector()` had no native path. ### Change - `quantile_over_time` sorts with `go_sort::sort_by` and `vectorByValueHeap.Less` (NaN first). - `absent_over_time(v[r])` is `select::absent` over the series of `present_over_time(v[r])`, with labels from `select::absent_labels` on the matchers of `v`; Prometheus derives both the same way. The rustdoc of `absent` states this, and a golden test pins it. No new kernel. - `vector(s)` needs no kernel: a scalar is already a label-less grid series with a value at every point. The `select` module docs state this. ### Non-goals Wiring `absent_over_time` and `vector()` into the planner. ### Constraints None. ### Verification - `quantile_picks_the_signed_zero_go_picks`: a 20-sample window of mixed zeros, a NaN and numbers; the sign of `quantile_over_time(k/19)` matches Go's `sort.Sort` on amd64 for k = 1..19, and fails with the previous stable sort. - `absent_over_time_is_absent_over_present_over_time`: 120 cases from the Prometheus v3.15.0 engine (`range_eval/absent_over_time_cases.txt`) with ranges of 5s to 45s and equality/inequality matchers. - `cargo nextest run -p promql`: 357 passed. `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`, `hawkeye check`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
41cfd237f6 |
test(promql): pin absent_over_time as absent over present_over_time
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
1147fafa5d |
fix(promql): sort quantile_over_time windows with Go's sort
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
dennis zhuang
|
7c8c74ac9b |
fix(promql): keep no samples past a skipped anchored window and trim Narthex operators (#9690)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Stacked on #9689; review the commits after `47f89b34b7`. Follow-up to the Narthex operator PRs merged into `narthex` (#9673–#9687). ## What's changed and what's your intention? ### Problem An `@`-anchored range selector whose one window ends before the series starts skips the cursor past the point it reads. `evaluate` cleared retained samples only when `cursor == points_read`, so every later chunk of that series stayed in memory until `is_last`, e.g. `x[5m] @ start()` over a series that starts after `start()`. Results were unaffected. The operator PRs also left duplicated helpers, a dead branch, a redundant reset, a duplicate test and comments that went stale during review. ### Change - `RangeEvaluator::evaluate` drops retained samples once `cursor >= points_read`. - `METADATA_LABELS` and `external` are defined once in `grid_series`; `range_eval`, `vector_match`, `map_time`, `select` and `histogram_eval` use them. - `kahan_inc` (Prometheus `kahansum.Inc`) is defined once in `functions`, on top of `compensated_sum_inc`, replacing the copies in `range_eval::function` and `histogram_eval::native`. - Removed: the unreachable `k == 0` return in `topk`, the field reset in `RangeEvaluator::finish_series` (`evaluate(None)` already leaves the fields empty), and `overlapping_window_spans_chunks`, whose only fault class is also caught by the random-chunk reference and the golden tests. - Fixed comments that no longer match the code: `RangeEvaluator::try_new` input, where the smoothing factors are checked, `binary_vectors` label dropping, `Results`, `rank`. ### Non-goals `quantile_over_time` still sorts with a stable sort; switching it to `go_sort` changes behavior and belongs in its own PR. ### Constraints None. No public interface changes. ### Verification - `anchored_window_before_the_series_retains_nothing` fails before the `>=` change and passes after it. - `cargo nextest run -p promql`: 356 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`, `cargo fmt --all -- --check`, `scripts/check-super-imports.py`: clean. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
d4c4da7b6e |
chore(promql): import parent items through crate paths (#9689)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Follow-up to #9673, #9686 and #9685, merged into `narthex`. ## What's changed and what's your intention? ### Problem `make fmt-check` fails on `narthex` at `4969bca60e`: `scripts/check-super-imports.py` rejects five top-level `use super::` imports in `histogram_eval/{native,float_histogram,aggregate}.rs` and `range_eval/function.rs`. ### Change The five imports use `crate::` paths. No behavior change. ### Non-goals None. ### Constraints None. ### Verification - `make fmt-check`: clean. - `cargo nextest run -p promql`: 356 passed. - `cargo clippy -p promql --all-targets --all-features -- -D warnings`: clean. - On `4969bca60e`, the rest of the full suite already passes: `make clippy`, `make test` (9950 passed; `json2::test_json2_single_mem_range_flush` failed once on a MySQL port bind and passed on retry) and `cargo sqlness bare`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [ ] I have written the necessary rustdoc comments. - [ ] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
438c44c109 |
chore: merge fix/narthex-super-imports into chore/narthex-ablation
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
4761d39690 |
refactor(promql): drop a redundant field reset and a duplicate range window test
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
72b06dc730 |
fix(promql): drop retained samples when an anchored window is skipped
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |