mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
b71676e72d3ffbe68f7cda70bbab32b35e6f1d10
5558
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dennis Zhuang
|
b71676e72d |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile/tests.rs |
||
 discord9
|
3b09dd75dc |
feat(promql): run a bound tree over raw series in one fused node (#9820)
* feat(promql): run a bound tree over raw series in one fused node
`FusedSeriesExec` evaluates a compiled `NativeTree` directly over the raw
series its child produces, instead of materialising a row batch through the
per-operator chain. It binds the time column, the fields and the tags once,
checks the input against that binding when the plan is built, and converts the
chunks the tree settles back into the statement's output columns.
The node's partition count is the child's, one tree per partition. The child
must be a raw-series scan that assigns every series whole to one partition:
`PerSeries` distribution is what makes a per-partition evaluation equivalent to
evaluating the whole series. A zero-partition input is refused; anything else
is the child's business.
A label set can recur across partitions once `rate` and the map kernels drop
`__name__`, so the occupancy map is plan state rather than per-stream. Two
series reaching one label set is an error only when their occupied grid points
overlap, which is Prometheus' range-query rule; disjoint points are legal and
are merged into one occupancy record. The reservation for that map is attached
to the first execution's pool and released when the plan is dropped, because a
later partition still has to collide with what an earlier one emitted.
`occupied_runs` reads a chunk's absolute positions rather than assuming one
contiguous run: a sparse evaluated chunk may hold several, and treating the
first as the whole range would count the holes as rows.
A poll does a bounded amount of work and wakes itself when the budget runs out,
so a long run of chunks that settle nothing cannot monopolise the executor
thread.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): charge for the label sets the plan keeps, not for one stream
The occupancy map is the plan's: entries are inserted by whichever partition
sees a label set first and no path removes them. The reservation paying for
them was the stream's, so the bytes went back when that stream ended while the
entries stayed. Measured on a plan with three partitions producing three label
sets: each grew 42 bytes, each stream's end returned them, and the pool then
read 0 while every entry was still there and still colliding with later
partitions. The plan's own drop produced no shrink at all.
So a label set's bytes were charged for part of the time and invisible for the
rest, and the invisibility grew with the number of label sets a query touches.
The charge now belongs to the map it pays for: `LabelSets` holds the entries
and the reservation together, `attach` registers it once on the first
execution's pool, and it is released when the plan is dropped.
`attach` is idempotent because every partition calls it. Re-registering would
move a charge the other partitions keep growing, and would give it back when
one stream ends while the entries stay - which is the shape of the bug.
A regression test states the invariant: after the first partition's stream
ends, the pool is still charged for its label set; a second partition with the
same set on disjoint points adds no further charge; dropping the plan returns
it to zero.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): take a raw series frame, not a row batch, and keep its series whole
Review found five things; this is all of them.
The node read ordinary Arrow rows and converted them itself, so a real
`SeriesScanExec` could never feed it: a frame's columns are `read_key`,
`is_last`, and the `time` and `fields` structs, and none of them is the time
column, a tag or a field the binding names, so construction refused it. It now
takes a raw series frame the way `NativeInstantManipulateExec` does - the frame
is validated when the plan is built and unpacked per partition when it runs -
and the row-to-series mapping is gone, because turning rows into series is a
different node's job. The series schema comes from the tree's own input node,
so there is one statement of it rather than two that could disagree.
A raw frame is a per-series stream, and this node holds one window per series
and closes it on the chunk that marks the end. Declaring that as
`KeyPartitioned(read_key)` keeps the optimizer from splitting a series across
partitions, which it would otherwise be free to do; `read_key` is the series'
primary key as a binary column, so hashing it groups exactly the rows of one
series. A round-robin split would have let each partition rate half a series
and answer half the truth; with the key declared, a series that somehow did
arrive in two pieces is refused by the frame validator instead.
Two label-set rules were wrong. Occupancy was tracked per position for any
field, so two series that each hold a different field at the same point looked
like a collision, while the rest of the stack merges exactly that as
complementary fields - it is now tracked per field, and only one field holding
two values at one point is refused. And the key encoded every tag including
empty ones, while `grid_series::label_set` drops them, so `{host="h", zone=""}`
and `{host="h"}` were two keys here and one everywhere else; the key is now
that same normalized label set.
Finally, a label set already in the map grew its run arrays on every merge
without telling the pool. The charge now follows the capacity the merge adds,
so a long-lived plan that keeps meeting one label set pays for what it holds.
The test that asserted the opposite was wrong and is corrected.
Signed-off-by: discord9 <discord9@163.com>
* fix(table): declare the distribution a raw series frame already has
`SeriesScan` guarantees every series arrives whole in one partition - its own
docs say so, and both modes enforce it in different ways: the two-phase
collector places a series by its tsid, and the legacy divider only emits a
batch once it holds all of a series' rows. But the leaf built for a raw frame
declared `UnknownPartitioning`, so nothing on the plan said it.
That mattered to a consumer holding one window per series: with the
distribution unspecified an optimizer is free to round-robin the partition,
and two halves of one series would each look final to their own stream. So the
leaf now declares `Hash(read_key)`. `read_key` is the primary key as a binary
column, so hashing it groups exactly the rows of one series and splits nothing;
`FusedSeriesExec` asks for the same key and is satisfied without a shuffle.
The three tests that pinned `UnknownPartitioning` now pin the key instead.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): keep the capacity a label set already paid for
`merge_runs` built a fresh vector sized to `stored + new` and replaced the old
one. When adjacent runs joined - or when a claim added nothing to a field,
which is what two complementary label sets do to each other - the entry came
back smaller than it was. The charge in `claim` is the growth over the
capacity, so the next merge into that field looked like growth again and was
charged for it, every time, while the entry itself never held more than a
couple of runs. A long query meeting one label set repeatedly would have paid
for the same few bytes over and over and could exhaust a pool it never used.
The merge now keeps whichever capacity is larger, and the charge follows that:
a merge that adds nothing to a field costs nothing.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): run the fused node over one series stream
`FusedSeriesExec` held one window per series and closed it on the chunk carrying
`is_last`, so it can answer a series only if nothing splits that run and nothing
puts part of it in another tree. Those are two different promises, and only the
first is a property of a stream: the frame decoder refuses a chunk interleaved
with another series', but nothing it can see says that half a series is in a
partition it never reads.
The node used to ask for `KeyPartitioned(read_key)` instead, expecting the
optimizer to keep whole series per partition. That does not hold: DataFusion
considers two hash layouts with the same count co-partitioned, and above
`subset_repartition_threshold` children it adds a hash exchange even when the
requirement is met (`enforce_distribution.rs`, the
`!partitioning_satisfied || target > child` branch). A hash exchange does not
preserve order, so series chunks interleave and the second half of a series is
answered as a series of its own.
So the count is now fixed at one, in `try_new` and in the reported properties,
and `benefits_from_input_partitioning` is false so no optimizer splits it for
parallelism. This is the phase-1 contract: reading a series spread over
partitions needs an exchange above this node, which does not exist yet.
Merging #9786 also moved `unpack_raw_series` to two arguments, with the frame
declaring its own label layout and ordering; the calls follow it, and the plan's
binding is checked against that declaration with
`ensure_frame_matches_label_source`.
Signed-off-by: discord9 <discord9@163.com>
* Revert "fix(table): declare the distribution a raw series frame already has"
The declaration was false. `Partitioning::Hash(read_key, n)` claims DataFusion
placed rows by its own hash of `read_key` mod `n`, and DataFusion treats any two
equal-count `Hash` layouts as co-partitioned, so a future co-partitioned consumer
would pair partitions that hold unrelated series. `SeriesScan` places a series by
tsid range (two-phase) or round-robin (legacy), neither of which is that hash.
It also did not buy what it was added for. `FusedSeriesExec` now takes one input
partition and reports `benefits_from_input_partitioning = false`, so nothing
repartitions between the scan and it, and the ordering guarantee the declaration
was meant to carry is the frame's own.
This reverts commit
|
||
|
discord9
|
a8737d0996 |
test(promql): skip an endpoint the platform cannot represent (#9894)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1205c18496 |
feat(promql): execute Complete aggregates in the native runtime (#9873)
* feat(promql): execute a bound native tree node by node, over sparse evaluated series Ports the native runtime onto the sparse evaluated-series API: a chunk is addressed by a `TimeAxis` that may hold several runs, so a consumer reads positions through the axis instead of assuming `start + row`. The driver and both node kinds move to that API: - `chunk.range().len()` becomes `chunk.series.time.len()`, the compact row count, in every metric and byte estimate. - `occupied_runs` walks `time.positions()` with one field cursor, so a chunk of several runs records each run where it belongs. A run open across a hole now ends where the hole begins; leaving it open to the last row made two runs look like one and refused a series that occupies the hole. - A window keeps the pieces its evaluator decided in a queue: one push can decide several runs, and reporting only what the push returned stranded the rest until the next push. Queued pieces are charged by `retained_bytes`, and the charge is recomputed when one leaves, so a released piece is not held. - `map_chunk` answers `Option`: an operation that empties every position (`clamp` with `max < min`) yields no chunk at all, unless the chunk closes its series -- a closing chunk holding no row is handed over so a consumer can tell "nothing more here" from "no more input". `RangeEvaluator::retained_bytes` is restored with the three byte helpers it reads: a field's buffers, a series' identity and the pending batch's vectors, all by capacity, and still counted as storage rather than as a peak over a push. Tests read absolute positions from the axis (`decided` no longer adds a row number to a start), and two sparse cases cover what the old reading got wrong: a `Window -> Map -> Window` chain over a chunk of two runs, and a series that occupies the hole between another's runs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop the window queue the sparse evaluator never needed `RangeEvaluator::push` and `push_series` each return `Result<Option<EvaluatedChunk>>`: the evaluator gathers every run it decided into one chunk and hands back at most one. So `pending.extend(decided)` inserted at most one chunk and `take_pending` removed it on the spot, leaving the queue empty after every successful push. That makes the queue dead weight: a `VecDeque` allocation per productive window, an accounting term that is always zero, a second drain site in `pull`, and doc comments claiming a problem the API does not have. The driver's "ask the input again when a window decides nothing" loop was already the whole story. Back to returning the evaluator's own answer, and drop `pending`, `take_pending` and `take_window_output`. Covers the empty-map protocol this port introduces while it is here: an operation that empties every position hands on nothing while its series is open, and one empty chunk when it closes. A non-closing empty chunk would be rejected downstream and a dropped closing one would leave the series unclosed. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the empty-map output count at one The test checked that every output was empty and that exactly one closed, which a stray extra non-closing empty output would also satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge the memory the native evaluator actually retains Four omissions made the reservation blind to most of what a window over native histograms keeps, and the existing incremental-equals-recomputed test could not see any of them because both sides used the same formula. - `FloatHistogram` owns four heap allocations -- both bucket lists, the custom bounds, and the span layout -- that `FieldSamples::bytes` never looked at. A retained 4096-bucket sample is tens of kilobytes against a ledger that saw a struct header. `heap_bytes` counts them, and both the retained sample slice and the histograms a row boxes are charged for what they own. - Queued chunks were estimated as one float per field per row, which is not what a histogram field holds: it holds indices into a shared `Arc<HistogramBatch>` and the buffers behind the Arrow arrays. The figure now walks the samples, and counts a shared table or a shared buffer slice once per pass rather than once per chunk that references it. - `history_bytes` included `size_of::<History>()` while the map slot term charged a `History` per entry as well, so every entry paid two headers. The per-entry figure now covers only the heap the entry owns. - The primary key is charged for the allocation its `Buffer` keeps alive rather than the bytes it shows: a scan slices the key out of a decoded dictionary-values buffer, so a short key can pin a batch-sized allocation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep the source queue's charge incremental The queue's reservation was recounted by walking every queued chunk on each enqueue and each pop. `push_raw` and `push_evaluated` are public and put no bound on how deep the queue gets, so a caller that queues a whole series before pulling it paid `N(N+1)/2` chunk visits to hold `N` chunks -- the same shape of defect this branch fixes in `absorb`. `QueuedCharge` keeps a reference count and the bytes behind each shared allocation: a table or buffer is charged when the first queued chunk reaches it and given back when the last one leaves, so an operation walks the one chunk it names. The count is per chunk rather than per field, because a pop re-walks the same chunk with the same dedupe and must remove exactly the references that were added. A chunk's own struct bytes are counted per chunk (`own_bytes`); only the allocations behind its fields are counted by address, since a struct has no address to dedupe on. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): append a forward-moving chunk to the emitted history `absorb` rebuilt the whole stored run array on every non-empty chunk. Legal input made that quadratic across chunks with no label collision: one series in `K` chunks, chunk `i` occupying only grid position `2*i`, stores one more run per chunk and moves `K(K+1)/2` of them -- about five billion for a hundred thousand rows. A chunk that starts at or past the end of the last stored run only ever adds to the end, so its runs are appended and the boundary run is joined where it touches. One that can interleave or reach backwards still takes the full one-pass merge, which can split a run. Both leave the runs sorted, disjoint and joined, which is what `Emitted::occupy` looks a position up by. The test records the chunks through `Emitted::occupy` itself and reads the history back from it, covering the forward case, the touching boundary, a hole that has to join both neighbours, and a position before everything stored. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): report a truncated input from a tree with no window `close` only asked windows whether a series was still open, so a tree of an input alone, or of a map over one, accepted a final chunk without `is_last` and handed the consumer a series that never closed. The input node now follows the series of the chunk it last handed over, and closing such a tree reports the truncation against it, which is the same refusal a window makes for the input it reads. `occupied_runs` read a row the axis names with `expect`, which panics on a `Series` whose fields are shorter than its axis; the fields are public, so that chunk is constructible by a caller. It now returns the misalignment as an error naming the node, the field and both lengths. The chunk protocol says one series closes once, which reads as one close per label set. State that it is per input series: a node that rewrites labels can hand two series over under one label set and both of them close. Also say in `Emitted`'s doc that the table is one node's own, so a label set costs its bytes again in every node that keeps one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute an instant selection as a tree node The tree could not run `NativeExpr::Instant`: the state builder recorded `not_implemented` and refused the whole tree before reading a chunk, while the compiler already produced the node for a bare selector. A query the plan and the compiler both accept therefore had no runtime path at all. `InstantRuntime` buffers one series to its `is_last` chunk -- the selection kernel reads a complete axis -- then calls the shared `select_instants` once and gathers the rows it picked onto the binding's own grid. Positions come from the grid lattice the kernel was asked for, and a timestamp off that lattice is an error rather than a skipped row. A series that selects nothing still closes, with an empty chunk on the grid; an input that ends inside a series is reported on `finish()`. The node does not record label-set occupancy: it keeps the metric verbatim, so it cannot map two series onto one label set, and the occupancy key drops empty-valued tags -- two genuinely distinct series would be refused as a collision. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved input, and close a label set only for its owner Two series-level identity holes, both reported against `42ea525`. The truncation flag followed the last chunk handed over, which is the same as "is a series open" only while at most one series can be open. A tree with no window still accepted `Interleaved`, and `a` open, `b` closed, end of input then passed while `a` never closed. Tracking the open set would need an identity that a raw chunk in bound-label mode does not have (`metric` is `None`, the key holds it), so the ordering is refused when the state is built instead, for every tree shape and not only under a window. The flag's doc now says what it means and why that is the whole answer. `History` closed on any series' `is_last`, so with `abs` dropping the metric name a series that contributed nothing could close a label set another series was still filling: `a{job="x"}` value, `b{job="x"}` empty close, `a{job="x"}` continuation was refused with "already closed the label set". Contiguous input hands over exactly that sequence, so it was not an interleaving artefact. A history now remembers the series that filled it and closes only for that series' closing chunk; a series that produced nothing under an open set is passed over, as it already was under a closed one. The owner is charged as part of the entry. The ordering check also reads every input node rather than only the first: a tree with two inputs would otherwise have its second declaration read and dropped. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve the instant gather's workspace and keep the histogram start Two findings from the review of the instant node. The gather allocates after the last chunk arrives: the concatenated axis, the descriptors that read it, the two arrays the selection returns, and the rows the builder keeps. Nothing asked the pool for any of it, so with one sample selected by every point of a million-point grid the node took tens of megabytes while its reservation fell to the buffered chunks' size -- nothing at all, on the chunk that ends the series. `gather_bytes` now derives a bound from the values in hand (rows, chunks, segments, fields, points, and the widest histogram row per field, since one row can be selected by every point), and the reservation grows by it before the gather runs. It shrinks back on both the success and the error path, so the workspace never outlives the call. A selection reads an existing sample, so it has to carry that sample's payload. `FloatHistogram` had no `start_timestamp` and `push_to` wrote `None`, which lost it for any histogram that had one: `detect_start_timestamp_reset` reads that field to spot a counter reset, so a window over this output would mis-detect resets. The field is now carried through a view and written back; the kernels that compute a histogram instead of reading one leave it `None`, which is what they produced before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): describe the gather test by what this branch does The doc on `an_oversized_selection_is_refused_by_the_pool` said the kernel has no point limit any more. It does on this branch: the limit is removed on its own branch, which this one does not carry, so the comment described a state that has not arrived. What is true here is the order: the estimate now applies to every gather, so the pool is asked before the kernel is, and the pool's refusal is the one the caller sees. That is worth saying on its own -- a bound in bytes on the path that decides the points, rather than a count that says nothing about what they cost -- and it stays true after the limit goes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute vector binary nodes in the native runtime Checkpoint the vector-vector runtime and single shared matching kernel. Keep addressed input delivery, whole-input buffering, nested composition and error cleanup together. The existing arithmetic/comparison implementation remains the only computation path. Local review checkpoint only, not ready for publication: all-targets check passes; the current run has three new fixture failures. Independent review requires current-live workspace resizing, field-local matched storage and kept-value merge budgeting before approval. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): infer the binary result fixture element type Drop a redundant complex type annotation flagged by the actual strict clippy run. The collection and literal assertions are unchanged. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve binary workspace along the shared kernel Remove the separate matching prepass and reserve current live workspace at the existing kernel allocation boundaries. Release per-sample and per-field scratch after ownership ends; charge actual retained rows, values, histograms and errors. Unbudgeted public and set callers bypass accounting without another computation path. Validate with cargo fmt -p promql -- --check; cargo check -p promql --all-targets; cargo nextest run -p promql --retries 0 --no-fail-fast; cargo clippy -p promql --all-targets -- -D warnings. The public nested Binary test also fails on swapped left/right operands and passes after byte-identical restoration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): scope binary reservations to live scans and output rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge binary buffers at actual allocation boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): skip accounting ledger for unbudgeted vector merges Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): simplify native execution memory estimates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): match binary operands by logical series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): cover flattened operand descriptor capacity Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge rewritten logical series before window evaluation Accept disjoint closed-source fragments under one output label set. Retain occupancy checks and merge affected window inputs once at EOF using shared typed sample segments. Keep direct input windows and root maps streaming, and release collected fragments on finish or error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): clarify approximate merge workspace estimate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate series boundaries before collected input merges Keep each collecting window's open metric and last position so EOF grouping cannot hide reversed runs or identity changes before closing. Preserve source and root-map piece semantics, and test malformed input release, empty closing chunks, and EOF truncation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): supply the default binary fill in the unimplemented-node fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): end a subtree before the node above it computes A window that collects its input answers only once its own input is finished, so ending the input now flushes the deepest ready window and lets the following pull carry its answer up; a binary node computes only when both operands are settled. Previously a binary node computed on an empty operand and marked itself computed, dropping what the window produced afterwards. Also stream the answer instead of collecting it: end_of_input() ends the input, and each pull runs the next stage, so the caller pulls the result and the last None closes the execution. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve an instant selection for the points it decides The gather reserved the whole planned grid, while select_instants clips the range to the series' first and last sample and its lookback: a week-long query at a one-minute step reserved ~504 KB for a series that produces five points, and a small pool refused it before gathering. Reserve for the clipped candidate count, derived with the same alignment the kernel uses. Also pin the start-timestamp semantics the histogram change spread to: last_over_time and first_over_time hand back the sample they selected with its start timestamp, and limitk and limit_ratio keep it on the sample they keep, so a reader downstream can still place a counter reset. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): reserve binary memory per phase, not per value The binary kernel adjusted the pool and the metrics once per matched point, per filed row and per merged row, so a million plain float points meant roughly three million reserve calls inside a numeric loop. Reserve at three boundaries instead: the operands as chunks are retained, one figure per phase (pairing, sort, merge) with an estimate over rows/fields/segments/payload, and each result buffer when a block of it is allocated. Delete the code that modelled another component's allocator: index_column_growth and its shadow index_capacity, BinaryOperand::ends_growth, and the flatten-capacity model in assembly_bytes. Nothing pre-reserves the left-times-right product and no large allocation happens before its reservation. Tests that pinned the callback sequence or exact small-object arithmetic go; the resource properties stay: a genuinely large input or output is refused, a sparse small result is admitted, and reservations are released on finish, error, truncation and drop. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): move the histogram_quantile doc back to its own test The start-timestamp test was inserted into the middle of another test's doc comment, leaving the earlier test without one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): compile a constant, a negation and a scalar-vector operation Three shapes reach the tree now: a constant binds as a `Scalar` on the statement's grid, a negation binds as the multiplication by `-1` it is, and a binary operation between one vector and one constant binds as a `Binary` node whose output is the vector side's own binding. The arithmetic and comparison operators all map, and `return_bool` is carried as its own field so `x > 0` (which filters) and `x > bool 0` (which reports zero or one) are one node with one field changed. The operands keep the order they were written in, because most operators are not commutative: `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them apart. A constant is bound by the same `constant_value` fold a call's parameters already use, so `x * (1 + 2)` carries `3` rather than an addition the tree would have to know how to run. Three shapes are refused rather than approximated. An operation between two vectors needs each `Input` to name the selector it reads, which the one binding does not carry, so `a + b` and `a + a` would compile to the same tree; a set operator combines two vectors' series and has no node at all; and a statement whose value is a scalar has nothing to scan, so a constant is only bound next to a vector that gives it something to broadcast over. Each refusal names what it found. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
Dennis Zhuang
|
b67aa655ac |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
dennis zhuang
|
6c7e534b92 |
feat(promql): run histogram_quantile and histogram_fraction in the native vector node (#9853)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, histogram category (#25). Stacked on #9852 → #9851 → #9850; its own change is the last two commits. Draft until those merge. ## What's changed and what's your intention? `histogram_quantile(q, v)` and `histogram_fraction(lower, upper, v)` compile and run natively, with constant parameters folded by the compiler. - **Node.** A classic histogram is read from the bucket series that share its labels, so these need the whole vector at a point, like the select node's other operators. They run there as `BoundSelect::Histogram(HistogramFunction)` through `evaluate_histograms`. The node now covers selections, sorts, `absent` and these; its name predates that and could become something like `Vector` in a follow-up. - **Pieces.** The kernel groups by label set and takes one native histogram series per set. Pieces of one label set (an empty tag and an absent one, or what a map folded) are therefore merged with `merge_same_labelset` first, and two pieces holding one point fail as they do elsewhere. Without the merge a second native piece was silently dropped and overlapping classic buckets were summed. The output holds one series per label set, so the node does not mark it as holding pieces. - **Deterministic order.** `evaluate_histograms` iterated its groups through a `HashMap`, so the result order varied between runs. Groups now keep first-arrival order, as `merge_same_labelset` does. - **Memory.** The estimate gets its own arm: the merge workspace, per-series group entries and cursors, per-row buckets and results. ### Tests - Runtime: classic buckets of one label set give the quantile without `le` and the name; two native pieces of one label set are answered at both points (fails without the merge). - Compiler: both calls bind with folded constants (`1 - 0.1` → `0.9`). ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
d7d9d9ed5e |
fix(promql): admit the label a label change writes before building it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bfe1bab672 | chore: merge narthex into feat/narthex-select-absent-sort | ||
|
Dennis Zhuang
|
6e5137f364 |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs # src/promql/src/native/execute/tests.rs # src/promql/src/native/plan/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
dennis zhuang
|
fbd5c8b850 |
feat(promql): run topk, bottomk, limitk and limit_ratio as a native select node (#9850)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, Select category (#25). Based on `narthex` directly, independent of the #9845 → #9849 chain. Compiler wiring is not here: it goes through the aggregate compile in #9833, which keeps `topk` and friends out of `AggregateKind` as a different shape. It follows once #9833 merges. ## What's changed and what's your intention? `topk`, `bottomk`, `limitk` and `limit_ratio` get a node and a runtime. - **Plan.** `NativeExpr::Select { input, op: BoundSelect }`, with `BoundSelect::{TopK { k, top, group }, LimitK { k, group }, LimitRatio { ratio }}`. Constants are bound, as for windows. A selection consumes and produces the same evaluated vector. - **Runtime.** Which samples are kept at a point depends on every series there, so the node works like the binary node: it retains its whole input, assembles the chunks of each logical series, runs the kernel from `select.rs` once the input has ended, and queues the results. - `limitk` reads series in label order. Prometheus uses label order once a query spans head and blocks; the scan here does not order distinct series, so this keeps the result stable across runs. - Pieces of one label set (e.g. under a map that dropped the name) stay pieces. They hold disjoint points, so ranking them apart picks what ranking the merged series would, and the output keeps the input's `holds_shared` flag. - Multi-field `topk` / `bottomk` / `limitk` is refused when the execution is built. - Memory: the retained chunks and the assembly are reserved as for the binary node, and a rough kernel workspace estimate (`selection_workspace_bytes`, same convention as `merge_workspace_bytes`) is reserved before the kernel runs. - **Shared code.** `BinaryOperand` becomes `RetainedInput`. The continuation and still-open checks move from `BinaryRuntime::retain` / `closed` into `RetainedInput::retain` / `ensure_closed`, so both nodes use them. Binary behavior is unchanged apart from error wording. `compute_binary`'s queueing is factored into `queue_results`. - **Kernel fix.** `topk` / `bottomk` skipped nothing and failed on a histogram sample; Prometheus ignores histograms there (`aggregationK`), and so does the kernel now. Shared files touched: `native/plan.rs`, `native/execute.rs` (`RetainedInput`, `NodeTag`, build/settle/pull), `native/display.rs`, `native/metrics.rs` (`walk`). ### Tests - Runtime: `topk` ranks a series delivered in two chunks as one candidate and gives the reservation back; `limitk` takes label order over arrival order, with map pieces kept where they hold the point; a binary node above a selection reads what it kept. - Kernel: `topk` skips a histogram sample. - Plan: a selection over raw samples is refused. Display: the three shapes. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
ca3d57cbaa |
feat(promql): compile and run date functions over a vector in the native tree (#9846)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, time/date category (the first small draft agreed on #25). Builds on #9845 (merged), whose `map_math` skips native histograms. ## What's changed and what's your intention? Date functions that take a vector — `day_of_month`, `day_of_week`, `day_of_year`, `days_in_month`, `hour`, `minute`, `month`, `year` — now compile into the native tree and run there. - **No new node.** Prometheus evaluates these like its math functions (`dateWrapper` has the same shape as `simpleFloatFunc`): per float sample, the value read as Unix seconds via `int64`, histograms skipped, the metric name dropped. So `MathOp` gains `Date(DateFunction)`, and the existing `Map` node, display and runtime (`map_chunk` → `map_math`) carry it. Shared files change in a few lines: one variant and one match arm in `map.rs`, one display arm, and the name table in the compiler. - **Compiler.** `math_signature` lists the eight names as unary. A date function written without an argument (`hour()`) reads the evaluation time; it has no input to map, so it is refused as `NotImplemented`, and the message says so. That form waits for the no-input and scalar output interfaces. - **One implementation.** `map_date` had no caller outside its tests, so it is removed. The Prometheus golden for date functions now drives `map_math(MathOp::Date(..))`, the path the runtime takes. A hole in a float field stays a missing row on the input's axis, as with every map, instead of cutting the axis. - **Comments.** `map_chunk`'s comments now account for #9845: a chunk loses the rows only histograms filled, and `None` means no float value is left. Output and labels: a `Map` keeps the input's fields and grid. It drops `__name__`, `__type__` and `__unit__`, so it is one of the nodes that can merge series into one label set, as before. ### Tests - Prometheus golden (`map_time/prometheus_cases.txt`, existing): 100 date cases, including NaN, ±Inf and label set collisions, now through `map_math`. - Compiler: each date name binds its kernel, and `hour()` is refused as `NotImplemented` naming the evaluation time. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 Yingwen
|
6a40c0b811 |
feat(mito): implement columnar samples format (#9887)
* docs(mito): specify columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): implement columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): address columnar samples review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * style(mito): use crate-rooted samples imports Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Dennis Zhuang
|
3ba4476b2f |
chore: merge feat/narthex-select-compile into feat/narthex-histogram-quantile-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
bd719b0859 |
chore: merge feat/narthex-select-absent-sort into feat/narthex-select-compile
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
f067b252da |
fix(promql): read the grid length from its spec now that timestamps are deferred
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
05442cc722 | chore: merge feat/narthex-select-node into feat/narthex-select-absent-sort | ||
|
Dennis Zhuang
|
4976ffa9b9 |
chore: merge narthex into feat/narthex-select-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs # src/promql/src/native/execute/tests.rs |
||
|
Dennis Zhuang
|
59a57d1ff6 |
chore: merge narthex into feat/narthex-date-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs |
||
|
dennis zhuang
|
c209feda3c |
feat(promql): count native histograms in count_values by their Prometheus text (#9855)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native operator kernels (#27). Builds on the Go float formatting from #9840 (merged). Removes the histogram caveat noted in #9854. ## What's changed and what's your intention? Prometheus `count_values` counts a native histogram sample under the text `FloatHistogram.String()` writes it as. The native kernel refused histogram input with `NotImplemented`; it now counts histograms the same way. - `go_format::format_float_g`: Go's `%g` / `%v` for a float64 (shortest digits, exponent below 1e-4 and from 1e6 on). `format_open_metrics_float` now builds on it, unchanged. - `histogram_eval::histogram_string`: Go `FloatHistogram.String`. Count and sum come first, then the non-empty buckets from the lowest up (negative, zero, positive), each written as `Bucket.String`. Bounds stay as the bucket index gives them (not clipped by the zero bucket), with the inclusiveness `baseBucketIterator.at` gives, so custom buckets print `[-Inf,b0]` first. - `count_values` keys a histogram by its text, so equal histograms share a group. Series are validated with `map::validate_grid_series` (from #9845), which accepts any sample type. ### Tests - Golden `histogram_eval/string_cases.txt` (17 KB, 47 cases) from `promql-compat/cmd/narthex-golden-histstring`, Prometheus v3.15.0. Fixed cases cover the zero bucket at threshold 0 (`[-0,0]`), negative bucket order, the exponent form, custom bounds, ±Inf and NaN, a zero-length span, a NaN bucket count, and buckets under a wide zero threshold; 40 random cases follow. Dropping the negative-bucket reversal or the custom-bucket inclusiveness fails it. - `count_values` groups two equal histograms together and a float apart. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
14f0ead84c |
feat(promql): execute constant scalar broadcasts in the native executor (#9859)
Execute compiler-emitted literal Scalar/vector arithmetic, filtering and bool comparisons, and unary negation in NativeExecutor. Reuse existing ScalarOperand and map kernels while pulling only the vector side; preserve sparse/missing values, correct labels, and series closure. Reuse existing reservations, metrics and merge handling. Track name-dropping collisions, merge disjoint collapsed series before a downstream Window, and retain no vector stream in the broadcast. Reject unsupported computed-scalar operands at construction. Add real compiler-to-executor and direct executor regressions for both operand directions, folded bool constants, exact 21-point results, streaming/collecting Window lifecycle, nonclosing empty filters, final closure, bounded retention and reservation release. No duplicate compiler lowering, dynamic scalar/scalar-scalar execution, Aggregate runtime or public Fused entrypoint wiring. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Yingwen
|
cff91d99bf |
feat(mito): implement columnar series index format (#9841)
* feat(mito): implement columnar series index format Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito): remove columnar compatibility fixtures Signed-off-by: evenyag <realevenyag@gmail.com> * refactor(mito): use recorded artifact sizes and inline columnar codecs Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): bound columnar series writer memory Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
discord9
|
bf97de2ad1 |
feat(promql): compile subqueries with deferred grid materialization (#9835)
Compile range functions over explicit/default-step, offset, anchored, and nested subqueries using the shared range-evaluator grid calculation. Retain complete sampled coverage and reject empty child grids explicitly. Keep compilation allocation proportional to plan structure, not timestamp count: validate deferred root and child grids; preserve eager EvalGrid::try_new; materialize shared timestamps fallibly only when a consumer requests them. Migrate Rust timestamp-buffer callers to Result without changing wire or persisted formats. Regression coverage includes grid phase/coverage, rounded sampled endpoints, offset overflow, huge and nested compile-only grids, sparse gathering, sharing, and allocation-error propagation. Deferred compilation does not bound execution memory. No public Fused entrypoint or Scalar runtime is added here. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
10dc8940f6 |
feat(promql): bind the remaining range kernels to their PromQL names (#9834)
* feat(promql): bind the remaining range kernels to their PromQL names `RangeFunction` already names 26 functions and every one of them answers: the kernels for `first_over_time`, `mad_over_time`, the four `ts_of_*` functions, `predict_linear` and `double_exponential_smoothing` are all in `range_eval`. What was missing was the compiler's name table, so these queries were refused as names with no kernel although the kernel existed. Six of them are the functions Prometheus marks experimental. This crate implements them unconditionally — `RangeFunction` names them without a gate and the kernels compute them — so binding them is the same decision as binding `rate` rather than a new policy. If a gate is ever wanted it belongs where a query is admitted, not scattered across the compiler's name table. The two parameterised ones needed a second look at where the selector sits, which is not one place in PromQL: `quantile_over_time(0.9, x[5m])` writes its quantile before the selector, while `predict_linear(x[5m], 300)` and `double_exponential_smoothing(x[5m], 0.5, 0.1)` write theirs after it. A single "parameters come first" rule read the selector as a constant for the second group and rejected an ordinary query, so the signature now states the selector's position as data instead of relying on a convention. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
discord9
|
3876831342 |
feat(promql): compile sum over an optionally grouped expression (#9833)
* feat(promql): bind every aggregator that folds a group's values
`AggregateKind` named only `Sum`, so the compiler bound `sum` and refused every
other aggregator by name. Six of the ones it refused fold exactly as `sum` does —
each reports one value per group per grid point — and two more (`stddev`,
`stdvar`) plus `quantile` fold the same way without a kernel yet. Binding them is
the same decision as binding `sum`: the compiler says what an aggregate means, and
the runtime refuses a kind it cannot execute. A kernel landing later then changes no
binding.
`quantile` is the one of the nine with a parameter, and it carries it in the
variant (`Quantile { quantile: f64 }`) as `RangeFunction::QuantileOverTime` does,
because `BoundAggregate` has no field a constant could go in and a field that meant
nothing for the other eight kinds would split a kind from its own parameter. That
costs `AggregateKind` `Eq`, which nothing on this tree needed: the only exhaustive
match is the display, and no collection is keyed on it.
The five that do not fold are refused with the reason rather than mapped to a kind
that would compute something else: `topk`, `bottomk`, `limitk` and `limit_ratio`
select some of the input's series, so they do not report one value per group per
grid point, and `count_values` writes one output label per distinct value.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit f1d1f3124ede003ae6df18fa0617aa13a56f3907)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
# Conflicts:
# src/query/src/promql/native_compile.rs
# src/query/src/promql/native_compile/tests.rs
* test(promql): keep folded argument regressions in their compiler layer
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
|
||
|
discord9
|
70201c9cf9 |
feat(promql): compile constants, negation and scalar-vector arithmetic (#9832)
* feat(promql): compile a constant, a negation and a scalar-vector operation Three shapes reach the tree now: a constant binds as a `Scalar` on the statement's grid, a negation binds as the multiplication by `-1` it is, and a binary operation between one vector and one constant binds as a `Binary` node whose output is the vector side's own binding. The arithmetic and comparison operators all map, and `return_bool` is carried as its own field so `x > 0` (which filters) and `x > bool 0` (which reports zero or one) are one node with one field changed. The operands keep the order they were written in, because most operators are not commutative: `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them apart. A constant is bound by the same `constant_value` fold a call's parameters already use, so `x * (1 + 2)` carries `3` rather than an addition the tree would have to know how to run. Three shapes are refused rather than approximated. An operation between two vectors needs each `Input` to name the selector it reads, which the one binding does not carry, so `a + b` and `a + a` would compile to the same tree; a set operator combines two vectors' series and has no node at all; and a statement whose value is a scalar has nothing to scan, so a constant is only bound next to a vector that gives it something to broadcast over. Each refusal names what it found. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
Dennis Zhuang
|
c10159cf10 |
chore: merge narthex into feat/narthex-count-values-histograms
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/map.rs |
||
|
dennis zhuang
|
0b6ba623f0 |
feat(promql): partial state and merge for native float aggregates (#9826)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task AGG-STATE. ## What's changed and what's your intention? This adds partial aggregation state for native float aggregates, so an execution partition can fold its series and a later stage can merge the states into the result. Frame encoding, Exchange and plan wiring are not part of it. - `FloatAggregator::finish_partial()` returns a `PartialAggregate`. Its groups are in first-seen order, and each group holds per grid point one `Option<AccumulatorState>` per field; `None` means the field had no contribution there. - `AggregateOp::state_fields()` declares which components each aggregation's state carries. A merge reads nothing else: - `sum`: value, compensation. - `count`: count. - `avg`: value, mean, compensation, count, and the incremental-mean flag. - `min` / `max`: value. - `group`: nothing. Its result is 1 wherever a state exists, so `finish` no longer reads the value. - `stddev` / `stdvar`: value (sum of squared deviations), mean, count. - `FloatMerger` merges partials in any order and identifies groups by their full label set. It rejects a partial with another aggregation, another field count, or points outside the grid, and leaves its state unchanged. - Merge rules, compared with the serial fold: - `sum`: the error of adding the two partial sums goes into the compensation, together with both compensations. - `avg`: sums are added while the result stays finite. Otherwise the merge continues with the count-weighted mean, as the serial update does once its sum overflows. - `stddev` / `stdvar`: Chan's pairwise update, computed as `δ·(δ·w)`, so `[0]` merged with `[1.5e154]` stays finite. - `min` / `max`: the serial comparison. For equal `0` and `-0`, the partial merged first wins. - `merge_count_values` adds the counts of `count_values` partials that share a full output label set. `quantile` has no partial and stays on the complete path. Regrouping changes the rounding, and it can also move an overflow of finite values: a serial `sum` of `[1e308, 1e308, -1e308]` is `+Inf`, while the merge of `[1e308]` and `[1e308, -1e308]` is finite. `stddev` / `stdvar` can also differ beyond that when samples differ by about the rounding of their mean: Prometheus' serial Welford rounds the mean per sample, and for `[1e16] | [1e16 + 2]` it gives a variance of 2 where the merge gives the exact 1. Explicit NaN and infinite inputs give the same class as the serial fold. The module docs state this contract. ### Tests - **Random partitions.** 300 seeds × 8 aggregations, plus `count_values`: 1–3 fields of sparse input, split into random partitions and merged in a shuffled order. Before merging, each partial is stripped to its `state_fields`. Results must match the serial fold: - groups, points and fields exactly; - values within 1e-10; - bit for bit when one partition holds every series. - **Directed cases:** - compensation carried by a partial and created by a merge, for `sum` and `avg`; - `avg` with unequal counts after one partition switched to a mean; - `stdvar` of `[0]` and `[1.5e154]`; - `min` / `max` sign of zero following merge order; - rejected partials. A mean-mode compensation is at most the last bit of a mean near 1e307, so dropping its weights is not observable within this contract and is not pinned by a test. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
99bf6a8403 |
feat(promql): add histogram_quantiles and share Go float formatting (#9840)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task NEW-MAP-FN, remaining part. ## What's changed and what's your intention? Adds the native kernel for the experimental `histogram_quantiles(v, label, q...)` from Prometheus v3.15.0. - `histogram_eval::histogram_quantiles` evaluates `histogram_quantile` for each `q` with the existing kernel and sets `label` to `q`, formatted as Prometheus' `labels.FormatOpenMetricsFloat` does. Results that then share a label set merge through `merge_same_labelset`. A repeated `q`, or a `label` that overwrites the label telling two histograms apart, fails at a point both hold, as in Prometheus. - The Go float formatting from #9799's `count_values` moves to a `go_format` module, next to `go_sort`. It now offers both `FormatFloat(v, 'f', -1, 64)` and `FormatOpenMetricsFloat`, which is Go's shortest `'g'` format: an exponent below 1e-4 and from 1e6 on, at least two exponent digits, `.0` appended to integers, and hardcoded `0.0`, `1.0`, `-1.0`. Both share the round-trip-checked shortest digits. ### Tests - `histogram_eval/quantiles_cases.txt` (37 KB, 30 cases): classic histograms evaluated by the Prometheus engine with experimental functions on (promql-compat `narthex-golden-histogram -quantiles`). It includes repeated quantiles and `job` as the quantile label; 13 cases end in the label set collision. - `go_format/open_metrics_cases.txt` (13 KB, 358 values): `FormatOpenMetricsFloat` outputs for the exponent switch points, the hardcoded values, powers of two from 2^-30 to 2^30 with neighbours, and random values (`narthex-golden-formatfloat -openmetrics`). The existing `'f'` fixture is unchanged. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
6afb0283cd |
feat(promql): project selected sample times for timestamp() and info() (#9836)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Narthex operator task EVAL-TS. It also provides the input #9829's `info()` kernel expects. ## What's changed and what's your intention? `timestamp()` needs the original time of the sample an instant selection chose. `select_instants` already returns the source row of each selected point, so this adds kernels that project from that selection onto the evaluation grid, next to `select_instants` in `instant_eval.rs`: - `timestamp_of_selection`: `timestamp(selector)`. The value is the original time of the selected sample in seconds. An offset or `@` changes which sample is selected, not the time reported. Metric name, `__type__` and `__unit__` are dropped. - `info_sample_times`: the series `info()` reads. The value is the original time in milliseconds, as Prometheus' `evalSeries(..., recordOrigT=true)` records it. Labels are kept. A native histogram fails with Prometheus' "this should be an info metric, with float samples". - `timestamp_of_evaluated`: `timestamp()` over an evaluated expression such as `timestamp(abs(x))`. The value is the evaluation time in seconds where the input has a value. Metadata labels are dropped. A field without a sample in the selected row has no value at that point. Lookback and stale handling stay those of `select_instants`. A selection must lie on the grid, with points and rows increasing; that lets one cursor per field read each selected row once, even when consecutive points select the same row. Out of scope: wiring into the Instant runtime and the native compiler (including `@` binding). ### Tests `instant_eval/timestamp_cases.txt` (22 KB) comes from the Prometheus engine via promql-compat `narthex-golden-timestamp`. Samples sit at arbitrary milliseconds, so original and evaluation times differ. It has 40 random cases plus fixed ones for: - a sample exactly one lookback before a point (excluded); - a stale marker; - an offset, over a selector and over an evaluated expression. Unit tests cover: - the millisecond output, labels and missing fields of `info_sample_times`; - histogram samples (rejected for `info`, accepted for `timestamp`); - rejection of selections off the grid or going back. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
0279e12889 |
chore: merge feat/narthex-select-compile into feat/narthex-histogram-quantile-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
abb55be608 |
refactor(promql): follow the Vector rename in the compiler
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bf3ae6ab60 | chore: merge feat/narthex-select-absent-sort into feat/narthex-select-compile | ||
|
Dennis Zhuang
|
b7dfb8ec3d |
fix(promql): keep a shared histogram table charged until the last queued result leaves
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
5dbda4d9cd |
chore: merge feat/narthex-select-node into feat/narthex-select-absent-sort
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs |
||
|
Dennis Zhuang
|
90478ec493 |
refactor(promql): name the whole-vector node Vector
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bec5f76637 |
fix(promql): keep the input order for limitk and merge label-set pieces instead of sorting
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bc782ac5da |
feat(promql): count native histograms in count_values by their Prometheus text
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
7e0325780e |
chore(promql): return the group slot directly
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
8bc2c8aa55 |
feat(promql): run histogram_quantile and histogram_fraction in the native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
3d19a1f3fc |
feat(promql): compile sort and absent into the native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
353149cf63 |
feat(promql): run sort and absent in the native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
63a7d814a0 |
feat(promql): run topk, bottomk, limitk and limit_ratio as a native select node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
0efeae0f1d |
feat(promql): compile and run label_replace and label_join as map operations
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
7217bf43fb |
refactor(promql): run date functions only through map_math and refine the review points
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
e06f5b3f58 |
feat(promql): compile and run date functions over a vector in the native tree
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
d79a5f6bfe |
test(promql): keep Go float format tests with go_format and pin histogram_quantiles constants
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
607719dc9d |
fix(promql): validate instant selections in one pass and format info errors like Prometheus
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
e61a6f42eb |
chore: merge narthex into feat/narthex-eval-ts
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
7c139a2ccc |
feat(promql): skip native histogram samples in math functions and cover review gaps
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
e4d1b79b76 |
feat(promql): evaluate round and clamp with time-varying scalar parameters
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |