mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
c48c5ec5e13cc055817d2f7bccc8a9b0adb2f3dc
5598
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
d62051e12a |
fix(query): preserve logical metric field order in native scans
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
cd64176418 |
fix(sql): preserve lookback when displaying TQL view definitions
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
27cd09ec55 |
fix(query): keep TQL views serializable and expose native diagnostics
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
340ea2d71e |
fix(query): retain metric table restriction in native scans
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
29ef60f6e8 |
test(promql): restore scan assignment coverage and update native guide
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
51da078dc8 |
chore(query): clear the two lints clippy reports on this branch
A doc comment on the categorizer was followed by a blank line, and the test table builder takes eight arguments. Both are what CI's `-D warnings` turns into errors; neither changes behaviour. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
ca804689eb |
chore(query): share the offset conversion and correct a column-order note
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
8e55a48952 |
refactor(promql): keep one cross-series predicate for both callers
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
ff1288f178 |
fix(query): match the raw scan's own field and engine rules
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
de97d86800 |
fix(query): only offer the native route to an engine that serves frames
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
638d485024 |
fix(query): decline the native route where it cannot reach a region
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
1ced0f7fdb |
chore(query): share the range assignment with every caller
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
8efec61743 |
fix(promql): keep the fused scan's parallelism
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
5527c751d2 |
fix(query): decline the native route where no raw scan exists
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
129faa4e49 |
chore(query): drop a needless borrow on the scan schema
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
0d29dc98e8 |
fix(query): emit the fused output in the existing column order
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
67a3d22c5b |
fix(query): keep a fused route to series a region owns
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
64f039eaf4 |
test(query): run the encoding checks for both transports
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
6121600789 |
fix(query): keep ordinary plans on the encoding path
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
da27d115ca |
feat(query): let a region input travel as a plan when regions are in process
A plan holding a node with no Substrait serializer was always cut at that node, because the analyzer asks whether a plan encodes before it asks whether a node commutes with the region split. That is right for a region reached over the wire: the plan is encoded into the request and has to decode on the other side. Standalone does not use that path. Its region server is the one built in the same process, its query handler passes the request straight to `RegionServer::handle_read`, and the plan stays a `LogicalPlan` from the frontend to the region. Encoding it is work nobody reads, and refusing to place a node because it cannot be encoded refuses a plan that would have travelled intact. So the engine now knows whether its regions are reached in process, and the encodability gate applies only when they are not. The capability is set where it can be stated truthfully - the standalone bootstrap, which creates both the frontend and the region server it reads - and it defaults to false everywhere else, so an ordinary build and every other deployment keeps today's behaviour byte for byte. Skipping the gate skips one early return and adds no reason to expand: `Categorizer::check_plan` still runs and still decides, and a plan that does not encode for any other reason still expands at the node that caused it. Two tests hold those two halves apart. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
ba9f36fdd0 |
docs(query): say what the fused placement does not prove
The arm above answers that a fused node commutes with the region split, and it listed "no series is split across regions" among its reasons. That is not established anywhere. A region evaluates the whole tree over the series its own scan returns, so a series whose samples reach two regions is evaluated per fragment, and a range function over a fragment is not the range function over the series. What the placement does have is three proofs about the node - its input is the scan, it binds a metric name, and no node of its tree folds across series. What it does not have is any check that the series stays in one region: the storage partition key is whatever `partition_column_names()` names and may be a column that varies within one series, and this categorizer runs before region selection, so there is no selected-region vector here to consult either. So the comment now says that plainly, names the deployment the placement is correct under, and says what would make it unconditional. A single-region deployment satisfies it; a partitioned table over several regions would answer wrongly rather than fail. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
ff6c5db307 |
feat(query): place a fused node region-side, and ask for per-series scans
A `FusedSeries` node is built for one metric's series, and a region holds a subset of those series with none split across regions, so the node commutes with the region split when what it reads is per-series. It says so now, and the analyzer may place it inside the merge scan. Three conditions, each drawn from the conversion that has to succeed there: - The node's input is exactly the table scan. The region builds the node from the scan's own raw-series entrypoint and cannot look through a filter, a projection or a sort in between, so a node over anything else has nothing to read where it would be placed. - The node binds a metric name, because the region resolves the series of one named metric. - No node of its tree folds across series. A region holds part of a metric's series, so an aggregate or a binary operation evaluated there would consume a partial input and return a wrong answer without saying so. Every operation the compiler emits today - input, instant, window, map - reads one series at a time, so this refuses nothing buildable; it is the conservative answer for a tree built some other way. A scan below such a node also asks for the per-series distribution now. That is what makes a region prepare its series scan, and only that path packs a series' chunks into the frame the node reads. Both are inert without a `FusedSeries` node, so an ordinary query keeps its plan and its scan request. Not yet reached: the analyzer asks whether a plan encodes before it asks whether a node commutes, and this node has no Substrait serializer, so the plan is expanded at the node and the region still receives the bare scan. A test pins that boundary so it fails the day the node can cross it. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
4beb5c7490 |
feat(query): offer a PromQL statement to the native fused-series route
Nothing called `native_compile::compile` or built a `FusedSeries`, so the native tree and its node were library code with tests and no query behind them. This adds the route, behind a per-query flag. The statement is offered to the compiler at the root, where the expression the reader wrote still exists. A shape the compiler refuses - `NotImplemented` - is this route declining the query, and planning continues exactly as it would with the flag off. Anything else is a real error. The physical half is an arm in `PromExtensionPlanner`. Its children are already built by then, so a metric scan arrives as the region scan, and the node asks that scan for its named raw-series binding: `try_to_named_series_scan` resolves the label source and ordering from the scanner itself, which is the only thing that can name them, and returns the leaf the node reads. A scanner that cannot serve a raw frame is reported, not silently re-planned. Two things this needed that the route does not get for free: - The region scan has to be prepared as one partition. `resolve_named_series_binding` refuses a scan whose partition count is not one, and a fresh scan has not set its partitions at all: `ParallelizeScan` runs after extension planning and would spread it, so the node would run one evaluator per partition and split a logical series. Preparing the ranges as one partition is what keeps `FusedSeriesExec` over one series stream, and it is why this route does not scan in parallel yet. - The scan needs the per-series distribution hint. That is what makes a region prepare its series scan, and only that path packs a series' chunks into the frame this node reads. The existing planner asks for it through a sort that `ScanHintRule` turns into the hint; this node has no such sort below it, so the hint is attached to the scan directly. A query the route cannot bind faithfully keeps the existing plan rather than a wrong one: more than one selector, a label matcher that would survive preprocessing, an `__field__` matcher, a native histogram field, a missing table, or a shape whose grid the compiler's statement cannot express. Signed-off-by: discord9 <discord9@163.com> |
||
 dennis zhuang
|
8497382673 |
feat(promql): compile and run label_replace and label_join as map operations (#9848)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, label category (#25). Builds on #9846 (date wiring, merged). ## What's changed and what's your intention? `label_replace` and `label_join` now compile into the native tree and run in the existing `Map` node. - **Plan.** `BoundMap.ops` becomes `Arc<[MapOp]>`, with `MapOp::Math(MathOp)` and `MapOp::Label(Arc<LabelRewrite>)`. A label change is pointwise per series like math, so it needs no new node, collector or output type. - **Kernel.** `LabelRewrite` (in `map_label.rs`) is the checked, compiled change: `replace` anchors the regex and checks the destination, `join` checks source and destination names, `apply` rewrites one `MetricName`. The whole-vector `label_replace` / `label_join` now go through it and still merge same-labelset series afterwards, so the Prometheus golden keeps covering the same code. Fields are private, so only the two constructors build one. - **Runtime.** `map_chunk` applies label ops to the series metric and math ops through `map_math`. When a math op empties a closing chunk, the labels of the closing series come from applying the remaining ops in order (label ops rewrite, math ops drop the name). Series a label change folds into one label set go through the `Emitted` ledger the map already keeps: disjoint values merge, a shared point is refused. - **Compiler.** `label_rewrite` reads the string-literal arguments. Arguments Prometheus rejects at evaluation (invalid regex, empty label name) fail the statement with `Plan`, as Prometheus fails the query; everything unsupported stays `NotImplemented`. - **Display.** `label_replace "dst" "repl" "src" "regex"` / `label_join "dst" "sep" ["a", "b"]`. Shared files touched: `native/plan.rs` (`MapOp`), `native/execute.rs` (`map_chunk`, `no_row_series` now takes the metric), `native/display.rs`, `native_compile.rs`. Test constructors move from `MathOp` to `MapOp::Math`. Known difference, not introduced here but now reachable: Prometheus checks same-labelset duplicates separately for floats and histograms, while the native ledger counts both as a value at a position. A `label_replace` that folds a float series and a histogram series onto one label set at the same step is refused natively. ### Tests - Runtime: a relabel that folds two series refuses the shared point; a closing chunk emptied mid-chain carries labels from label ops on both sides of the emptying op; a label-only map hands histogram samples and holes through unchanged. - Compiler: both functions bind their change into a map; an invalid regex fails the statement with `Plan`. - `map_label` Prometheus golden (existing) now runs through `LabelRewrite`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
3674b33db8 |
fix(promql): keep fill_left and fill_right on their written sides under group_right (#9899)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) prometheus/prometheus#19939 ## What's changed and what's your intention? The native binary kernel applied `fill_left` / `fill_right` after swapping the operands of `group_right`, so under `group_right` `fill_left` filled a missing right-hand sample and `fill_right` a missing left-hand one. That is the Prometheus behavior before prometheus/prometheus#19939, which was merged into Prometheus main on 2026-10-13: the fill values now swap together with the operands, so they always refer to the side written in the query. For example, `node_meta - on(instance) group_right fill_left(5) fill_right(7) cpu_info`: a missing `node_meta` must give `5 - cpu_info` and a missing `cpu_info` must give `node_meta - 7`. - `binary_field` swaps `fill.lhs` / `fill.rhs` when the one side is the left, matching `VectorBinop` after #19939. The `FillValues` doc now states the as-written meaning. - `fill_cases.txt` is regenerated by promql-compat `narthex-golden-vector -experimental fill` built against Prometheus at the #19939 merge commit (same seed and case count). Only the 24 `group_right` cases whose fill fires on an asymmetric pair change. No released Prometheus contains #19939 yet (v3.15.1 still has the old behavior), so until the next release native results differ from released Prometheus for asymmetric fills under `group_right`. The compiler does not bind vector-vector fill modifiers yet, so this is not reachable from a query today; it has to land before that wiring does. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
1c63b06265 |
feat(promql): compile topk, bottomk, limitk, limit_ratio and count_values into the vector node (#9890)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, Select / aggregate categories (#25). Follows #9850 → #9851 → #9852 → #9853 → #9854 (all merged) and #9833. ## What's changed and what's your intention? `topk`, `bottomk`, `limitk`, `limit_ratio` and `count_values` now compile. #9833 left them out of `AggregateKind` because they do not fold a group into one value per point. They now bind as a `Vector` node through `vector_aggregate`: - `topk` / `bottomk` / `limitk`: `k` is folded to a constant as `quantile`'s parameter is, and the `by` / `without` grouping is kept. A `k` only known at run time is refused with `NotImplemented`. NaN, negative and out-of-range `k` are left to the kernel, which fails or answers at evaluation as Prometheus does. - `limit_ratio`: the ratio is folded the same way. The grouping is dropped, since Prometheus keeps or drops each series by its label hash alone. - `count_values`: the label is a string literal. Label validation stays in the kernel, an execution error as in Prometheus. The module doc, the grammar and the `SUPPORTED` text name the new shapes. ### Tests - Compiler: each aggregator binds its operation with the folded parameter and grouping. A run-time `k` is refused, and the rejection quotes every aggregator this compiler takes. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
13f4fb9cdb |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs |
||
 Yingwen
|
ce4fc4529e |
feat(mito): read columnar series through prepared sources (#9892)
* feat(mito): read columnar series through prepared sources Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): rebuild schema-incompatible columnar sidecars Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito): regenerate series format configuration Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): batch prepared series reads and preflight inputs Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Dennis Zhuang
|
4ca04d2790 | chore: merge narthex into fix/narthex-group-right-fill | ||
|
Dennis Zhuang
|
5e621d481b | chore: merge narthex into feat/narthex-vector-agg-compile | ||
|
dennis zhuang
|
069b942518 |
feat(promql): run count_values in the native vector node (#9854)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, aggregate category (`count_values`, #25). Stacked on #9853 → #9852 → #9851 → #9850; its own change is the last commit. Draft until those merge. ## What's changed and what's your intention? `count_values(label, v)` runs natively as `BoundSelect::CountValues { label, group }` through `aggregate::count_values`. - **Why this node.** `count_values` writes one label set per distinct value, which #9833 leaves out of `AggregateKind` as a different shape. It reads the whole vector at a point, which is what the select node does. - **Output.** Groups are keyed by their final label set, so each result label set is one series and the node does not mark its output as holding pieces. Input pieces of one label set hold disjoint points, so their counts add. - **Memory.** The estimate charges a group per row, since a gauge with distinct values makes about one group per sample: the group's label set twice with the value label, its counts and its result series. Not here: compiling `count_values`, which is an aggregate expression and goes through #9833. Before it is compiled, it needs a decision on native histogram input. Prometheus counts histograms by their string form; the kernel refuses them with `NotImplemented`. Until the kernel covers them, the compiler has to refuse such input. ### Tests - Runtime: values shared across series are counted per point, including a series delivered in two chunks. - Display: the `count_values` line. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
85c5dc0d69 |
test(promql): pin the refusal to a kind the runtime still refuses (#9900)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dennis Zhuang
|
b71676e72d |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile/tests.rs |
||
|
discord9
|
3b09dd75dc |
feat(promql): run a bound tree over raw series in one fused node (#9820)
* feat(promql): run a bound tree over raw series in one fused node
`FusedSeriesExec` evaluates a compiled `NativeTree` directly over the raw
series its child produces, instead of materialising a row batch through the
per-operator chain. It binds the time column, the fields and the tags once,
checks the input against that binding when the plan is built, and converts the
chunks the tree settles back into the statement's output columns.
The node's partition count is the child's, one tree per partition. The child
must be a raw-series scan that assigns every series whole to one partition:
`PerSeries` distribution is what makes a per-partition evaluation equivalent to
evaluating the whole series. A zero-partition input is refused; anything else
is the child's business.
A label set can recur across partitions once `rate` and the map kernels drop
`__name__`, so the occupancy map is plan state rather than per-stream. Two
series reaching one label set is an error only when their occupied grid points
overlap, which is Prometheus' range-query rule; disjoint points are legal and
are merged into one occupancy record. The reservation for that map is attached
to the first execution's pool and released when the plan is dropped, because a
later partition still has to collide with what an earlier one emitted.
`occupied_runs` reads a chunk's absolute positions rather than assuming one
contiguous run: a sparse evaluated chunk may hold several, and treating the
first as the whole range would count the holes as rows.
A poll does a bounded amount of work and wakes itself when the budget runs out,
so a long run of chunks that settle nothing cannot monopolise the executor
thread.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): charge for the label sets the plan keeps, not for one stream
The occupancy map is the plan's: entries are inserted by whichever partition
sees a label set first and no path removes them. The reservation paying for
them was the stream's, so the bytes went back when that stream ended while the
entries stayed. Measured on a plan with three partitions producing three label
sets: each grew 42 bytes, each stream's end returned them, and the pool then
read 0 while every entry was still there and still colliding with later
partitions. The plan's own drop produced no shrink at all.
So a label set's bytes were charged for part of the time and invisible for the
rest, and the invisibility grew with the number of label sets a query touches.
The charge now belongs to the map it pays for: `LabelSets` holds the entries
and the reservation together, `attach` registers it once on the first
execution's pool, and it is released when the plan is dropped.
`attach` is idempotent because every partition calls it. Re-registering would
move a charge the other partitions keep growing, and would give it back when
one stream ends while the entries stay - which is the shape of the bug.
A regression test states the invariant: after the first partition's stream
ends, the pool is still charged for its label set; a second partition with the
same set on disjoint points adds no further charge; dropping the plan returns
it to zero.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): take a raw series frame, not a row batch, and keep its series whole
Review found five things; this is all of them.
The node read ordinary Arrow rows and converted them itself, so a real
`SeriesScanExec` could never feed it: a frame's columns are `read_key`,
`is_last`, and the `time` and `fields` structs, and none of them is the time
column, a tag or a field the binding names, so construction refused it. It now
takes a raw series frame the way `NativeInstantManipulateExec` does - the frame
is validated when the plan is built and unpacked per partition when it runs -
and the row-to-series mapping is gone, because turning rows into series is a
different node's job. The series schema comes from the tree's own input node,
so there is one statement of it rather than two that could disagree.
A raw frame is a per-series stream, and this node holds one window per series
and closes it on the chunk that marks the end. Declaring that as
`KeyPartitioned(read_key)` keeps the optimizer from splitting a series across
partitions, which it would otherwise be free to do; `read_key` is the series'
primary key as a binary column, so hashing it groups exactly the rows of one
series. A round-robin split would have let each partition rate half a series
and answer half the truth; with the key declared, a series that somehow did
arrive in two pieces is refused by the frame validator instead.
Two label-set rules were wrong. Occupancy was tracked per position for any
field, so two series that each hold a different field at the same point looked
like a collision, while the rest of the stack merges exactly that as
complementary fields - it is now tracked per field, and only one field holding
two values at one point is refused. And the key encoded every tag including
empty ones, while `grid_series::label_set` drops them, so `{host="h", zone=""}`
and `{host="h"}` were two keys here and one everywhere else; the key is now
that same normalized label set.
Finally, a label set already in the map grew its run arrays on every merge
without telling the pool. The charge now follows the capacity the merge adds,
so a long-lived plan that keeps meeting one label set pays for what it holds.
The test that asserted the opposite was wrong and is corrected.
Signed-off-by: discord9 <discord9@163.com>
* fix(table): declare the distribution a raw series frame already has
`SeriesScan` guarantees every series arrives whole in one partition - its own
docs say so, and both modes enforce it in different ways: the two-phase
collector places a series by its tsid, and the legacy divider only emits a
batch once it holds all of a series' rows. But the leaf built for a raw frame
declared `UnknownPartitioning`, so nothing on the plan said it.
That mattered to a consumer holding one window per series: with the
distribution unspecified an optimizer is free to round-robin the partition,
and two halves of one series would each look final to their own stream. So the
leaf now declares `Hash(read_key)`. `read_key` is the primary key as a binary
column, so hashing it groups exactly the rows of one series and splits nothing;
`FusedSeriesExec` asks for the same key and is satisfied without a shuffle.
The three tests that pinned `UnknownPartitioning` now pin the key instead.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): keep the capacity a label set already paid for
`merge_runs` built a fresh vector sized to `stored + new` and replaced the old
one. When adjacent runs joined - or when a claim added nothing to a field,
which is what two complementary label sets do to each other - the entry came
back smaller than it was. The charge in `claim` is the growth over the
capacity, so the next merge into that field looked like growth again and was
charged for it, every time, while the entry itself never held more than a
couple of runs. A long query meeting one label set repeatedly would have paid
for the same few bytes over and over and could exhaust a pool it never used.
The merge now keeps whichever capacity is larger, and the charge follows that:
a merge that adds nothing to a field costs nothing.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): run the fused node over one series stream
`FusedSeriesExec` held one window per series and closed it on the chunk carrying
`is_last`, so it can answer a series only if nothing splits that run and nothing
puts part of it in another tree. Those are two different promises, and only the
first is a property of a stream: the frame decoder refuses a chunk interleaved
with another series', but nothing it can see says that half a series is in a
partition it never reads.
The node used to ask for `KeyPartitioned(read_key)` instead, expecting the
optimizer to keep whole series per partition. That does not hold: DataFusion
considers two hash layouts with the same count co-partitioned, and above
`subset_repartition_threshold` children it adds a hash exchange even when the
requirement is met (`enforce_distribution.rs`, the
`!partitioning_satisfied || target > child` branch). A hash exchange does not
preserve order, so series chunks interleave and the second half of a series is
answered as a series of its own.
So the count is now fixed at one, in `try_new` and in the reported properties,
and `benefits_from_input_partitioning` is false so no optimizer splits it for
parallelism. This is the phase-1 contract: reading a series spread over
partitions needs an exchange above this node, which does not exist yet.
Merging #9786 also moved `unpack_raw_series` to two arguments, with the frame
declaring its own label layout and ordering; the calls follow it, and the plan's
binding is checked against that declaration with
`ensure_frame_matches_label_source`.
Signed-off-by: discord9 <discord9@163.com>
* Revert "fix(table): declare the distribution a raw series frame already has"
The declaration was false. `Partitioning::Hash(read_key, n)` claims DataFusion
placed rows by its own hash of `read_key` mod `n`, and DataFusion treats any two
equal-count `Hash` layouts as co-partitioned, so a future co-partitioned consumer
would pair partitions that hold unrelated series. `SeriesScan` places a series by
tsid range (two-phase) or round-robin (legacy), neither of which is that hash.
It also did not buy what it was added for. `FusedSeriesExec` now takes one input
partition and reports `benefits_from_input_partitioning = false`, so nothing
repartitions between the scan and it, and the ordering guarantee the declaration
was meant to carry is the frame's own.
This reverts commit
|
||
|
discord9
|
a8737d0996 |
test(promql): skip an endpoint the platform cannot represent (#9894)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1205c18496 |
feat(promql): execute Complete aggregates in the native runtime (#9873)
* feat(promql): execute a bound native tree node by node, over sparse evaluated series Ports the native runtime onto the sparse evaluated-series API: a chunk is addressed by a `TimeAxis` that may hold several runs, so a consumer reads positions through the axis instead of assuming `start + row`. The driver and both node kinds move to that API: - `chunk.range().len()` becomes `chunk.series.time.len()`, the compact row count, in every metric and byte estimate. - `occupied_runs` walks `time.positions()` with one field cursor, so a chunk of several runs records each run where it belongs. A run open across a hole now ends where the hole begins; leaving it open to the last row made two runs look like one and refused a series that occupies the hole. - A window keeps the pieces its evaluator decided in a queue: one push can decide several runs, and reporting only what the push returned stranded the rest until the next push. Queued pieces are charged by `retained_bytes`, and the charge is recomputed when one leaves, so a released piece is not held. - `map_chunk` answers `Option`: an operation that empties every position (`clamp` with `max < min`) yields no chunk at all, unless the chunk closes its series -- a closing chunk holding no row is handed over so a consumer can tell "nothing more here" from "no more input". `RangeEvaluator::retained_bytes` is restored with the three byte helpers it reads: a field's buffers, a series' identity and the pending batch's vectors, all by capacity, and still counted as storage rather than as a peak over a push. Tests read absolute positions from the axis (`decided` no longer adds a row number to a start), and two sparse cases cover what the old reading got wrong: a `Window -> Map -> Window` chain over a chunk of two runs, and a series that occupies the hole between another's runs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop the window queue the sparse evaluator never needed `RangeEvaluator::push` and `push_series` each return `Result<Option<EvaluatedChunk>>`: the evaluator gathers every run it decided into one chunk and hands back at most one. So `pending.extend(decided)` inserted at most one chunk and `take_pending` removed it on the spot, leaving the queue empty after every successful push. That makes the queue dead weight: a `VecDeque` allocation per productive window, an accounting term that is always zero, a second drain site in `pull`, and doc comments claiming a problem the API does not have. The driver's "ask the input again when a window decides nothing" loop was already the whole story. Back to returning the evaluator's own answer, and drop `pending`, `take_pending` and `take_window_output`. Covers the empty-map protocol this port introduces while it is here: an operation that empties every position hands on nothing while its series is open, and one empty chunk when it closes. A non-closing empty chunk would be rejected downstream and a dropped closing one would leave the series unclosed. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the empty-map output count at one The test checked that every output was empty and that exactly one closed, which a stray extra non-closing empty output would also satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge the memory the native evaluator actually retains Four omissions made the reservation blind to most of what a window over native histograms keeps, and the existing incremental-equals-recomputed test could not see any of them because both sides used the same formula. - `FloatHistogram` owns four heap allocations -- both bucket lists, the custom bounds, and the span layout -- that `FieldSamples::bytes` never looked at. A retained 4096-bucket sample is tens of kilobytes against a ledger that saw a struct header. `heap_bytes` counts them, and both the retained sample slice and the histograms a row boxes are charged for what they own. - Queued chunks were estimated as one float per field per row, which is not what a histogram field holds: it holds indices into a shared `Arc<HistogramBatch>` and the buffers behind the Arrow arrays. The figure now walks the samples, and counts a shared table or a shared buffer slice once per pass rather than once per chunk that references it. - `history_bytes` included `size_of::<History>()` while the map slot term charged a `History` per entry as well, so every entry paid two headers. The per-entry figure now covers only the heap the entry owns. - The primary key is charged for the allocation its `Buffer` keeps alive rather than the bytes it shows: a scan slices the key out of a decoded dictionary-values buffer, so a short key can pin a batch-sized allocation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep the source queue's charge incremental The queue's reservation was recounted by walking every queued chunk on each enqueue and each pop. `push_raw` and `push_evaluated` are public and put no bound on how deep the queue gets, so a caller that queues a whole series before pulling it paid `N(N+1)/2` chunk visits to hold `N` chunks -- the same shape of defect this branch fixes in `absorb`. `QueuedCharge` keeps a reference count and the bytes behind each shared allocation: a table or buffer is charged when the first queued chunk reaches it and given back when the last one leaves, so an operation walks the one chunk it names. The count is per chunk rather than per field, because a pop re-walks the same chunk with the same dedupe and must remove exactly the references that were added. A chunk's own struct bytes are counted per chunk (`own_bytes`); only the allocations behind its fields are counted by address, since a struct has no address to dedupe on. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): append a forward-moving chunk to the emitted history `absorb` rebuilt the whole stored run array on every non-empty chunk. Legal input made that quadratic across chunks with no label collision: one series in `K` chunks, chunk `i` occupying only grid position `2*i`, stores one more run per chunk and moves `K(K+1)/2` of them -- about five billion for a hundred thousand rows. A chunk that starts at or past the end of the last stored run only ever adds to the end, so its runs are appended and the boundary run is joined where it touches. One that can interleave or reach backwards still takes the full one-pass merge, which can split a run. Both leave the runs sorted, disjoint and joined, which is what `Emitted::occupy` looks a position up by. The test records the chunks through `Emitted::occupy` itself and reads the history back from it, covering the forward case, the touching boundary, a hole that has to join both neighbours, and a position before everything stored. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): report a truncated input from a tree with no window `close` only asked windows whether a series was still open, so a tree of an input alone, or of a map over one, accepted a final chunk without `is_last` and handed the consumer a series that never closed. The input node now follows the series of the chunk it last handed over, and closing such a tree reports the truncation against it, which is the same refusal a window makes for the input it reads. `occupied_runs` read a row the axis names with `expect`, which panics on a `Series` whose fields are shorter than its axis; the fields are public, so that chunk is constructible by a caller. It now returns the misalignment as an error naming the node, the field and both lengths. The chunk protocol says one series closes once, which reads as one close per label set. State that it is per input series: a node that rewrites labels can hand two series over under one label set and both of them close. Also say in `Emitted`'s doc that the table is one node's own, so a label set costs its bytes again in every node that keeps one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute an instant selection as a tree node The tree could not run `NativeExpr::Instant`: the state builder recorded `not_implemented` and refused the whole tree before reading a chunk, while the compiler already produced the node for a bare selector. A query the plan and the compiler both accept therefore had no runtime path at all. `InstantRuntime` buffers one series to its `is_last` chunk -- the selection kernel reads a complete axis -- then calls the shared `select_instants` once and gathers the rows it picked onto the binding's own grid. Positions come from the grid lattice the kernel was asked for, and a timestamp off that lattice is an error rather than a skipped row. A series that selects nothing still closes, with an empty chunk on the grid; an input that ends inside a series is reported on `finish()`. The node does not record label-set occupancy: it keeps the metric verbatim, so it cannot map two series onto one label set, and the occupancy key drops empty-valued tags -- two genuinely distinct series would be refused as a collision. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved input, and close a label set only for its owner Two series-level identity holes, both reported against `42ea525`. The truncation flag followed the last chunk handed over, which is the same as "is a series open" only while at most one series can be open. A tree with no window still accepted `Interleaved`, and `a` open, `b` closed, end of input then passed while `a` never closed. Tracking the open set would need an identity that a raw chunk in bound-label mode does not have (`metric` is `None`, the key holds it), so the ordering is refused when the state is built instead, for every tree shape and not only under a window. The flag's doc now says what it means and why that is the whole answer. `History` closed on any series' `is_last`, so with `abs` dropping the metric name a series that contributed nothing could close a label set another series was still filling: `a{job="x"}` value, `b{job="x"}` empty close, `a{job="x"}` continuation was refused with "already closed the label set". Contiguous input hands over exactly that sequence, so it was not an interleaving artefact. A history now remembers the series that filled it and closes only for that series' closing chunk; a series that produced nothing under an open set is passed over, as it already was under a closed one. The owner is charged as part of the entry. The ordering check also reads every input node rather than only the first: a tree with two inputs would otherwise have its second declaration read and dropped. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve the instant gather's workspace and keep the histogram start Two findings from the review of the instant node. The gather allocates after the last chunk arrives: the concatenated axis, the descriptors that read it, the two arrays the selection returns, and the rows the builder keeps. Nothing asked the pool for any of it, so with one sample selected by every point of a million-point grid the node took tens of megabytes while its reservation fell to the buffered chunks' size -- nothing at all, on the chunk that ends the series. `gather_bytes` now derives a bound from the values in hand (rows, chunks, segments, fields, points, and the widest histogram row per field, since one row can be selected by every point), and the reservation grows by it before the gather runs. It shrinks back on both the success and the error path, so the workspace never outlives the call. A selection reads an existing sample, so it has to carry that sample's payload. `FloatHistogram` had no `start_timestamp` and `push_to` wrote `None`, which lost it for any histogram that had one: `detect_start_timestamp_reset` reads that field to spot a counter reset, so a window over this output would mis-detect resets. The field is now carried through a view and written back; the kernels that compute a histogram instead of reading one leave it `None`, which is what they produced before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): describe the gather test by what this branch does The doc on `an_oversized_selection_is_refused_by_the_pool` said the kernel has no point limit any more. It does on this branch: the limit is removed on its own branch, which this one does not carry, so the comment described a state that has not arrived. What is true here is the order: the estimate now applies to every gather, so the pool is asked before the kernel is, and the pool's refusal is the one the caller sees. That is worth saying on its own -- a bound in bytes on the path that decides the points, rather than a count that says nothing about what they cost -- and it stays true after the limit goes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute vector binary nodes in the native runtime Checkpoint the vector-vector runtime and single shared matching kernel. Keep addressed input delivery, whole-input buffering, nested composition and error cleanup together. The existing arithmetic/comparison implementation remains the only computation path. Local review checkpoint only, not ready for publication: all-targets check passes; the current run has three new fixture failures. Independent review requires current-live workspace resizing, field-local matched storage and kept-value merge budgeting before approval. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): infer the binary result fixture element type Drop a redundant complex type annotation flagged by the actual strict clippy run. The collection and literal assertions are unchanged. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve binary workspace along the shared kernel Remove the separate matching prepass and reserve current live workspace at the existing kernel allocation boundaries. Release per-sample and per-field scratch after ownership ends; charge actual retained rows, values, histograms and errors. Unbudgeted public and set callers bypass accounting without another computation path. Validate with cargo fmt -p promql -- --check; cargo check -p promql --all-targets; cargo nextest run -p promql --retries 0 --no-fail-fast; cargo clippy -p promql --all-targets -- -D warnings. The public nested Binary test also fails on swapped left/right operands and passes after byte-identical restoration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): scope binary reservations to live scans and output rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge binary buffers at actual allocation boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): skip accounting ledger for unbudgeted vector merges Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): simplify native execution memory estimates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): match binary operands by logical series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): cover flattened operand descriptor capacity Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge rewritten logical series before window evaluation Accept disjoint closed-source fragments under one output label set. Retain occupancy checks and merge affected window inputs once at EOF using shared typed sample segments. Keep direct input windows and root maps streaming, and release collected fragments on finish or error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): clarify approximate merge workspace estimate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate series boundaries before collected input merges Keep each collecting window's open metric and last position so EOF grouping cannot hide reversed runs or identity changes before closing. Preserve source and root-map piece semantics, and test malformed input release, empty closing chunks, and EOF truncation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): supply the default binary fill in the unimplemented-node fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): end a subtree before the node above it computes A window that collects its input answers only once its own input is finished, so ending the input now flushes the deepest ready window and lets the following pull carry its answer up; a binary node computes only when both operands are settled. Previously a binary node computed on an empty operand and marked itself computed, dropping what the window produced afterwards. Also stream the answer instead of collecting it: end_of_input() ends the input, and each pull runs the next stage, so the caller pulls the result and the last None closes the execution. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve an instant selection for the points it decides The gather reserved the whole planned grid, while select_instants clips the range to the series' first and last sample and its lookback: a week-long query at a one-minute step reserved ~504 KB for a series that produces five points, and a small pool refused it before gathering. Reserve for the clipped candidate count, derived with the same alignment the kernel uses. Also pin the start-timestamp semantics the histogram change spread to: last_over_time and first_over_time hand back the sample they selected with its start timestamp, and limitk and limit_ratio keep it on the sample they keep, so a reader downstream can still place a counter reset. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): reserve binary memory per phase, not per value The binary kernel adjusted the pool and the metrics once per matched point, per filed row and per merged row, so a million plain float points meant roughly three million reserve calls inside a numeric loop. Reserve at three boundaries instead: the operands as chunks are retained, one figure per phase (pairing, sort, merge) with an estimate over rows/fields/segments/payload, and each result buffer when a block of it is allocated. Delete the code that modelled another component's allocator: index_column_growth and its shadow index_capacity, BinaryOperand::ends_growth, and the flatten-capacity model in assembly_bytes. Nothing pre-reserves the left-times-right product and no large allocation happens before its reservation. Tests that pinned the callback sequence or exact small-object arithmetic go; the resource properties stay: a genuinely large input or output is refused, a sparse small result is admitted, and reservations are released on finish, error, truncation and drop. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): move the histogram_quantile doc back to its own test The start-timestamp test was inserted into the middle of another test's doc comment, leaving the earlier test without one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): compile a constant, a negation and a scalar-vector operation Three shapes reach the tree now: a constant binds as a `Scalar` on the statement's grid, a negation binds as the multiplication by `-1` it is, and a binary operation between one vector and one constant binds as a `Binary` node whose output is the vector side's own binding. The arithmetic and comparison operators all map, and `return_bool` is carried as its own field so `x > 0` (which filters) and `x > bool 0` (which reports zero or one) are one node with one field changed. The operands keep the order they were written in, because most operators are not commutative: `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them apart. A constant is bound by the same `constant_value` fold a call's parameters already use, so `x * (1 + 2)` carries `3` rather than an addition the tree would have to know how to run. Three shapes are refused rather than approximated. An operation between two vectors needs each `Input` to name the selector it reads, which the one binding does not carry, so `a + b` and `a + a` would compile to the same tree; a set operator combines two vectors' series and has no node at all; and a statement whose value is a scalar has nothing to scan, so a constant is only bound next to a vector that gives it something to broadcast over. Each refusal names what it found. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
Dennis Zhuang
|
47c40eeb01 | chore: merge narthex into feat/narthex-vector-agg-compile | ||
|
Dennis Zhuang
|
b67aa655ac |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
e46f1f63e0 |
fix(promql): keep fill_left and fill_right on their written sides under group_right
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
2f718528d5 | chore: merge narthex into feat/narthex-count-values-node | ||
|
dennis zhuang
|
6c7e534b92 |
feat(promql): run histogram_quantile and histogram_fraction in the native vector node (#9853)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, histogram category (#25). Stacked on #9852 → #9851 → #9850; its own change is the last two commits. Draft until those merge. ## What's changed and what's your intention? `histogram_quantile(q, v)` and `histogram_fraction(lower, upper, v)` compile and run natively, with constant parameters folded by the compiler. - **Node.** A classic histogram is read from the bucket series that share its labels, so these need the whole vector at a point, like the select node's other operators. They run there as `BoundSelect::Histogram(HistogramFunction)` through `evaluate_histograms`. The node now covers selections, sorts, `absent` and these; its name predates that and could become something like `Vector` in a follow-up. - **Pieces.** The kernel groups by label set and takes one native histogram series per set. Pieces of one label set (an empty tag and an absent one, or what a map folded) are therefore merged with `merge_same_labelset` first, and two pieces holding one point fail as they do elsewhere. Without the merge a second native piece was silently dropped and overlapping classic buckets were summed. The output holds one series per label set, so the node does not mark it as holding pieces. - **Deterministic order.** `evaluate_histograms` iterated its groups through a `HashMap`, so the result order varied between runs. Groups now keep first-arrival order, as `merge_same_labelset` does. - **Memory.** The estimate gets its own arm: the merge workspace, per-series group entries and cursors, per-row buckets and results. ### Tests - Runtime: classic buckets of one label set give the quantile without `le` and the name; two native pieces of one label set are answered at both points (fails without the merge). - Compiler: both calls bind with folded constants (`1 - 0.1` → `0.9`). ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
d7d9d9ed5e |
fix(promql): admit the label a label change writes before building it
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
bfe1bab672 | chore: merge narthex into feat/narthex-select-absent-sort | ||
|
Dennis Zhuang
|
6e5137f364 |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/display.rs # src/promql/src/native/execute.rs # src/promql/src/native/execute/tests.rs # src/promql/src/native/plan/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
dennis zhuang
|
fbd5c8b850 |
feat(promql): run topk, bottomk, limitk and limit_ratio as a native select node (#9850)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, Select category (#25). Based on `narthex` directly, independent of the #9845 → #9849 chain. Compiler wiring is not here: it goes through the aggregate compile in #9833, which keeps `topk` and friends out of `AggregateKind` as a different shape. It follows once #9833 merges. ## What's changed and what's your intention? `topk`, `bottomk`, `limitk` and `limit_ratio` get a node and a runtime. - **Plan.** `NativeExpr::Select { input, op: BoundSelect }`, with `BoundSelect::{TopK { k, top, group }, LimitK { k, group }, LimitRatio { ratio }}`. Constants are bound, as for windows. A selection consumes and produces the same evaluated vector. - **Runtime.** Which samples are kept at a point depends on every series there, so the node works like the binary node: it retains its whole input, assembles the chunks of each logical series, runs the kernel from `select.rs` once the input has ended, and queues the results. - `limitk` reads series in label order. Prometheus uses label order once a query spans head and blocks; the scan here does not order distinct series, so this keeps the result stable across runs. - Pieces of one label set (e.g. under a map that dropped the name) stay pieces. They hold disjoint points, so ranking them apart picks what ranking the merged series would, and the output keeps the input's `holds_shared` flag. - Multi-field `topk` / `bottomk` / `limitk` is refused when the execution is built. - Memory: the retained chunks and the assembly are reserved as for the binary node, and a rough kernel workspace estimate (`selection_workspace_bytes`, same convention as `merge_workspace_bytes`) is reserved before the kernel runs. - **Shared code.** `BinaryOperand` becomes `RetainedInput`. The continuation and still-open checks move from `BinaryRuntime::retain` / `closed` into `RetainedInput::retain` / `ensure_closed`, so both nodes use them. Binary behavior is unchanged apart from error wording. `compute_binary`'s queueing is factored into `queue_results`. - **Kernel fix.** `topk` / `bottomk` skipped nothing and failed on a histogram sample; Prometheus ignores histograms there (`aggregationK`), and so does the kernel now. Shared files touched: `native/plan.rs`, `native/execute.rs` (`RetainedInput`, `NodeTag`, build/settle/pull), `native/display.rs`, `native/metrics.rs` (`walk`). ### Tests - Runtime: `topk` ranks a series delivered in two chunks as one candidate and gives the reservation back; `limitk` takes label order over arrival order, with map pieces kept where they hold the point; a binary node above a selection reads what it kept. - Kernel: `topk` skips a histogram sample. - Plan: a selection over raw samples is refused. Display: the three shapes. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
ca3d57cbaa |
feat(promql): compile and run date functions over a vector in the native tree (#9846)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, time/date category (the first small draft agreed on #25). Builds on #9845 (merged), whose `map_math` skips native histograms. ## What's changed and what's your intention? Date functions that take a vector — `day_of_month`, `day_of_week`, `day_of_year`, `days_in_month`, `hour`, `minute`, `month`, `year` — now compile into the native tree and run there. - **No new node.** Prometheus evaluates these like its math functions (`dateWrapper` has the same shape as `simpleFloatFunc`): per float sample, the value read as Unix seconds via `int64`, histograms skipped, the metric name dropped. So `MathOp` gains `Date(DateFunction)`, and the existing `Map` node, display and runtime (`map_chunk` → `map_math`) carry it. Shared files change in a few lines: one variant and one match arm in `map.rs`, one display arm, and the name table in the compiler. - **Compiler.** `math_signature` lists the eight names as unary. A date function written without an argument (`hour()`) reads the evaluation time; it has no input to map, so it is refused as `NotImplemented`, and the message says so. That form waits for the no-input and scalar output interfaces. - **One implementation.** `map_date` had no caller outside its tests, so it is removed. The Prometheus golden for date functions now drives `map_math(MathOp::Date(..))`, the path the runtime takes. A hole in a float field stays a missing row on the input's axis, as with every map, instead of cutting the axis. - **Comments.** `map_chunk`'s comments now account for #9845: a chunk loses the rows only histograms filled, and `None` means no float value is left. Output and labels: a `Map` keeps the input's fields and grid. It drops `__name__`, `__type__` and `__unit__`, so it is one of the nodes that can merge series into one label set, as before. ### Tests - Prometheus golden (`map_time/prometheus_cases.txt`, existing): 100 date cases, including NaN, ±Inf and label set collisions, now through `map_math`. - Compiler: each date name binds its kernel, and `hour()` is refused as `NotImplemented` naming the evaluation time. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Yingwen
|
6a40c0b811 |
feat(mito): implement columnar samples format (#9887)
* docs(mito): specify columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * feat(mito): implement columnar samples format Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): address columnar samples review feedback Signed-off-by: evenyag <realevenyag@gmail.com> * style(mito): use crate-rooted samples imports Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Dennis Zhuang
|
c230fb123a |
feat(promql): compile topk, bottomk, limitk, limit_ratio and count_values into the vector node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
71dbde9b59 | chore: merge feat/narthex-histogram-quantile-node into feat/narthex-count-values-node | ||
|
Dennis Zhuang
|
3ba4476b2f |
chore: merge feat/narthex-select-compile into feat/narthex-histogram-quantile-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |