mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-14 14:56:54 +00:00
de97d868007d23762557166b4f28c1d276608564
6742
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
de97d86800 |
fix(query): only offer the native route to an engine that serves frames
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
638d485024 |
fix(query): decline the native route where it cannot reach a region
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
1ced0f7fdb |
chore(query): share the range assignment with every caller
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
8efec61743 |
fix(promql): keep the fused scan's parallelism
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
5527c751d2 |
fix(query): decline the native route where no raw scan exists
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
129faa4e49 |
chore(query): drop a needless borrow on the scan schema
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
0d29dc98e8 |
fix(query): emit the fused output in the existing column order
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
67a3d22c5b |
fix(query): keep a fused route to series a region owns
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
64f039eaf4 |
test(query): run the encoding checks for both transports
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
6121600789 |
fix(query): keep ordinary plans on the encoding path
Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
da27d115ca |
feat(query): let a region input travel as a plan when regions are in process
A plan holding a node with no Substrait serializer was always cut at that node, because the analyzer asks whether a plan encodes before it asks whether a node commutes with the region split. That is right for a region reached over the wire: the plan is encoded into the request and has to decode on the other side. Standalone does not use that path. Its region server is the one built in the same process, its query handler passes the request straight to `RegionServer::handle_read`, and the plan stays a `LogicalPlan` from the frontend to the region. Encoding it is work nobody reads, and refusing to place a node because it cannot be encoded refuses a plan that would have travelled intact. So the engine now knows whether its regions are reached in process, and the encodability gate applies only when they are not. The capability is set where it can be stated truthfully - the standalone bootstrap, which creates both the frontend and the region server it reads - and it defaults to false everywhere else, so an ordinary build and every other deployment keeps today's behaviour byte for byte. Skipping the gate skips one early return and adds no reason to expand: `Categorizer::check_plan` still runs and still decides, and a plan that does not encode for any other reason still expands at the node that caused it. Two tests hold those two halves apart. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
ba9f36fdd0 |
docs(query): say what the fused placement does not prove
The arm above answers that a fused node commutes with the region split, and it listed "no series is split across regions" among its reasons. That is not established anywhere. A region evaluates the whole tree over the series its own scan returns, so a series whose samples reach two regions is evaluated per fragment, and a range function over a fragment is not the range function over the series. What the placement does have is three proofs about the node - its input is the scan, it binds a metric name, and no node of its tree folds across series. What it does not have is any check that the series stays in one region: the storage partition key is whatever `partition_column_names()` names and may be a column that varies within one series, and this categorizer runs before region selection, so there is no selected-region vector here to consult either. So the comment now says that plainly, names the deployment the placement is correct under, and says what would make it unconditional. A single-region deployment satisfies it; a partitioned table over several regions would answer wrongly rather than fail. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
ff6c5db307 |
feat(query): place a fused node region-side, and ask for per-series scans
A `FusedSeries` node is built for one metric's series, and a region holds a subset of those series with none split across regions, so the node commutes with the region split when what it reads is per-series. It says so now, and the analyzer may place it inside the merge scan. Three conditions, each drawn from the conversion that has to succeed there: - The node's input is exactly the table scan. The region builds the node from the scan's own raw-series entrypoint and cannot look through a filter, a projection or a sort in between, so a node over anything else has nothing to read where it would be placed. - The node binds a metric name, because the region resolves the series of one named metric. - No node of its tree folds across series. A region holds part of a metric's series, so an aggregate or a binary operation evaluated there would consume a partial input and return a wrong answer without saying so. Every operation the compiler emits today - input, instant, window, map - reads one series at a time, so this refuses nothing buildable; it is the conservative answer for a tree built some other way. A scan below such a node also asks for the per-series distribution now. That is what makes a region prepare its series scan, and only that path packs a series' chunks into the frame the node reads. Both are inert without a `FusedSeries` node, so an ordinary query keeps its plan and its scan request. Not yet reached: the analyzer asks whether a plan encodes before it asks whether a node commutes, and this node has no Substrait serializer, so the plan is expanded at the node and the region still receives the bare scan. A test pins that boundary so it fails the day the node can cross it. Signed-off-by: discord9 <discord9@163.com> |
||
|
discord9
|
4beb5c7490 |
feat(query): offer a PromQL statement to the native fused-series route
Nothing called `native_compile::compile` or built a `FusedSeries`, so the native tree and its node were library code with tests and no query behind them. This adds the route, behind a per-query flag. The statement is offered to the compiler at the root, where the expression the reader wrote still exists. A shape the compiler refuses - `NotImplemented` - is this route declining the query, and planning continues exactly as it would with the flag off. Anything else is a real error. The physical half is an arm in `PromExtensionPlanner`. Its children are already built by then, so a metric scan arrives as the region scan, and the node asks that scan for its named raw-series binding: `try_to_named_series_scan` resolves the label source and ordering from the scanner itself, which is the only thing that can name them, and returns the leaf the node reads. A scanner that cannot serve a raw frame is reported, not silently re-planned. Two things this needed that the route does not get for free: - The region scan has to be prepared as one partition. `resolve_named_series_binding` refuses a scan whose partition count is not one, and a fresh scan has not set its partitions at all: `ParallelizeScan` runs after extension planning and would spread it, so the node would run one evaluator per partition and split a logical series. Preparing the ranges as one partition is what keeps `FusedSeriesExec` over one series stream, and it is why this route does not scan in parallel yet. - The scan needs the per-series distribution hint. That is what makes a region prepare its series scan, and only that path packs a series' chunks into the frame this node reads. The existing planner asks for it through a sort that `ScanHintRule` turns into the hint; this node has no such sort below it, so the hint is attached to the scan directly. A query the route cannot bind faithfully keeps the existing plan rather than a wrong one: more than one selector, a label matcher that would survive preprocessing, an `__field__` matcher, a native histogram field, a missing table, or a shape whose grid the compiler's statement cannot express. Signed-off-by: discord9 <discord9@163.com> |
||
 dennis zhuang
|
8497382673 |
feat(promql): compile and run label_replace and label_join as map operations (#9848)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, label category (#25). Builds on #9846 (date wiring, merged). ## What's changed and what's your intention? `label_replace` and `label_join` now compile into the native tree and run in the existing `Map` node. - **Plan.** `BoundMap.ops` becomes `Arc<[MapOp]>`, with `MapOp::Math(MathOp)` and `MapOp::Label(Arc<LabelRewrite>)`. A label change is pointwise per series like math, so it needs no new node, collector or output type. - **Kernel.** `LabelRewrite` (in `map_label.rs`) is the checked, compiled change: `replace` anchors the regex and checks the destination, `join` checks source and destination names, `apply` rewrites one `MetricName`. The whole-vector `label_replace` / `label_join` now go through it and still merge same-labelset series afterwards, so the Prometheus golden keeps covering the same code. Fields are private, so only the two constructors build one. - **Runtime.** `map_chunk` applies label ops to the series metric and math ops through `map_math`. When a math op empties a closing chunk, the labels of the closing series come from applying the remaining ops in order (label ops rewrite, math ops drop the name). Series a label change folds into one label set go through the `Emitted` ledger the map already keeps: disjoint values merge, a shared point is refused. - **Compiler.** `label_rewrite` reads the string-literal arguments. Arguments Prometheus rejects at evaluation (invalid regex, empty label name) fail the statement with `Plan`, as Prometheus fails the query; everything unsupported stays `NotImplemented`. - **Display.** `label_replace "dst" "repl" "src" "regex"` / `label_join "dst" "sep" ["a", "b"]`. Shared files touched: `native/plan.rs` (`MapOp`), `native/execute.rs` (`map_chunk`, `no_row_series` now takes the metric), `native/display.rs`, `native_compile.rs`. Test constructors move from `MathOp` to `MapOp::Math`. Known difference, not introduced here but now reachable: Prometheus checks same-labelset duplicates separately for floats and histograms, while the native ledger counts both as a value at a position. A `label_replace` that folds a float series and a histogram series onto one label set at the same step is refused natively. ### Tests - Runtime: a relabel that folds two series refuses the shared point; a closing chunk emptied mid-chain carries labels from label ops on both sides of the emptying op; a label-only map hands histogram samples and holes through unchanged. - Compiler: both functions bind their change into a map; an invalid regex fails the statement with `Plan`. - `map_label` Prometheus golden (existing) now runs through `LabelRewrite`. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
3674b33db8 |
fix(promql): keep fill_left and fill_right on their written sides under group_right (#9899)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) prometheus/prometheus#19939 ## What's changed and what's your intention? The native binary kernel applied `fill_left` / `fill_right` after swapping the operands of `group_right`, so under `group_right` `fill_left` filled a missing right-hand sample and `fill_right` a missing left-hand one. That is the Prometheus behavior before prometheus/prometheus#19939, which was merged into Prometheus main on 2026-10-13: the fill values now swap together with the operands, so they always refer to the side written in the query. For example, `node_meta - on(instance) group_right fill_left(5) fill_right(7) cpu_info`: a missing `node_meta` must give `5 - cpu_info` and a missing `cpu_info` must give `node_meta - 7`. - `binary_field` swaps `fill.lhs` / `fill.rhs` when the one side is the left, matching `VectorBinop` after #19939. The `FillValues` doc now states the as-written meaning. - `fill_cases.txt` is regenerated by promql-compat `narthex-golden-vector -experimental fill` built against Prometheus at the #19939 merge commit (same seed and case count). Only the 24 `group_right` cases whose fill fires on an asymmetric pair change. No released Prometheus contains #19939 yet (v3.15.1 still has the old behavior), so until the next release native results differ from released Prometheus for asymmetric fills under `group_right`. The compiler does not bind vector-vector fill modifiers yet, so this is not reachable from a query today; it has to land before that wiring does. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
dennis zhuang
|
1c63b06265 |
feat(promql): compile topk, bottomk, limitk, limit_ratio and count_values into the vector node (#9890)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, Select / aggregate categories (#25). Follows #9850 → #9851 → #9852 → #9853 → #9854 (all merged) and #9833. ## What's changed and what's your intention? `topk`, `bottomk`, `limitk`, `limit_ratio` and `count_values` now compile. #9833 left them out of `AggregateKind` because they do not fold a group into one value per point. They now bind as a `Vector` node through `vector_aggregate`: - `topk` / `bottomk` / `limitk`: `k` is folded to a constant as `quantile`'s parameter is, and the `by` / `without` grouping is kept. A `k` only known at run time is refused with `NotImplemented`. NaN, negative and out-of-range `k` are left to the kernel, which fails or answers at evaluation as Prometheus does. - `limit_ratio`: the ratio is folded the same way. The grouping is dropped, since Prometheus keeps or drops each series by its label hash alone. - `count_values`: the label is a string literal. Label validation stays in the kernel, an execution error as in Prometheus. The module doc, the grammar and the `SUPPORTED` text name the new shapes. ### Tests - Compiler: each aggregator binds its operation with the folded parameter and grouping. A run-time `k` is refused, and the rejection quotes every aggregator this compiler takes. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
13f4fb9cdb |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs |
||
 Yingwen
|
ce4fc4529e |
feat(mito): read columnar series through prepared sources (#9892)
* feat(mito): read columnar series through prepared sources Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): rebuild schema-incompatible columnar sidecars Signed-off-by: evenyag <realevenyag@gmail.com> * docs(mito): regenerate series format configuration Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito): batch prepared series reads and preflight inputs Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
Dennis Zhuang
|
4ca04d2790 | chore: merge narthex into fix/narthex-group-right-fill | ||
|
Dennis Zhuang
|
968f5a4525 | chore: merge narthex into feat/narthex-vector-agg-compile | ||
|
Dennis Zhuang
|
5e621d481b | chore: merge narthex into feat/narthex-vector-agg-compile | ||
|
dennis zhuang
|
0356a1607a |
ci: bring the narthex branch's CI workflows up to main (#9912)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Unblocks the CI checks of #9868 and the rest of the `narthex` stack. ## What's changed and what's your intention? `narthex` is a long-lived feature branch that has fallen behind `main`, so its CI still runs the older workflows. Three checks fail for reasons that have nothing to do with the branch's code: - **Prepare Fuzz Targets** — the setup step installs `libfuzzer-14-dev`, which is no longer available on the current runner image. - **Run Multi-language SDK Tests** — the job asks for Python 3.8, which the current runners no longer provide. - **Clippy** — the failure is inside `cargo clippy`, not in the setup steps, so it is not addressed here; see the note at the end. This PR cherry-picks the CI commits that landed on `main` but not on `narthex`, so the branch's workflows match `main` again. It changes **no Rust code** — all 33 changed files are workflow definitions, composite actions, `Makefile` targets, `.github/scripts/`, and one how-to doc. The commits are applied with `-x` so each carries the upstream SHA, and they apply without conflicts. ### What is picked | Commit | Why | | --- | --- | | `97e3bfd547b` | manual S3 metric snapshot benchmark | | `f1fb5e4e7a5` | `RUST_MIN_STACK=8MB` for the merge_group coverage job | | `813b132221f` | compare release validation against the previous stable version | | `b2b606115b7` | crash/restart/multi-region/drop cells for the S3 WAL benchmark | | `db61aecadbc` | do not run these actions on a fork | | `a0a7ec02b5d` | compare several images in one object store WAL S3 run | | `fdb8ceeb542` | increase release integration test stack size | | `f855355ed04` | independent Linux release architecture selection | | `630cca33956` | align nightly Windows and macOS artifact test stack sizes | | `99d907cf1d2` | **fixes the two failing jobs above**: drops `libfuzzer-14-dev`, moves the SDK tests to Python 3.13, installs protoc via `taiki-e/install-action` | | `6e40daed98e` | move `monitoring.datanodeStorage` out of the standalone `base` | | `de0fea673eb` | specify the Kind storage class for datanodes | ### Limitations The Clippy failure is **not** fixed by this PR. Its setup steps (checkout, protoc, toolchain, cache) all succeeded and the failure is in the `cargo clippy` step itself, so it needs its own diagnosis — a lint in the branch's code or a toolchain difference. I could not retrieve that job's log to say which. `kafka-wal-helper.yaml` is a pre-existing multi-document YAML file; it is not touched here and not affected by these picks. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
 Weny Xu
|
de0fea673e |
fix(ci): specify Kind storage class for datanodes (#9908)
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit
|
||
 liyang
|
6e40daed98 |
fix(ci): move monitoring datanodeStorage out of standalone base (#9902)
The `datanodeStorage` field belongs directly under `monitoring.standalone`
(GreptimeDBStandaloneSpec), not under `monitoring.standalone.base`
(PodTemplateSpec). Nesting it under `base` renders an invalid
GreptimeDBCluster resource and the CRD schema rejects it during
server-side apply:
.spec.monitoring.standalone.base.datanodeStorage: field not declared in schema
(cherry picked from commit
|
||
|
dennis zhuang
|
99d907cf1d |
ci: fix CI setup steps that fail on Ubuntu 26.04 runners (#9901)
* ci: install protoc with taiki-e/install-action
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: run multi-language SDK tests on Python 3.13
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: drop the unused libfuzzer-14-dev package from fuzz builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: build fuzz targets with the bundled libFuzzer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit
|
||
|
Yingwen
|
630cca3395 |
fix(ci): align nightly Windows and macOS artifact test stack sizes (#9860)
* fix(ci): increase Windows nightly test stack size
Signed-off-by: evenyag <realevenyag@gmail.com>
* fix(ci): align optional macOS artifact test stack size
Signed-off-by: evenyag <realevenyag@gmail.com>
---------
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit
|
||
|
Weny Xu
|
f855355ed0 |
ci: allow independent Linux release architecture selection (#9865)
* ci: allow independent Linux release architecture selection
Signed-off-by: WenyXu <wenymedia@gmail.com>
* ci: use Linux architecture checkboxes and preserve release defaults
Signed-off-by: WenyXu <wenymedia@gmail.com>
---------
Signed-off-by: WenyXu <wenymedia@gmail.com>
(cherry picked from commit
|
||
|
Yingwen
|
fdb8ceeb54 |
fix(ci): increase release integration test stack size (#9817)
Signed-off-by: evenyag <realevenyag@gmail.com>
(cherry picked from commit
|
||
 jeremyhi
|
a0a7ec02b5 |
ci: compare several images in one object store WAL S3 benchmark run (#9691)
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit
|
||
 yihong
|
db61aecadb |
ci: do not run these actions on fork (#9721)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
(cherry picked from commit
|
||
|
jeremyhi
|
b2b606115b |
ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark (#9632)
* ci: add crash, restart, multi-region and drop cells to the S3 WAL benchmark
Extend the object store WAL S3 benchmark beyond one load per configuration:
- Inputs for a full image reference, the configurations to run (R0 adds Raft
Engine with the standalone default sync_write = false), extra environment
for every server container, a restart after each load, and a JSON list of
named cell groups that override those inputs.
- Restart cells SIGKILL the loaded server, restart it on the same prefix and
data home, and time /health, the first correct count and the log store
recovery; min_objects fails a cell that left too few objects indexed.
- Crash groups write striped 1000-row batches from several writers, kill the
server at a random point, restart it and check every acknowledged batch
against the table (durable: present exactly once; enqueued: no duplicate,
no partial batch, losses counted). A pinned variant keeps one row of a
second table unflushed and samples the indexed objects while it runs.
- Idle windows scrape the server before and after a quiet period.
- A load group may split the loaded table into equal event_time ranges, so
every batch writes to every region, and checks that each region took rows.
- Drop groups write a table, drop it, kill and restart the server, then
write and flush another table and wait for the dropped table's objects to
be collected.
- The summary adds WAL traffic, objects per second, mean object size,
entries and appends per object, S3 requests per operation, server and host
CPU, the loader's own request latency, rows per region, crash cycles, and
each restart's replay per region from the server log.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: report the object sequences a replay spans instead of objects replayed
The width of a region's replayed entry range counts every object sequence in it, including objects that hold no entry of the region and are skipped, so the summary labels it as a span.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: fail a crash cycle that loses a batch an earlier restart recovered
The oracle accepted every missing acknowledged batch in the enqueued mode,
so a batch that one restart recovered and a later restart no longer found
passed. Every cycle keeps what its restart found, and a cycle now fails in
either mode unless every batch an earlier restart recovered is still
present, so only the current cycle's backlog that was not durable at its
kill may be missing.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: keep the pinned WAL pinned and label OpenDAL operations as such
The pin table inherited the engine's 10-minute auto flush, so a longer
pinned cycle could flush its row and release the WAL before the kill. It now
sets its own auto_flush_interval of one day, longer than any crash group can
run, and a pinned cycle fails unless its indexed objects never fall while
it runs.
The summary's request counts come from opendal_operation_duration_seconds,
which counts OpenDAL calls rather than HTTP requests (a paginated list or a
multipart write is one operation), so the fields and columns now say S3
operations.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check a pinned cycle's retention by the collected-object counter
Net growth of the indexed objects can hide collection, and short cycles take no periodic sample, so a pinned cycle now fails unless deleted_objects_total is the same in the scrape before its writers start and in the one before the kill, both of the same server process. The operation columns name OpenDAL.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: let a pinned cell collect only the objects below its pin
The first write after a start lets the store collect the empty start object asynchronously, so a deletion after a cycle's baseline does not mean the pin let go. A pinned cell now scrapes the server before it writes the pin row and fails only once its servers together have collected more objects than were indexed then, counting the first server from that scrape and every restarted one from its start.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: test that a pinned cell's allowance spans its server restarts
Two servers that each collect fewer objects than lie below the pin but more together now fail the cell in the second cycle.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit
|
||
 localhost
|
813b132221 |
fix(ci): compare release validation against the previous stable version (#9636)
Ancestry-based tag selection skips stable tags that live only on older release branches.
Signed-off-by: paomian <xpaomian@gmail.com>
(cherry picked from commit
|
||
|
discord9
|
f1fb5e4e7a |
ci(coverage): set RUST_MIN_STACK=8MB in merge_group coverage job (#9615)
The merge_group coverage job did not set RUST_MIN_STACK, so test threads
ran with the default 2MiB stack while the regular test job (and the
production runtime in debug) use 8MiB. The deeper PromQL planning chain
from #9494 overflows 2MiB under llvm-cov instrumentation, aborting
tests::promql_test::anon_promql_ratio_repro::case_1_test_with_standalone
with SIGABRT (12/12 failures across three merge-queue candidates).
Align the coverage job with the existing 8MiB test policy.
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit
|
||
|
jeremyhi
|
97e3bfd547 |
test: add manual S3 metric snapshot benchmark (#9573)
* test: add repeatable metric snapshot S3 benchmark
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: use only existing CI S3 permissions in the metric snapshot benchmark
The CI S3 user is not allowed GetBucketLocation or ListBucketMultipartUploads,
so the bucket probe and the prefix cleanup failed with AccessDenied. Use only
list, put, get and delete under the run prefix, as the object store WAL
benchmark does, and probe S3 before allocating the EC2 runner.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: scope metric snapshot inventory to the current round
The full round's S3 inventory listed the whole run prefix, so it also counted
the smoke snapshot. List only the round's snapshot prefix while cleanup still
covers the run, and assert that the teardown's S3 cleanup step runs always()
with the validated run prefix.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* docs: record the AWS validation run of the metric snapshot benchmark
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit
|
||
|
dennis zhuang
|
069b942518 |
feat(promql): run count_values in the native vector node (#9854)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, aggregate category (`count_values`, #25). Stacked on #9853 → #9852 → #9851 → #9850; its own change is the last commit. Draft until those merge. ## What's changed and what's your intention? `count_values(label, v)` runs natively as `BoundSelect::CountValues { label, group }` through `aggregate::count_values`. - **Why this node.** `count_values` writes one label set per distinct value, which #9833 leaves out of `AggregateKind` as a different shape. It reads the whole vector at a point, which is what the select node does. - **Output.** Groups are keyed by their final label set, so each result label set is one series and the node does not mark its output as holding pieces. Input pieces of one label set hold disjoint points, so their counts add. - **Memory.** The estimate charges a group per row, since a gauge with distinct values makes about one group per sample: the group's label set twice with the value label, its counts and its result series. Not here: compiling `count_values`, which is an aggregate expression and goes through #9833. Before it is compiled, it needs a decision on native histogram input. Prometheus counts histograms by their string form; the kernel refuses them with `NotImplemented`. Until the kernel covers them, the compiler has to refuse such input. ### Tests - Runtime: values shared across series are counted per point, including a series delivered in two chunks. - Display: the `count_values` line. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
85c5dc0d69 |
test(promql): pin the refusal to a kind the runtime still refuses (#9900)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
Dennis Zhuang
|
b71676e72d |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile/tests.rs |
||
|
discord9
|
3b09dd75dc |
feat(promql): run a bound tree over raw series in one fused node (#9820)
* feat(promql): run a bound tree over raw series in one fused node
`FusedSeriesExec` evaluates a compiled `NativeTree` directly over the raw
series its child produces, instead of materialising a row batch through the
per-operator chain. It binds the time column, the fields and the tags once,
checks the input against that binding when the plan is built, and converts the
chunks the tree settles back into the statement's output columns.
The node's partition count is the child's, one tree per partition. The child
must be a raw-series scan that assigns every series whole to one partition:
`PerSeries` distribution is what makes a per-partition evaluation equivalent to
evaluating the whole series. A zero-partition input is refused; anything else
is the child's business.
A label set can recur across partitions once `rate` and the map kernels drop
`__name__`, so the occupancy map is plan state rather than per-stream. Two
series reaching one label set is an error only when their occupied grid points
overlap, which is Prometheus' range-query rule; disjoint points are legal and
are merged into one occupancy record. The reservation for that map is attached
to the first execution's pool and released when the plan is dropped, because a
later partition still has to collide with what an earlier one emitted.
`occupied_runs` reads a chunk's absolute positions rather than assuming one
contiguous run: a sparse evaluated chunk may hold several, and treating the
first as the whole range would count the holes as rows.
A poll does a bounded amount of work and wakes itself when the budget runs out,
so a long run of chunks that settle nothing cannot monopolise the executor
thread.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): charge for the label sets the plan keeps, not for one stream
The occupancy map is the plan's: entries are inserted by whichever partition
sees a label set first and no path removes them. The reservation paying for
them was the stream's, so the bytes went back when that stream ended while the
entries stayed. Measured on a plan with three partitions producing three label
sets: each grew 42 bytes, each stream's end returned them, and the pool then
read 0 while every entry was still there and still colliding with later
partitions. The plan's own drop produced no shrink at all.
So a label set's bytes were charged for part of the time and invisible for the
rest, and the invisibility grew with the number of label sets a query touches.
The charge now belongs to the map it pays for: `LabelSets` holds the entries
and the reservation together, `attach` registers it once on the first
execution's pool, and it is released when the plan is dropped.
`attach` is idempotent because every partition calls it. Re-registering would
move a charge the other partitions keep growing, and would give it back when
one stream ends while the entries stay - which is the shape of the bug.
A regression test states the invariant: after the first partition's stream
ends, the pool is still charged for its label set; a second partition with the
same set on disjoint points adds no further charge; dropping the plan returns
it to zero.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): take a raw series frame, not a row batch, and keep its series whole
Review found five things; this is all of them.
The node read ordinary Arrow rows and converted them itself, so a real
`SeriesScanExec` could never feed it: a frame's columns are `read_key`,
`is_last`, and the `time` and `fields` structs, and none of them is the time
column, a tag or a field the binding names, so construction refused it. It now
takes a raw series frame the way `NativeInstantManipulateExec` does - the frame
is validated when the plan is built and unpacked per partition when it runs -
and the row-to-series mapping is gone, because turning rows into series is a
different node's job. The series schema comes from the tree's own input node,
so there is one statement of it rather than two that could disagree.
A raw frame is a per-series stream, and this node holds one window per series
and closes it on the chunk that marks the end. Declaring that as
`KeyPartitioned(read_key)` keeps the optimizer from splitting a series across
partitions, which it would otherwise be free to do; `read_key` is the series'
primary key as a binary column, so hashing it groups exactly the rows of one
series. A round-robin split would have let each partition rate half a series
and answer half the truth; with the key declared, a series that somehow did
arrive in two pieces is refused by the frame validator instead.
Two label-set rules were wrong. Occupancy was tracked per position for any
field, so two series that each hold a different field at the same point looked
like a collision, while the rest of the stack merges exactly that as
complementary fields - it is now tracked per field, and only one field holding
two values at one point is refused. And the key encoded every tag including
empty ones, while `grid_series::label_set` drops them, so `{host="h", zone=""}`
and `{host="h"}` were two keys here and one everywhere else; the key is now
that same normalized label set.
Finally, a label set already in the map grew its run arrays on every merge
without telling the pool. The charge now follows the capacity the merge adds,
so a long-lived plan that keeps meeting one label set pays for what it holds.
The test that asserted the opposite was wrong and is corrected.
Signed-off-by: discord9 <discord9@163.com>
* fix(table): declare the distribution a raw series frame already has
`SeriesScan` guarantees every series arrives whole in one partition - its own
docs say so, and both modes enforce it in different ways: the two-phase
collector places a series by its tsid, and the legacy divider only emits a
batch once it holds all of a series' rows. But the leaf built for a raw frame
declared `UnknownPartitioning`, so nothing on the plan said it.
That mattered to a consumer holding one window per series: with the
distribution unspecified an optimizer is free to round-robin the partition,
and two halves of one series would each look final to their own stream. So the
leaf now declares `Hash(read_key)`. `read_key` is the primary key as a binary
column, so hashing it groups exactly the rows of one series and splits nothing;
`FusedSeriesExec` asks for the same key and is satisfied without a shuffle.
The three tests that pinned `UnknownPartitioning` now pin the key instead.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): keep the capacity a label set already paid for
`merge_runs` built a fresh vector sized to `stored + new` and replaced the old
one. When adjacent runs joined - or when a claim added nothing to a field,
which is what two complementary label sets do to each other - the entry came
back smaller than it was. The charge in `claim` is the growth over the
capacity, so the next merge into that field looked like growth again and was
charged for it, every time, while the entry itself never held more than a
couple of runs. A long query meeting one label set repeatedly would have paid
for the same few bytes over and over and could exhaust a pool it never used.
The merge now keeps whichever capacity is larger, and the charge follows that:
a merge that adds nothing to a field costs nothing.
Signed-off-by: discord9 <discord9@163.com>
* fix(promql): run the fused node over one series stream
`FusedSeriesExec` held one window per series and closed it on the chunk carrying
`is_last`, so it can answer a series only if nothing splits that run and nothing
puts part of it in another tree. Those are two different promises, and only the
first is a property of a stream: the frame decoder refuses a chunk interleaved
with another series', but nothing it can see says that half a series is in a
partition it never reads.
The node used to ask for `KeyPartitioned(read_key)` instead, expecting the
optimizer to keep whole series per partition. That does not hold: DataFusion
considers two hash layouts with the same count co-partitioned, and above
`subset_repartition_threshold` children it adds a hash exchange even when the
requirement is met (`enforce_distribution.rs`, the
`!partitioning_satisfied || target > child` branch). A hash exchange does not
preserve order, so series chunks interleave and the second half of a series is
answered as a series of its own.
So the count is now fixed at one, in `try_new` and in the reported properties,
and `benefits_from_input_partitioning` is false so no optimizer splits it for
parallelism. This is the phase-1 contract: reading a series spread over
partitions needs an exchange above this node, which does not exist yet.
Merging #9786 also moved `unpack_raw_series` to two arguments, with the frame
declaring its own label layout and ordering; the calls follow it, and the plan's
binding is checked against that declaration with
`ensure_frame_matches_label_source`.
Signed-off-by: discord9 <discord9@163.com>
* Revert "fix(table): declare the distribution a raw series frame already has"
The declaration was false. `Partitioning::Hash(read_key, n)` claims DataFusion
placed rows by its own hash of `read_key` mod `n`, and DataFusion treats any two
equal-count `Hash` layouts as co-partitioned, so a future co-partitioned consumer
would pair partitions that hold unrelated series. `SeriesScan` places a series by
tsid range (two-phase) or round-robin (legacy), neither of which is that hash.
It also did not buy what it was added for. `FusedSeriesExec` now takes one input
partition and reports `benefits_from_input_partitioning = false`, so nothing
repartitions between the scan and it, and the ordering guarantee the declaration
was meant to carry is the frame's own.
This reverts commit
|
||
|
discord9
|
a8737d0996 |
test(promql): skip an endpoint the platform cannot represent (#9894)
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1205c18496 |
feat(promql): execute Complete aggregates in the native runtime (#9873)
* feat(promql): execute a bound native tree node by node, over sparse evaluated series Ports the native runtime onto the sparse evaluated-series API: a chunk is addressed by a `TimeAxis` that may hold several runs, so a consumer reads positions through the axis instead of assuming `start + row`. The driver and both node kinds move to that API: - `chunk.range().len()` becomes `chunk.series.time.len()`, the compact row count, in every metric and byte estimate. - `occupied_runs` walks `time.positions()` with one field cursor, so a chunk of several runs records each run where it belongs. A run open across a hole now ends where the hole begins; leaving it open to the last row made two runs look like one and refused a series that occupies the hole. - A window keeps the pieces its evaluator decided in a queue: one push can decide several runs, and reporting only what the push returned stranded the rest until the next push. Queued pieces are charged by `retained_bytes`, and the charge is recomputed when one leaves, so a released piece is not held. - `map_chunk` answers `Option`: an operation that empties every position (`clamp` with `max < min`) yields no chunk at all, unless the chunk closes its series -- a closing chunk holding no row is handed over so a consumer can tell "nothing more here" from "no more input". `RangeEvaluator::retained_bytes` is restored with the three byte helpers it reads: a field's buffers, a series' identity and the pending batch's vectors, all by capacity, and still counted as storage rather than as a peak over a push. Tests read absolute positions from the axis (`decided` no longer adds a row number to a start), and two sparse cases cover what the old reading got wrong: a `Window -> Map -> Window` chain over a chunk of two runs, and a series that occupies the hole between another's runs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): drop the window queue the sparse evaluator never needed `RangeEvaluator::push` and `push_series` each return `Result<Option<EvaluatedChunk>>`: the evaluator gathers every run it decided into one chunk and hands back at most one. So `pending.extend(decided)` inserted at most one chunk and `take_pending` removed it on the spot, leaving the queue empty after every successful push. That makes the queue dead weight: a `VecDeque` allocation per productive window, an accounting term that is always zero, a second drain site in `pull`, and doc comments claiming a problem the API does not have. The driver's "ask the input again when a window decides nothing" loop was already the whole story. Back to returning the evaluator's own answer, and drop `pending`, `take_pending` and `take_window_output`. Covers the empty-map protocol this port introduces while it is here: an operation that empties every position hands on nothing while its series is open, and one empty chunk when it closes. A non-closing empty chunk would be rejected downstream and a dropped closing one would leave the series unclosed. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): pin the empty-map output count at one The test checked that every output was empty and that exactly one closed, which a stray extra non-closing empty output would also satisfy. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge the memory the native evaluator actually retains Four omissions made the reservation blind to most of what a window over native histograms keeps, and the existing incremental-equals-recomputed test could not see any of them because both sides used the same formula. - `FloatHistogram` owns four heap allocations -- both bucket lists, the custom bounds, and the span layout -- that `FieldSamples::bytes` never looked at. A retained 4096-bucket sample is tens of kilobytes against a ledger that saw a struct header. `heap_bytes` counts them, and both the retained sample slice and the histograms a row boxes are charged for what they own. - Queued chunks were estimated as one float per field per row, which is not what a histogram field holds: it holds indices into a shared `Arc<HistogramBatch>` and the buffers behind the Arrow arrays. The figure now walks the samples, and counts a shared table or a shared buffer slice once per pass rather than once per chunk that references it. - `history_bytes` included `size_of::<History>()` while the map slot term charged a `History` per entry as well, so every entry paid two headers. The per-entry figure now covers only the heap the entry owns. - The primary key is charged for the allocation its `Buffer` keeps alive rather than the bytes it shows: a scan slices the key out of a decoded dictionary-values buffer, so a short key can pin a batch-sized allocation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): keep the source queue's charge incremental The queue's reservation was recounted by walking every queued chunk on each enqueue and each pop. `push_raw` and `push_evaluated` are public and put no bound on how deep the queue gets, so a caller that queues a whole series before pulling it paid `N(N+1)/2` chunk visits to hold `N` chunks -- the same shape of defect this branch fixes in `absorb`. `QueuedCharge` keeps a reference count and the bytes behind each shared allocation: a table or buffer is charged when the first queued chunk reaches it and given back when the last one leaves, so an operation walks the one chunk it names. The count is per chunk rather than per field, because a pop re-walks the same chunk with the same dedupe and must remove exactly the references that were added. A chunk's own struct bytes are counted per chunk (`own_bytes`); only the allocations behind its fields are counted by address, since a struct has no address to dedupe on. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * perf(promql): append a forward-moving chunk to the emitted history `absorb` rebuilt the whole stored run array on every non-empty chunk. Legal input made that quadratic across chunks with no label collision: one series in `K` chunks, chunk `i` occupying only grid position `2*i`, stores one more run per chunk and moves `K(K+1)/2` of them -- about five billion for a hundred thousand rows. A chunk that starts at or past the end of the last stored run only ever adds to the end, so its runs are appended and the boundary run is joined where it touches. One that can interleave or reach backwards still takes the full one-pass merge, which can split a run. Both leave the runs sorted, disjoint and joined, which is what `Emitted::occupy` looks a position up by. The test records the chunks through `Emitted::occupy` itself and reads the history back from it, covering the forward case, the touching boundary, a hole that has to join both neighbours, and a position before everything stored. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): report a truncated input from a tree with no window `close` only asked windows whether a series was still open, so a tree of an input alone, or of a map over one, accepted a final chunk without `is_last` and handed the consumer a series that never closed. The input node now follows the series of the chunk it last handed over, and closing such a tree reports the truncation against it, which is the same refusal a window makes for the input it reads. `occupied_runs` read a row the axis names with `expect`, which panics on a `Series` whose fields are shorter than its axis; the fields are public, so that chunk is constructible by a caller. It now returns the misalignment as an error naming the node, the field and both lengths. The chunk protocol says one series closes once, which reads as one close per label set. State that it is per input series: a node that rewrites labels can hand two series over under one label set and both of them close. Also say in `Emitted`'s doc that the table is one node's own, so a label set costs its bytes again in every node that keeps one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute an instant selection as a tree node The tree could not run `NativeExpr::Instant`: the state builder recorded `not_implemented` and refused the whole tree before reading a chunk, while the compiler already produced the node for a bare selector. A query the plan and the compiler both accept therefore had no runtime path at all. `InstantRuntime` buffers one series to its `is_last` chunk -- the selection kernel reads a complete axis -- then calls the shared `select_instants` once and gathers the rows it picked onto the binding's own grid. Positions come from the grid lattice the kernel was asked for, and a timestamp off that lattice is an error rather than a skipped row. A series that selects nothing still closes, with an empty chunk on the grid; an input that ends inside a series is reported on `finish()`. The node does not record label-set occupancy: it keeps the metric verbatim, so it cannot map two series onto one label set, and the occupancy key drops empty-valued tags -- two genuinely distinct series would be refused as a collision. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): refuse an interleaved input, and close a label set only for its owner Two series-level identity holes, both reported against `42ea525`. The truncation flag followed the last chunk handed over, which is the same as "is a series open" only while at most one series can be open. A tree with no window still accepted `Interleaved`, and `a` open, `b` closed, end of input then passed while `a` never closed. Tracking the open set would need an identity that a raw chunk in bound-label mode does not have (`metric` is `None`, the key holds it), so the ordering is refused when the state is built instead, for every tree shape and not only under a window. The flag's doc now says what it means and why that is the whole answer. `History` closed on any series' `is_last`, so with `abs` dropping the metric name a series that contributed nothing could close a label set another series was still filling: `a{job="x"}` value, `b{job="x"}` empty close, `a{job="x"}` continuation was refused with "already closed the label set". Contiguous input hands over exactly that sequence, so it was not an interleaving artefact. A history now remembers the series that filled it and closes only for that series' closing chunk; a series that produced nothing under an open set is passed over, as it already was under a closed one. The owner is charged as part of the entry. The ordering check also reads every input node rather than only the first: a tree with two inputs would otherwise have its second declaration read and dropped. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve the instant gather's workspace and keep the histogram start Two findings from the review of the instant node. The gather allocates after the last chunk arrives: the concatenated axis, the descriptors that read it, the two arrays the selection returns, and the rows the builder keeps. Nothing asked the pool for any of it, so with one sample selected by every point of a million-point grid the node took tens of megabytes while its reservation fell to the buffered chunks' size -- nothing at all, on the chunk that ends the series. `gather_bytes` now derives a bound from the values in hand (rows, chunks, segments, fields, points, and the widest histogram row per field, since one row can be selected by every point), and the reservation grows by it before the gather runs. It shrinks back on both the success and the error path, so the workspace never outlives the call. A selection reads an existing sample, so it has to carry that sample's payload. `FloatHistogram` had no `start_timestamp` and `push_to` wrote `None`, which lost it for any histogram that had one: `detect_start_timestamp_reset` reads that field to spot a counter reset, so a window over this output would mis-detect resets. The field is now carried through a view and written back; the kernels that compute a histogram instead of reading one leave it `None`, which is what they produced before. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): describe the gather test by what this branch does The doc on `an_oversized_selection_is_refused_by_the_pool` said the kernel has no point limit any more. It does on this branch: the limit is removed on its own branch, which this one does not carry, so the comment described a state that has not arrived. What is true here is the order: the estimate now applies to every gather, so the pool is asked before the kernel is, and the pool's refusal is the one the caller sees. That is worth saying on its own -- a bound in bytes on the path that decides the points, rather than a count that says nothing about what they cost -- and it stays true after the limit goes. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): execute vector binary nodes in the native runtime Checkpoint the vector-vector runtime and single shared matching kernel. Keep addressed input delivery, whole-input buffering, nested composition and error cleanup together. The existing arithmetic/comparison implementation remains the only computation path. Local review checkpoint only, not ready for publication: all-targets check passes; the current run has three new fixture failures. Independent review requires current-live workspace resizing, field-local matched storage and kept-value merge budgeting before approval. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): infer the binary result fixture element type Drop a redundant complex type annotation flagged by the actual strict clippy run. The collection and literal assertions are unchanged. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve binary workspace along the shared kernel Remove the separate matching prepass and reserve current live workspace at the existing kernel allocation boundaries. Release per-sample and per-field scratch after ownership ends; charge actual retained rows, values, histograms and errors. Unbudgeted public and set callers bypass accounting without another computation path. Validate with cargo fmt -p promql -- --check; cargo check -p promql --all-targets; cargo nextest run -p promql --retries 0 --no-fail-fast; cargo clippy -p promql --all-targets -- -D warnings. The public nested Binary test also fails on swapped left/right operands and passes after byte-identical restoration. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): scope binary reservations to live scans and output rows Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): charge binary buffers at actual allocation boundaries Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): skip accounting ledger for unbudgeted vector merges Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): simplify native execution memory estimates Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): match binary operands by logical series Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): cover flattened operand descriptor capacity Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): merge rewritten logical series before window evaluation Accept disjoint closed-source fragments under one output label set. Retain occupancy checks and merge affected window inputs once at EOF using shared typed sample segments. Keep direct input windows and root maps streaming, and release collected fragments on finish or error. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * docs(promql): clarify approximate merge workspace estimate Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): validate series boundaries before collected input merges Keep each collecting window's open metric and last position so EOF grouping cannot hide reversed runs or identity changes before closing. Preserve source and root-map piece semantics, and test malformed input release, empty closing chunks, and EOF truncation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): supply the default binary fill in the unimplemented-node fixture Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): end a subtree before the node above it computes A window that collects its input answers only once its own input is finished, so ending the input now flushes the deepest ready window and lets the following pull carry its answer up; a binary node computes only when both operands are settled. Previously a binary node computed on an empty operand and marked itself computed, dropping what the window produced afterwards. Also stream the answer instead of collecting it: end_of_input() ends the input, and each pull runs the next stage, so the caller pulls the result and the last None closes the execution. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(promql): reserve an instant selection for the points it decides The gather reserved the whole planned grid, while select_instants clips the range to the series' first and last sample and its lookback: a week-long query at a one-minute step reserved ~504 KB for a series that produces five points, and a small pool refused it before gathering. Reserve for the clipped candidate count, derived with the same alignment the kernel uses. Also pin the start-timestamp semantics the histogram change spread to: last_over_time and first_over_time hand back the sample they selected with its start timestamp, and limitk and limit_ratio keep it on the sample they keep, so a reader downstream can still place a counter reset. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * refactor(promql): reserve binary memory per phase, not per value The binary kernel adjusted the pool and the metrics once per matched point, per filed row and per merged row, so a million plain float points meant roughly three million reserve calls inside a numeric loop. Reserve at three boundaries instead: the operands as chunks are retained, one figure per phase (pairing, sort, merge) with an estimate over rows/fields/segments/payload, and each result buffer when a block of it is allocated. Delete the code that modelled another component's allocator: index_column_growth and its shadow index_capacity, BinaryOperand::ends_growth, and the flatten-capacity model in assembly_bytes. Nothing pre-reserves the left-times-right product and no large allocation happens before its reservation. Tests that pinned the callback sequence or exact small-object arithmetic go; the resource properties stay: a genuinely large input or output is refused, a sparse small result is admitted, and reservations are released on finish, error, truncation and drop. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(promql): move the histogram_quantile doc back to its own test The start-timestamp test was inserted into the middle of another test's doc comment, leaving the earlier test without one. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * feat(promql): compile a constant, a negation and a scalar-vector operation Three shapes reach the tree now: a constant binds as a `Scalar` on the statement's grid, a negation binds as the multiplication by `-1` it is, and a binary operation between one vector and one constant binds as a `Binary` node whose output is the vector side's own binding. The arithmetic and comparison operators all map, and `return_bool` is carried as its own field so `x > 0` (which filters) and `x > bool 0` (which reports zero or one) are one node with one field changed. The operands keep the order they were written in, because most operators are not commutative: `1 / x` is not `x / 1`, and which side the constant is on is the only thing that tells them apart. A constant is bound by the same `constant_value` fold a call's parameters already use, so `x * (1 + 2)` carries `3` rather than an addition the tree would have to know how to run. Three shapes are refused rather than approximated. An operation between two vectors needs each `Input` to name the selector it reads, which the one binding does not carry, so `a + b` and `a + a` would compile to the same tree; a set operator combines two vectors' series and has no node at all; and a statement whose value is a scalar has nothing to scan, so a constant is only bound next to a vector that gives it something to broadcast over. Each refusal names what it found. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> (cherry picked from commit |
||
|
Dennis Zhuang
|
47c40eeb01 | chore: merge narthex into feat/narthex-vector-agg-compile | ||
|
dennis zhuang
|
72077149e9 |
feat(promql): compile sort and absent into the native vector node (#9852)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, Select category (#25). Builds on #9851 and #9850 (merged). ## What's changed and what's your intention? `sort`, `sort_desc`, `sort_by_label`, `sort_by_label_desc` and `absent` calls now compile into the vector node. - `select_op` binds the order (label arguments are string literals) or the `absent` labels; `select_call` wraps the compiled vector in `NativeExpr::Select`, keeping its type as a map does. - `absent` labels follow Prometheus `createLabelsForAbsentFunction` through `select::absent_labels`: built from a vector selector's matchers (`matches_empty` from the anchored `Matcher::is_match("")`), and empty for any other argument. `__field__`, `__schema__` and `__database__` choose what is read and are not labels, as in the mainline planner. Selectors with `or` alternatives are refused. - Unlike the mainline planner, `absent(rate(x{job="a"}[5m]))` answers without labels, as Prometheus does. The mainline reads the matchers of the last selector it planned. Known difference to settle when the compiler gets a caller: over a table with several fields and no `__field__`, `absent` answers per field, while the mainline answers from one field. ### Tests - Compiler: each sort shape and `absent` bind their selection. `absent` cases cover an equality kept, a matcher that accepts a missing label ordered after the equality and removing it, `__name__` and `__field__` skipped, and a non-selector argument. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
3476f76219 | chore: merge narthex into feat/narthex-select-compile | ||
|
Dennis Zhuang
|
b67aa655ac |
chore: merge narthex into feat/narthex-label-node
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> # Conflicts: # src/promql/src/native/execute/tests.rs # src/query/src/promql/native_compile.rs # src/query/src/promql/native_compile/tests.rs |
||
|
Dennis Zhuang
|
e46f1f63e0 |
fix(promql): keep fill_left and fill_right on their written sides under group_right
Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
Dennis Zhuang
|
2f718528d5 | chore: merge narthex into feat/narthex-count-values-node | ||
|
dennis zhuang
|
6c7e534b92 |
feat(promql): run histogram_quantile and histogram_fraction in the native vector node (#9853)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, histogram category (#25). Stacked on #9852 → #9851 → #9850; its own change is the last two commits. Draft until those merge. ## What's changed and what's your intention? `histogram_quantile(q, v)` and `histogram_fraction(lower, upper, v)` compile and run natively, with constant parameters folded by the compiler. - **Node.** A classic histogram is read from the bucket series that share its labels, so these need the whole vector at a point, like the select node's other operators. They run there as `BoundSelect::Histogram(HistogramFunction)` through `evaluate_histograms`. The node now covers selections, sorts, `absent` and these; its name predates that and could become something like `Vector` in a follow-up. - **Pieces.** The kernel groups by label set and takes one native histogram series per set. Pieces of one label set (an empty tag and an absent one, or what a map folded) are therefore merged with `merge_same_labelset` first, and two pieces holding one point fail as they do elsewhere. Without the merge a second native piece was silently dropped and overlapping classic buckets were summed. The output holds one series per label set, so the node does not mark it as holding pieces. - **Deterministic order.** `evaluate_histograms` iterated its groups through a `HashMap`, so the result order varied between runs. Groups now keep first-arrival order, as `merge_same_labelset` does. - **Memory.** The estimate gets its own arm: the merge workspace, per-series group entries and cursors, per-row buckets and results. ### Tests - Runtime: classic buckets of one label set give the quantile without `le` and the name; two native pieces of one label set are answered at both points (fails without the merge). - Compiler: both calls bind with folded constants (`1 - 0.1` → `0.9`). ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
Dennis Zhuang
|
8ba8a31762 | chore: merge narthex into feat/narthex-select-compile | ||
|
dennis zhuang
|
012828cf65 |
feat(promql): run sort and absent in the native vector node (#9851)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Native tree operator wiring, Select category (#25). Builds on #9850 (the `Vector` node, merged). Also carries the review fix for shared histogram tables (queue-level charge) and the `LimitK(SortDesc)` regression requested on #9850. ## What's changed and what's your intention? `sort`, `sort_desc`, `sort_by_label`, `sort_by_label_desc` and `absent` run in the vector node. - **Plan.** `BoundSelect::Sort(SortOrder)` and `BoundSelect::Absent { labels }`. The labels are what the compiler will derive with `absent_labels` from the argument's matchers. - **Runtime.** - A sort hands back its input series reordered, and the queue bills them the way the input was retained (slices, each histogram table once). Billing by the allocations would count a shared buffer once per input chunk. - `absent` answers with one series, so its output never holds pieces of a label set. - The workspace estimate is now per operator: a sort keys its series and rebuilds only what `sort` / `sort_desc` drop histograms from; `absent` keeps a flag and a value per point and field. - Value sorts take one field, like `topk`. - **Kernel fix.** Prometheus' `sort` / `sort_desc` drop histogram samples at every step (`filterFloats`), in range queries too. `sort_instant` used to fail on them at an instant and keep them in a range. It now drops them in both. `sort_by_label*` keep every sample. Compiler wiring for these function calls comes next and does not depend on #9833. ### Tests - Kernel: `sort` drops histogram samples at an instant and per step in a range; `sort_by_label` keeps them. - Runtime: `absent` answers where the input holds nothing, and at every point with no input; `sort_desc` hands series over in value order; a sorted series delivered as ten slices of one buffer fits a pool sized for the input (fails with the old billing). - Display: sort and absent lines. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |