A doc comment on the categorizer was followed by a blank line, and the test
table builder takes eight arguments. Both are what CI's `-D warnings` turns
into errors; neither changes behaviour.
Signed-off-by: discord9 <discord9@163.com>
A plan holding a node with no Substrait serializer was always cut at that node,
because the analyzer asks whether a plan encodes before it asks whether a node
commutes with the region split. That is right for a region reached over the wire:
the plan is encoded into the request and has to decode on the other side.
Standalone does not use that path. Its region server is the one built in the same
process, its query handler passes the request straight to `RegionServer::handle_read`,
and the plan stays a `LogicalPlan` from the frontend to the region. Encoding it is
work nobody reads, and refusing to place a node because it cannot be encoded
refuses a plan that would have travelled intact.
So the engine now knows whether its regions are reached in process, and the
encodability gate applies only when they are not. The capability is set where it
can be stated truthfully - the standalone bootstrap, which creates both the
frontend and the region server it reads - and it defaults to false everywhere
else, so an ordinary build and every other deployment keeps today's behaviour
byte for byte.
Skipping the gate skips one early return and adds no reason to expand:
`Categorizer::check_plan` still runs and still decides, and a plan that does not
encode for any other reason still expands at the node that caused it. Two tests
hold those two halves apart.
Signed-off-by: discord9 <discord9@163.com>
The arm above answers that a fused node commutes with the region split, and it
listed "no series is split across regions" among its reasons. That is not
established anywhere. A region evaluates the whole tree over the series its own
scan returns, so a series whose samples reach two regions is evaluated per
fragment, and a range function over a fragment is not the range function over the
series.
What the placement does have is three proofs about the node - its input is the
scan, it binds a metric name, and no node of its tree folds across series. What it
does not have is any check that the series stays in one region: the storage
partition key is whatever `partition_column_names()` names and may be a column
that varies within one series, and this categorizer runs before region selection,
so there is no selected-region vector here to consult either.
So the comment now says that plainly, names the deployment the placement is
correct under, and says what would make it unconditional. A single-region
deployment satisfies it; a partitioned table over several regions would answer
wrongly rather than fail.
Signed-off-by: discord9 <discord9@163.com>
A `FusedSeries` node is built for one metric's series, and a region holds a subset
of those series with none split across regions, so the node commutes with the
region split when what it reads is per-series. It says so now, and the analyzer
may place it inside the merge scan.
Three conditions, each drawn from the conversion that has to succeed there:
- The node's input is exactly the table scan. The region builds the node from the
scan's own raw-series entrypoint and cannot look through a filter, a projection
or a sort in between, so a node over anything else has nothing to read where it
would be placed.
- The node binds a metric name, because the region resolves the series of one
named metric.
- No node of its tree folds across series. A region holds part of a metric's
series, so an aggregate or a binary operation evaluated there would consume a
partial input and return a wrong answer without saying so. Every operation the
compiler emits today - input, instant, window, map - reads one series at a
time, so this refuses nothing buildable; it is the conservative answer for a
tree built some other way.
A scan below such a node also asks for the per-series distribution now. That is
what makes a region prepare its series scan, and only that path packs a series'
chunks into the frame the node reads.
Both are inert without a `FusedSeries` node, so an ordinary query keeps its plan
and its scan request.
Not yet reached: the analyzer asks whether a plan encodes before it asks whether
a node commutes, and this node has no Substrait serializer, so the plan is
expanded at the node and the region still receives the bare scan. A test pins
that boundary so it fails the day the node can cross it.
Signed-off-by: discord9 <discord9@163.com>
Nothing called `native_compile::compile` or built a `FusedSeries`, so the native
tree and its node were library code with tests and no query behind them. This
adds the route, behind a per-query flag.
The statement is offered to the compiler at the root, where the expression the
reader wrote still exists. A shape the compiler refuses - `NotImplemented` - is
this route declining the query, and planning continues exactly as it would with
the flag off. Anything else is a real error.
The physical half is an arm in `PromExtensionPlanner`. Its children are already
built by then, so a metric scan arrives as the region scan, and the node asks that
scan for its named raw-series binding: `try_to_named_series_scan` resolves the
label source and ordering from the scanner itself, which is the only thing that
can name them, and returns the leaf the node reads. A scanner that cannot serve a
raw frame is reported, not silently re-planned.
Two things this needed that the route does not get for free:
- The region scan has to be prepared as one partition. `resolve_named_series_binding`
refuses a scan whose partition count is not one, and a fresh scan has not set
its partitions at all: `ParallelizeScan` runs after extension planning and would
spread it, so the node would run one evaluator per partition and split a logical
series. Preparing the ranges as one partition is what keeps `FusedSeriesExec`
over one series stream, and it is why this route does not scan in parallel yet.
- The scan needs the per-series distribution hint. That is what makes a region
prepare its series scan, and only that path packs a series' chunks into the frame
this node reads. The existing planner asks for it through a sort that
`ScanHintRule` turns into the hint; this node has no such sort below it, so the
hint is attached to the scan directly.
A query the route cannot bind faithfully keeps the existing plan rather than a
wrong one: more than one selector, a label matcher that would survive
preprocessing, an `__field__` matcher, a native histogram field, a missing table,
or a shape whose grid the compiler's statement cannot express.
Signed-off-by: discord9 <discord9@163.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Native tree operator wiring, label category (#25). Builds on #9846 (date
wiring, merged).
## What's changed and what's your intention?
`label_replace` and `label_join` now compile into the native tree and
run in the existing `Map` node.
- **Plan.** `BoundMap.ops` becomes `Arc<[MapOp]>`, with
`MapOp::Math(MathOp)` and `MapOp::Label(Arc<LabelRewrite>)`. A label
change is pointwise per series like math, so it needs no new node,
collector or output type.
- **Kernel.** `LabelRewrite` (in `map_label.rs`) is the checked,
compiled change: `replace` anchors the regex and checks the destination,
`join` checks source and destination names, `apply` rewrites one
`MetricName`. The whole-vector `label_replace` / `label_join` now go
through it and still merge same-labelset series afterwards, so the
Prometheus golden keeps covering the same code. Fields are private, so
only the two constructors build one.
- **Runtime.** `map_chunk` applies label ops to the series metric and
math ops through `map_math`. When a math op empties a closing chunk, the
labels of the closing series come from applying the remaining ops in
order (label ops rewrite, math ops drop the name). Series a label change
folds into one label set go through the `Emitted` ledger the map already
keeps: disjoint values merge, a shared point is refused.
- **Compiler.** `label_rewrite` reads the string-literal arguments.
Arguments Prometheus rejects at evaluation (invalid regex, empty label
name) fail the statement with `Plan`, as Prometheus fails the query;
everything unsupported stays `NotImplemented`.
- **Display.** `label_replace "dst" "repl" "src" "regex"` / `label_join
"dst" "sep" ["a", "b"]`.
Shared files touched: `native/plan.rs` (`MapOp`), `native/execute.rs`
(`map_chunk`, `no_row_series` now takes the metric),
`native/display.rs`, `native_compile.rs`. Test constructors move from
`MathOp` to `MapOp::Math`.
Known difference, not introduced here but now reachable: Prometheus
checks same-labelset duplicates separately for floats and histograms,
while the native ledger counts both as a value at a position. A
`label_replace` that folds a float series and a histogram series onto
one label set at the same step is refused natively.
### Tests
- Runtime: a relabel that folds two series refuses the shared point; a
closing chunk emptied mid-chain carries labels from label ops on both
sides of the emptying op; a label-only map hands histogram samples and
holes through unchanged.
- Compiler: both functions bind their change into a map; an invalid
regex fails the statement with `Plan`.
- `map_label` Prometheus golden (existing) now runs through
`LabelRewrite`.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
prometheus/prometheus#19939
## What's changed and what's your intention?
The native binary kernel applied `fill_left` / `fill_right` after
swapping the operands of `group_right`, so under `group_right`
`fill_left` filled a missing right-hand sample and `fill_right` a
missing left-hand one. That is the Prometheus behavior before
prometheus/prometheus#19939, which was merged into Prometheus main on
2026-10-13: the fill values now swap together with the operands, so they
always refer to the side written in the query.
For example, `node_meta - on(instance) group_right fill_left(5)
fill_right(7) cpu_info`: a missing `node_meta` must give `5 - cpu_info`
and a missing `cpu_info` must give `node_meta - 7`.
- `binary_field` swaps `fill.lhs` / `fill.rhs` when the one side is the
left, matching `VectorBinop` after #19939. The `FillValues` doc now
states the as-written meaning.
- `fill_cases.txt` is regenerated by promql-compat
`narthex-golden-vector -experimental fill` built against Prometheus at
the #19939 merge commit (same seed and case count). Only the 24
`group_right` cases whose fill fires on an asymmetric pair change.
No released Prometheus contains #19939 yet (v3.15.1 still has the old
behavior), so until the next release native results differ from released
Prometheus for asymmetric fills under `group_right`. The compiler does
not bind vector-vector fill modifiers yet, so this is not reachable from
a query today; it has to land before that wiring does.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Native tree operator wiring, Select / aggregate categories (#25).
Follows #9850 → #9851 → #9852 → #9853 → #9854 (all merged) and #9833.
## What's changed and what's your intention?
`topk`, `bottomk`, `limitk`, `limit_ratio` and `count_values` now
compile. #9833 left them out of `AggregateKind` because they do not fold
a group into one value per point. They now bind as a `Vector` node
through `vector_aggregate`:
- `topk` / `bottomk` / `limitk`: `k` is folded to a constant as
`quantile`'s parameter is, and the `by` / `without` grouping is kept. A
`k` only known at run time is refused with `NotImplemented`. NaN,
negative and out-of-range `k` are left to the kernel, which fails or
answers at evaluation as Prometheus does.
- `limit_ratio`: the ratio is folded the same way. The grouping is
dropped, since Prometheus keeps or drops each series by its label hash
alone.
- `count_values`: the label is a string literal. Label validation stays
in the kernel, an execution error as in Prometheus.
The module doc, the grammar and the `SUPPORTED` text name the new
shapes.
### Tests
- Compiler: each aggregator binds its operation with the folded
parameter and grouping. A run-time `k` is refused, and the rejection
quotes every aggregator this compiler takes.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Unblocks the CI checks of #9868 and the rest of the `narthex` stack.
## What's changed and what's your intention?
`narthex` is a long-lived feature branch that has fallen behind `main`,
so its
CI still runs the older workflows. Three checks fail for reasons that
have
nothing to do with the branch's code:
- **Prepare Fuzz Targets** — the setup step installs `libfuzzer-14-dev`,
which
is no longer available on the current runner image.
- **Run Multi-language SDK Tests** — the job asks for Python 3.8, which
the
current runners no longer provide.
- **Clippy** — the failure is inside `cargo clippy`, not in the setup
steps, so
it is not addressed here; see the note at the end.
This PR cherry-picks the CI commits that landed on `main` but not on
`narthex`,
so the branch's workflows match `main` again. It changes **no Rust
code** — all
33 changed files are workflow definitions, composite actions, `Makefile`
targets, `.github/scripts/`, and one how-to doc.
The commits are applied with `-x` so each carries the upstream SHA, and
they
apply without conflicts.
### What is picked
| Commit | Why |
| --- | --- |
| `97e3bfd547b` | manual S3 metric snapshot benchmark |
| `f1fb5e4e7a5` | `RUST_MIN_STACK=8MB` for the merge_group coverage job
|
| `813b132221f` | compare release validation against the previous stable
version |
| `b2b606115b7` | crash/restart/multi-region/drop cells for the S3 WAL
benchmark |
| `db61aecadbc` | do not run these actions on a fork |
| `a0a7ec02b5d` | compare several images in one object store WAL S3 run
|
| `fdb8ceeb542` | increase release integration test stack size |
| `f855355ed04` | independent Linux release architecture selection |
| `630cca33956` | align nightly Windows and macOS artifact test stack
sizes |
| `99d907cf1d2` | **fixes the two failing jobs above**: drops
`libfuzzer-14-dev`, moves the SDK tests to Python 3.13, installs protoc
via `taiki-e/install-action` |
| `6e40daed98e` | move `monitoring.datanodeStorage` out of the
standalone `base` |
| `de0fea673eb` | specify the Kind storage class for datanodes |
### Limitations
The Clippy failure is **not** fixed by this PR. Its setup steps
(checkout,
protoc, toolchain, cache) all succeeded and the failure is in the `cargo
clippy`
step itself, so it needs its own diagnosis — a lint in the branch's code
or a
toolchain difference. I could not retrieve that job's log to say which.
`kafka-wal-helper.yaml` is a pre-existing multi-document YAML file; it
is not
touched here and not affected by these picks.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
The `datanodeStorage` field belongs directly under `monitoring.standalone`
(GreptimeDBStandaloneSpec), not under `monitoring.standalone.base`
(PodTemplateSpec). Nesting it under `base` renders an invalid
GreptimeDBCluster resource and the CRD schema rejects it during
server-side apply:
.spec.monitoring.standalone.base.datanodeStorage: field not declared in schema
(cherry picked from commit 3483f4a284)
* ci: install protoc with taiki-e/install-action
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: run multi-language SDK tests on Python 3.13
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: drop the unused libfuzzer-14-dev package from fuzz builds
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
* ci: build fuzz targets with the bundled libFuzzer
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
---------
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
(cherry picked from commit 00ab09012d)
* ci: compare several images in one object store WAL S3 benchmark run
A new `images` input takes a JSON list of 2..4 {label, image}. Every
cell then runs once per image, back to back, and the image order
reverses with the configuration order every other round. The cell id
gains the label. The single-image inputs keep their meaning and cell ids.
The validate job pulls every image before an instance is allocated. It
pins each image to its digest and records the version and commit that
the image reports. Each cell starts the image of its label. An optional
`warmup` runs the first load cell once per image as round 0, and no
statistic uses it.
The plan step estimates the benchmark job's minutes on c6id.2xlarge
from the cell times of earlier runs. It refuses a plan that does not
fit the job timeout.
The summary keeps the Raft Engine ratios within each image. With
several images it also reports, per group, concurrency and config:
- every round's rows/s and client p50/p99;
- each image's ratio to the first, with the geometric mean, minimum
and maximum over the rounds.
Every cell records the host load averages at its start and end.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: scale the S3 benchmark time estimate with the batch size
The per-cell rates were measured with 1000-row batches. A smaller batch
keeps about the request rate, not the row rate, so the estimate now
shrinks rows/s with the batch. A one-row durable plan is refused instead
of passing a timeout it cannot fit.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin the S3 benchmark runtime image without pulling it in validate
An anonymous pull from ECR Public on a hosted runner can exceed the
registry's data limit, and it failed the validate job of several
dispatches with "toomanyrequests: Data limit exceeded". Validate now
reads only the registry manifest to pin the runtime image's digest. A
reference that names a digest must be served under that digest.
The check that the image holds the loader and the generator moves to
the benchmark host. It runs right after the runtime pull and before any
GreptimeDB image is pulled or any cell starts.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test: run the S3 benchmark runtime content check against both files
The fake docker now runs the pull step's own check in a directory that stands in for the runtime image. It covers both files present, the loader missing and the generator missing, so dropping either file check fails the test.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: pin a digest-named S3 benchmark runtime image without the registry
ECR Public's anonymous data limit now refuses hosted runners even the
manifest read, and validate failed in "Pin the runtime image digest".
A runtime reference that names its sha256 digest, like the default one,
is already pinned, so validate takes that digest with no network call.
The format checks stay. Only a tag-only reference is resolved through
the registry. The content check on the benchmark host is unchanged.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* ci: check the full S3 benchmark runtime reference before pinning it offline
A named digest no longer reaches the registry, so nothing parsed the rest of the reference. Validate now checks the whole reference against Docker's reference grammar first. It refuses an empty repository, a doubled '@' or a bad tag before any pin is emitted or any instance is allocated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
(cherry picked from commit a436213bf3)