* perf(promql): append ordinary raw chunks to the evaluator in bulk
`ingest` copied every input sample into `FieldSamples` one at a time. A
`RawSeriesChunk` hands the evaluator Arrow buffers `Samples::from_float`
wrapped without copying, and `ingest` read them back through `Samples::iter`
only to perform, per sample, an enum match, a stale-marker comparison, an
overflow-checked offset addition and two `Vec::push` calls.
A chunk of null-free float columns read with no offset now appends with
`extend_from_slice`. Every other shape keeps the per-sample loop: a shifted
timestamp reads different timestamps, a null has no sample, a field mixing
native histograms reports `DenseSamplesRef::Mixed` and must keep its windows
on the histogram path where the mixed-window rules decide the result, and a
chunk whose values contain a stale marker drops its row and needs the per-row
decision.
The reset index stays per-sample inside the bulk append, so the recorded
resets are the same ones the per-sample path records. Tests assert both that
the values still match the brute-force reference and that the bulk path is the
one that ran, since both paths produce identical output by construction; a
mutation that lets a mixed field through the admission check fails them.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
* refactor(promql): keep the ingest docs with ingest and drop untested guards
Three review findings on the bulk-ingest path, none changing behaviour.
The doc comment describing `ingest` sat on top of `ingest_in_bulk` after the
bulk path was inserted above it, so `ingest` had lost its docs and the bulk
function carried documentation for work it does not do. The comment moves back
to `ingest`.
`ingest_in_bulk` returned `Result<bool>` but never returned `Err`: the two
length checks inside it are already done by `check_chunk` for raw input and by
`push_series` for evaluated input. It now returns `bool`, and the length checks
go with the `Result`.
The stale-marker and offset "is not appended in bulk" tests guarded faults the
behaviour tests already catch -- the differential and reference comparisons
fail if either condition is admitted. The histogram one stays: it is the only
test covering that admission rule.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Co-authored-by: discord9 <discord9@users.noreply.github.com>
* perf(promql): index counter resets per field instead of rescanning the window
The fused evaluator rescanned every window for counter resets, once per grid
point. Record the resets of each field as the samples are ingested instead,
keyed by an absolute sequence number, and read the ones a window covers with a
binary search. Only `rate` and `increase` correct counters, so only they
maintain the index.
The binary search runs over the field's retained reset index rather than over
the window's samples: its input is the remaining slice of resets, so its cost
follows the number of retained resets, not the window length.
A drop across an intervening native histogram sample is recorded as a reset of
two samples that are not neighbours. That is unobservable: a window holding
both floats also holds the histogram between them, so it is a mixed window,
whose counter functions report nothing; and a window starting after the earlier
float excludes a reset on its first sample.
Verified with `cargo nextest run -p promql` (453 passed) and `cargo clippy -p
promql --all-targets --all-features -D warnings`. Tests assert both that the
indexed results match the brute-force scan and that the index is what ran.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* refactor(promql): drop per-field clearing the evaluator never needs
Two review findings, both about code that does nothing.
`finish_series` cleared every field after `evaluate(None)` returned. That loop
is redundant: `evaluate` ends with either every point evaluated -- so `head`
moved to the end and `advance` dropped the samples -- or no field holding a
sample, and either way the fields are empty. It also did not clear
`histograms`, which made it read as though it should. Removing it keeps the
suite green, and `push` already documents that an evaluator is not reused after
an error. `FieldSamples::clear` existed only for that loop.
`overlapping_window_spans_chunks` used a `sum` kernel and never touched resets.
Windows spanning chunks are already covered by
`evaluator_matches_brute_force_reference`, which chunks randomly, and no
mutation was caught only by this test. Its `sum` helper went with it.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@users.noreply.github.com>
* perf(promql): widen the shared extrapolation arithmetic to 64 bits
`extrapolation_factor` is the one place both the float and the native-histogram
range paths compute Prometheus' extrapolation. Its three millisecond
differences were widened to `i128` and narrowed back with `as f64`; each of
those narrowing conversions compiles to a call to the `__floattidf` builtin
rather than an instruction, and the function runs once per grid point per
series.
Use saturating `i64` subtraction instead, which is exact here: a difference too
large for `i64` is either a window start below every representable instant, or a
distance far past the extrapolation threshold where both values take the same
branch. Both callers now share the same arithmetic; neither path carries its own
copy.
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
* perf(promql): wrap the extrapolation arithmetic like Go instead of saturating it
`extrapolation_factor` widened to `i128`, subtracted, and narrowed back with
`as f64`; that compiles to a call to the `__floattidf` builtin once per grid
point per series. Doing the subtraction in `i64` removes the calls, but the
first form of it clamped with `saturating_sub`, and clamping is not what Go
computes: with `eval_ts = i64::MIN + 10`, `range_ms = 100`, `first_t =
i64::MIN + 1` and `last_t = i64::MIN + 9`, Prometheus returns 1.625 and the
clamped subtraction returns 1.25.
`wrapping_sub` is Go's `int64` arithmetic bit for bit, at the same cost. Real
timestamps are nowhere near that range, so the two agree on every input a query
can produce; the difference only shows up where the old code silently disagreed
with Prometheus.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
---------
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
Signed-off-by: discord9 <discord9@users.noreply.github.com>
* feat(promql): add evaluated series math kernels
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(promql): add evaluated series scalar arithmetic
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(promql): complete evaluated series math functions
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(promql): add series comparisons and scalar results
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(promql): add sparse evaluated series basic aggregates
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(promql): align time-varying scalar maps on evaluated grids
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): drop metric name in evaluated series math like Prometheus
Prometheus' simpleFloatFunc drops __name__ together with __type__ and
__unit__. Check abs, ceil, floor, sqrt and round against fixtures
generated by the Prometheus v3.15.0 engine.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): drop empty labels with the metric name in series math
Prometheus has no empty-valued labels, so a kept empty value would hide
a same-labelset collision from the caller. Round shares its calculation
with the UDF, so check it only against the Prometheus fixtures.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): align evaluated series map kernels with Prometheus
Follow Prometheus v3.15.0 simpleFloatFunc, clamp, VectorscalarBinop and
vectorElemBinop: drop __name__, __type__ and __unit__ for math,
arithmetic and bool comparisons; use ports of Go math (including the
amd64 Log and Exp assembly) for ^, %, atan2 and the transcendental
functions; empty a clamp with max < min; keep -0 in sgn and Prometheus'
operation order in deg and rad. Fields are compared independently and
filtered output is trimmed with GridSeriesBuilder.
Check the kernels against fixtures generated by the Prometheus engine
and Go math results built for amd64.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): align evaluated series aggregation with Prometheus
Follow Prometheus aggregation and generateGroupingLabels: no modifier is
by (), without drops __name__, empty label values are absent, sum and
avg use Neumaier compensation and avg switches to an incremental mean
on overflow. Fields are aggregated independently, so a field without
values at a row is null also for count and group. Output is trimmed
with GridSeriesBuilder.
Check against fixtures generated by the Prometheus engine and drop the
main UDAF oracle and the common-function dependency.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): validate whole segments in series comparisons
map_compare checked sample types one by one, so a histogram or mixed
segment without a present sample passed. Validate segments up front as
the other kernels do, check them before an empty clamp, share the
empty-label rule with aggregation, and pin exact bits in the deg, rad
and incremental avg tests.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: exclude promql golden fixtures from typos
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(promql): point series round at Prometheus funcRound
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): keep Go math ports only where std leaves the tolerance
Series math now follows the relative-1e-10 comparison rule: functions use
Rust's standard library unless Go's amd64 result differs beyond the
tolerance or in overflow and sign. asin, atan, tanh, asinh, acosh and
atanh use std; the Exp assembly port is reduced to its overflow edge,
and sinh/cosh build on it above 21.
Golden and Go math fixtures are compared with almost.Equal. The Go cases
are evaluated through the kernels' own per-sample functions, and each
retained port must have a case where std fails, so a port that is no
longer needed, or a swap back to std, fails a test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* refactor(promql): validate aggregate input like series maps
aggregate_float repeated map's schema, grid-axis and float-segment
checks in a different order, so one bad series could fail with a
different error category in each. Both now use validate_float_series,
and map_math/map_negate/map_arithmetic run it before mapping instead of
their own axis check.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(promql): pin avg compensation to Prometheus bits
With fixtures compared within the tolerance, dropping or misplacing the
Kahan compensation in avg only moves the last bits. Three Prometheus
cases where it does are compared exactly: compensation on the
incremental mean, the separate division of sum and compensation, and
rescaling the compensation when avg switches to the incremental mean.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(promql): pin the zero sign of clamp and the Neumaier swap
Under the fixture tolerance, +0 and -0 compare equal and no case had an
addend larger than the running sum, so two branches lost their tests:
Go's Max picking +0 over -0, whose sign a later 1/x turns into the sign
of an infinity, and the Neumaier swap, without which
sum(1, 1e100, 1, -1e100) is 1 instead of 2.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: allow the Unicode category Nd in typos
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(promql): keep aggregate state in fixed-size grid blocks
Aggregation kept one B-tree entry and one Vec allocation per populated
evaluation point, plus a tree lookup per sample.
State is now stored in 64-point blocks keyed by absolute grid index /
64, holding one state per field and point. A block is allocated only
once a sample falls in it, so sparse input never expands to the full
grid, and each field's run through a block costs one lookup. Points
without samples are skipped on output, which also keeps the last block
from reaching past the grid.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(promql): drop reserved and empty labels in one rebuild
drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(promql): pin that aggregation keeps type and unit labels
Prometheus generateGroupingLabels drops only the metric name, unlike math and arithmetic. Nothing failed when grouping dropped __type__ or __unit__.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): return aggregate groups in first-seen order
Groups were collected in a BTreeMap and returned sorted by label. Prometheus' rangeEvalAgg appends groups in the order their first input appears, and a nested aggregation folds them in that order: for sum(sum by (g) (m)) over 1e308, -1e308, 1e308 with g = a, c, b, Prometheus gives 1e308 and label order overflows to +Inf. Look groups up by label key and keep them in a Vec.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* test(promql): pin which zero min and max keep
Prometheus replaces the min/max accumulator only on a strict comparison, so of +0 and -0 the first is kept. The golden tolerance treats the zeros as equal, and a non-strict comparison passed every test.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* feat(promql): add evaluated series math kernels
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): drop metric name in evaluated series math like Prometheus
Prometheus' simpleFloatFunc drops __name__ together with __type__ and
__unit__. Check abs, ceil, floor, sqrt and round against fixtures
generated by the Prometheus v3.15.0 engine.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* fix(promql): drop empty labels with the metric name in series math
Prometheus has no empty-valued labels, so a kept empty value would hide
a same-labelset collision from the caller. Round shares its calculation
with the UDF, so check it only against the Prometheus fixtures.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: exclude promql golden fixtures from typos
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* docs(promql): point series round at Prometheus funcRound
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* chore: allow the Unicode category Nd in typos
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
* perf(promql): drop reserved and empty labels in one rebuild
drop_metric_name removed empty-valued labels one at a time, and each MetricName::remove copies the whole label array, so the cost grew with the square of the number of empty labels. Filter the labels once and rebuild the name with MetricName::try_new.
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
---------
Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
**Stacked on #9657** (raw request types). Also requires **#9651**
(scanner preflight), **#9655** (frame ordering), **#9656** (recordbatch
adapter).
## What's changed and what's your intention?
Add scan-bound typed raw series production to `TableScanExec`:
- When the scan binding has a `RawSeriesRequest` and preflight passes,
produces `SeriesScanExec` that streams typed `RawSeries` frames.
- Shares MVCC/projection/filter semantics with the existing RecordBatch
path.
- Strictly opt-in: absent or declined preflight returns the ordinary
plan unchanged.
**2 files, +475/-2 production + 1506 test lines.**
Note: diff includes cherry-picked commits from #9651 and #9656. Once
they merge, the diff will shrink.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
**Stacked on #9654** (source materializer). Also requires **#9649**
(instant eval) and **#9655** (frame ordering).
## What's changed and what's your intention?
Add native instant manipulate execution over raw series streams:
- `NativeInstantMapping`: preflighted mapping from typed raw series to
query output schema.
- `NativeInstantManipulateExec`: ExecutionPlan that consumes
`SendableRawSeriesStream`, materializes labels, selects instants, and
emits aligned RecordBatches.
- `InstantManipulate::try_to_native_execution_plan()`: validates the raw
frame schema and constructs the native plan.
**5 files, +1830/-0 lines.** No callers wired yet.
Note: diff currently includes cherry-picked commits from #9649 and
#9655. Once those merge into narthex and the base is updated, the diff
will shrink to only the 5 native exec files.
### Verification
- `cargo check --all-targets` ✅, `cargo nextest run -p promql` →
**357/357 passed** ✅, `cargo clippy -- -D warnings` ✅
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
**Stacked on #9662** (native config). Also requires **#9655** (frame
ordering metadata).
## What's changed and what's your intention?
Add native raw-series route capture and MergeScan typed-frame support:
- `CaptureNativeRawRoute`: captures region routing during planning when
`experimental_native_promql` is enabled.
- `NativeRawSource`: borrows the captured route for native raw-series
execution.
- Schema identity validation for `greptime.series.` metadata.
- Wires `experimental_native_promql` from `QueryEngineState` →
`DfQueryPlanner` → `DistExtensionPlanner`.
**4 files, +1746/-113 lines.**
Note: diff includes cherry-picked commits from #9655. Once it merges,
the diff will shrink.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
No dependencies on other open PRs.
## What's changed and what's your intention?
Add raw series request types for the named raw-series preflight
contract:
- `RawSeriesRequest` / `RawSeriesTag`: versioned binding for a
single-region raw-series request.
- `PreparedNamedRawSeries` + encode/decode: JSON body for the
`PREPARE_RAW_SERIES_ACTION` Flight action.
Strict serde (`deny_unknown_fields`), version 1 only. Includes wire
compatibility fixtures.
**7 files, +661 production + wire fixtures.**
### Verification
- `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅,
`cargo clippy -- -D warnings` ✅
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
**Depends on #9650** (labels decoder). Must merge after (or be rebased
onto) #9650.
## What's changed and what's your intention?
Add `promql::source::materialize_raw_labels`: a stream decorator that
converts raw series chunks with primary-key-encoded labels into chunks
with complete explicit labels.
Uses `PrimaryKeyLabelDecoder` (from #9650) to decode dense/sparse
primary keys against the scan-bound region metadata snapshot. Rejects
schemas with eval grids or empty fields, and requires complete (not
projected) label decoding. Output always advertises
`RawLabelSource::Explicit`.
**5 files, +1216/-0 lines (169 production + 1040 tests + 7 config).**
The merge module is intentionally excluded.
### Verification
- Compiled and tested with #9650 cherry-picked: `cargo check
--all-targets` ✅, `cargo nextest run -p promql` ✅, `cargo clippy -- -D
warnings` ✅
- Cannot compile standalone until #9650 merges.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Builds on the shared series types already present on `narthex` (#9598,
#9599). This PR targets `narthex`, not `main`, and has no dependency on
another open PR.
## What's changed and what's your intention?
Extract instant sample selection into `select_instants`, operating on
the existing `RawTimestamps` and `SeriesFields`. It returns source row
indices and evaluation timestamps without converting the payload into a
new representation.
Use it in the existing `InstantManipulate` path when timestamps are
non-null, strictly increasing milliseconds and the configured fields are
Float64. Other inputs retain the existing path, including duplicate
timestamps and Histogram columns. Selection preserves lookback
boundaries, offsets, stale markers, ordinary NaNs, null samples, and the
existing output-point limit.
This is one extracted slice of previously local work: **5 files, +732/-3
lines, including tests**. It does not add native planning,
scanner/Flight integration, configuration, storage formats, or any new
Histogram type.
### Verification
- Passed: scoped Rust formatting, `hawkeye format`, `hawkeye check`,
`git diff --check`, and normal commit hooks.
- Running on this exact extracted branch; results not yet claimed:
`cargo check --locked -p promql --all-targets` and `cargo nextest run
--locked -p promql --retries 0`.
- Independent source review is in progress. Kept as a draft until
verification and review are complete.
- Full-workspace tests, full-workspace Clippy, and SQLness have not been
run for this extracted branch.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Independent PR. No dependencies.
## What's changed and what's your intention?
Recognize `SeriesScanExec` in query metrics and preserve typed errors
through the record batch adapter:
- `PlanMetrics`: recognizes `SeriesScanExec` nodes so region-scoped
stats attribution works.
- `RecordBatchStreamAdapter`: preserves the original typed error via
`BoxedError` instead of wrapping into a generic execution error. This
lets downstream recover the specific status code (e.g.,
`InvalidArguments` instead of `EngineExecuteQuery`).
**2 files, +203/-9 production + 1 sqlness expected result.**
### Error code change
The typed error preservation changes the error code for the
`semantic_graph` upper-bound-only query from `3001(EngineExecuteQuery)`
to `1004(InvalidArguments)`. The error message text is unchanged. The
new code is correct: the original error was an argument validation
error, not a generic execution failure. The sqlness expected result has
been updated.
### Verification
- `cargo check --all-targets` ✅, `cargo nextest run -p
common-recordbatch` → **65/65 passed** ✅
- sqlness expected result updated for the typed error code change
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
No dependencies on other open PRs. Builds on shared series types from
#9598/#9599.
## What's changed and what's your intention?
Add `record_batches_to_raw_series` in `common-query`: converts an
ordinary DataFusion `RecordBatch` stream into contiguous raw series
chunks.
The adapter validates column types at construction (time must be
Timestamp(Millisecond, None), tags must be string/dictionary, fields
must be Float64 or native histogram), encodes series identity via Arrow
RowConverter, and produces `RawSeriesChunk`s with explicit labels and
contiguous ordering. Float64 fields share source buffers (zero-copy);
histograms reuse the existing `histogram_samples_from_arrow` bridge.
Chunking merges equal-key runs within and across batches; a key change
or end-of-input sets `is_last`. Timestamps must be strictly increasing
within a series. Empty batches are skipped with a bounded poll loop.
**3 files, +1666/-0 lines, including 20 tests.** No production callers
yet — consumed by follow-up scan wiring PRs.
### Verification
- `cargo check --locked -p common-query --all-targets` ✅
- `cargo nextest run --locked -p common-query --retries 0` ✅
- `cargo clippy --locked -p common-query --all-targets -- -D warnings` ✅
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
No dependencies on other open PRs.
## What's changed and what's your intention?
Add two default methods to `RegionScanner` for raw-series scan
preflight:
- `raw_series_schema()` — non-consuming check whether this scanner can
provide raw series for a given label source. Returns `None`
(unsupported) by default; overrides promise `scan_series_partition` can
execute.
- `raw_series_ordering()` — returns the ordering guarantee without
consuming scanner state. Defaults to `Contiguous` (conservative for
legacy sources).
Both are additive-only trait defaults with no behavior change for
existing implementors. **32 insertions, 0 deletions, 1 file.**
### Verification
- `cargo check --locked -p store-api --all-targets` ✅
- `cargo nextest run --locked -p store-api --retries 0` ✅
- `cargo clippy --locked -p store-api --all-targets -- -D warnings` ✅
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Independent PR. No dependencies.
## What's changed and what's your intention?
Add the `experimental_native_promql` configuration option (default:
`false`):
- `QueryOptions.experimental_native_promql`: new boolean field, default
`false`.
- Four example TOML configs updated with the commented-out option.
- `config/config.md` regenerated with the new option.
The option is stored in process configuration only. Session-level
override and the actual gate in query planning are in follow-up PRs.
**6 files, +24/-0 lines.**
### Verification
- Compiles standalone: no unused fields or methods.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [x] This PR requires documentation updates. (config.md regenerated)
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
No dependencies on other open PRs.
## What's changed and what's your intention?
Add ordering metadata to raw series frame schemas:
- `raw_series_frame_schema_with_ordering()`: builds the canonical v1
frame schema and stamps the producer's ordering (`contiguous` or
`read_key_then_time`) into Arrow schema metadata. Rejects `Interleaved`
(no v1 wire representation).
- `raw_series_frame_ordering()`: reads the declaration back, defaulting
absent metadata to `Contiguous` (legacy frames), rejecting unknown
values.
- `pack_raw_series` now declares ordering on every yielded batch via
`RecordBatch::with_schema`.
Includes a wire compatibility fixture
(`tests/compatibility/wire/raw_series_ordering/`) with 4 cases.
**4 files, +277/-7 lines.**
### Verification
- `cargo check --all-targets` ✅, `cargo nextest run -p common-query` ✅,
`cargo clippy -- -D warnings` ✅
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Builds on `narthex` base types (#9598, #9599, #9602, #9603). No
dependency on other open PRs.
## What's changed and what's your intention?
Add `PrimaryKeyLabelDecoder` to `mito-codec`: given a scan-bound
`RegionMetadata` snapshot and a primary key byte sequence, decode the
complete metric name and label set (or a projected subset).
Supports both dense and sparse primary key encodings. Sparse keys get a
new `visit_label_ids` method that validates the full tail (rejecting
duplicates, reserved IDs, and truncated framing) before reading values.
Dense keys check that all fields are present and decode each value with
exact type fidelity.
This is one extracted slice of previously local work: **4 files,
+1046/-13 lines, including tests**. It has no production call site yet —
it will be consumed by the raw series scan wiring in follow-up PRs.
### Verification
- `cargo check --locked -p mito-codec --all-targets` ✅
- `cargo nextest run --locked -p mito-codec --retries 0` → **62/62
passed** ✅
- `cargo clippy --locked -p mito-codec --all-targets -- -D warnings` ✅
- `hawkeye format`, `hawkeye check`, `git diff --check` ✅
- Full-workspace tests and SQLness not run for this extracted branch.
## PR Checklist
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
Targets `narthex`. Stacked on #9673 for `GridSeriesBuilder` and the test
reference; review only the two `vector_match` commits.
## What's changed and what's your intention?
### Problem
The native path has no vector–vector operations: arithmetic,
comparisons, `group_left`/`group_right` and `and`/`or`/`unless` between
two sets of evaluated series.
### Change
`vector_match::binary_vectors` and `vector_match::set_vectors` take both
operands as evaluated series on one grid and return trimmed grid series.
Semantics follow Prometheus `VectorBinop`, `resultMetric`,
`signatureFunc`, `VectorAnd`, `VectorOr` and `VectorUnless` at every
grid point:
- Match groups: `on(...)` keeps the listed labels (`__name__` when
listed); `ignoring(...)` and the default keep every other label except
`__name__`. Empty label values count as absent.
- Cardinality is checked per point, never over a series' whole lifetime:
- The "one" side (right, or left with `group_right`) may hold one series
per match group at a point where both operands are non-empty.
- One-to-one matching may match a group once per point, filtered
comparisons included.
- `group_x` results must have unique label sets per point.
Series of one group that are present at disjoint points never conflict.
- Result labels: arithmetic and `bool` comparisons drop `__name__`,
`__type__` and `__unit__`; one-to-one keeps the `on` labels or drops the
`ignoring` ones; `group_x` labels are copied from the "one" side, and
removed when it has no value for them.
- Values: comparisons without `bool` keep the left value where they
hold; `bool` returns 1 or 0. `^` and `atan2` use the standard library.
- Errors: the error of the earliest failing grid point is returned. At
one point a one-side duplicate is reported before a one-to-one or
`group_x` violation, as Prometheus checks the one side before matching.
- Fields are paired by position; the output carries the shorter side's
field count, named after the left side. Each field is matched on its
own, as if the operands held only that field, and the per-field results
are joined by label set. Set operators require equal field counts and
return `NotImplemented` otherwise.
- `or` requires the same field count on both sides and rejects others
with `NotImplemented`. Histogram samples are rejected with
`NotImplemented`.
Series are processed whole over their own grid ranges: the "one" side is
indexed by match group, and per-point checks run only over the points a
series covers.
Review focus:
- `binary_vectors`: the one-side duplicate check is limited to points
where both operands are non-empty, as Prometheus short-circuits empty
steps before checking.
- `binary_vectors`: the one-to-one check counts a match even when the
comparison filters it out.
- `Results::sort_rows`: duplicate rows of one result label set at a
point are the `group_x` uniqueness error; the earliest point across all
label sets is returned.
- `binary_vectors` error selection: earliest point wins; ties go to the
one-side duplicate.
- Results of different match groups that share a label set once
`__type__` is dropped fail as duplicate label sets after the operator,
as Prometheus does; two results within one group fail the `group_x`
uniqueness check.
### Non-goals
- Scalar operands (MAP-ARITH), `fill` modifiers, histogram operands
(MATCH-HIST), planner wiring.
### Constraints
Results agree with Prometheus within promql-compat's relative tolerance
(1e-10), not bit for bit.
The ordinary planner deviates from Prometheus in ways this kernel does
not reproduce:
- An operand whose plan has no tag columns is joined on time only and
broadcast to every series of the other side (`only_join_time_index`).
Prometheus matches it by label signature, so `sum(m) / m` returns
nothing in Prometheus.
- Labels with an empty value and absent labels join as different values
(`NullEquality::NullEqualsNull` over nullable tag columns). Prometheus
treats them as equal.
### Verification
- `cargo nextest run -p promql`: 317 passed.
- `cargo clippy -p promql --all-targets --all-features -- -D warnings`,
`cargo fmt --check`, `hawkeye check`: clean.
- `binary_vectors_match_prometheus_reference`: 3,000 random cases
against a per-point transcription of Prometheus
`VectorBinop`/`resultMetric`/`signatureFunc` that keeps labels in a map
with `__name__` as an ordinary label. The cases cover all 13 operators,
`bool`, random `on`/`ignoring` subsets including `__name__`, all three
cardinalities with random `group_x` labels, 1–2 fields per side, and
series with holes and disjoint lifetimes. Over 1,000 cases succeed and
over 100 hit a cardinality error; results compare label sets, absolute
grid ranges and value bits, and errors compare their class.
- `set_vectors_match_prometheus_reference`: 3,000 random cases for
`and`/`or`/`unless`.
- `vector_operations_match_prometheus`: 180 fixtures (120 binary, 60
set) produced by the Prometheus v3.15.0 engine
(`vector_match/prometheus_cases.txt`); results compare label sets and
grid points exactly and values within the tolerance, errors compare
their class.
- `pow_and_atan2_match_go`: 452 Go `math.Pow`/`math.Atan2` results
(`vector_match/go_math_cases.txt`) agree with the standard library
within the tolerance.
- `reports_the_earliest_failing_point`: a `group_left` violation at the
first point wins over a one-side duplicate at the second.
- Mutation check: dropping `__name__` for comparisons, ignoring the
`ignoring` list, checking one-side duplicates at points where an operand
is empty, removing the one-to-one match check, keeping a `group_x` label
the one side lacks, not filtering comparisons, keeping matched rows in
`or`, swapping `and`/`unless`, keeping all labels under `on`, accepting
duplicate result rows, and taking the wrong "many" side for
`group_right` each fail a test. So do replacing the Go ports with
`powf`/`atan2`, misplacing the fractional-exponent split in `pow`,
returning the first one-side or one-to-one violation found instead of
the earliest, always or never preferring the one-side duplicate on ties,
returning the first duplicate label set instead of the earliest, and
skipping the row sort.
- `colliding_results_fail_by_where_they_collide`: a collision across
match groups reports duplicate label sets, one within a group the
grouping error; the random reference now includes `__type__`.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
discord9anddiscord9
Lei, HUANG
discord9
Dennis Zhuang