Go reads a group name as Unicode letters, decimal digits and _, so in
$1² the name is 1; char::is_alphanumeric also takes ² and other number
characters.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Prometheus compares bounds with < and >, so +0 and -0 are equal, keep
input order, and coalescing keeps the first; ordering -0 first returned
the other zero and flipped the sign of an infinite reciprocal.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The aggregation kept one owned histogram slot per point of the whole
input span before trimming, which for a sparse group allocated far more
than the result.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
label_replace expands its template with Go's regexp.Expand rules, where
$01 or a reference of more than nine digits names a group. Series that
share a label set merge field by field, colliding only when one field
has two values at one point.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Results only need to agree with Prometheus within the relative tolerance
promql-compat uses, so Go's bound table and the ports of math.Log2,
Exp2, Ldexp and Frexp go; the fixtures are compared within that
tolerance.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
The evaluator stopped at a grid point whose window was not complete yet
before advancing past the samples that window cannot use, so a long gap
between grid points kept every sample pushed in the meantime.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Prometheus drops __type__ and __unit__ with __name__ for these functions,
but only __name__ for the single-histogram functions.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Port Prometheus FloatHistogram Add, KahanAdd, Sub, Mul, Div, Compact,
CopyToSchema, DetectReset and Equals, with zero bucket reconciliation,
resolution reduction and custom bound intersection, and use them for
sum and avg of native histograms within an aggregation group.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Add histogram_count/sum/avg/stddev/stdvar and native histogram_quantile
and histogram_fraction, ported from Prometheus HistogramQuantile,
HistogramFraction and histogramVariance with Go's bucket iteration,
exponential bound table and math.Log2/Exp2. histogram_quantile and
histogram_fraction now evaluate classic and native histograms in one
vector and skip both where they conflict, as resetHistograms does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Group bucket series by their labels other than le and evaluate each
point as Prometheus BucketQuantile and BucketFraction do, including
bucket coalescing, the small-delta tolerance and forced monotonicity.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
Add time(), the date functions and label_replace/label_join for
evaluated series, following Prometheus funcTime, dateWrapper,
evalLabelReplace and evalLabelJoin, and merge series that end up with
one label set as mergeSeriesWithSameLabelset does.
Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Depends on #9603, which defines the shared raw-series read contract;
it is merged into `narthex` as
`3fd69e312ef214d85b0fe7ef29db956cba33711c` with its CI checks green, and
this PR adds the Mito-side producer on top of that contract.
- Target branch: `narthex`, the shared integration branch, not `main`.
## What's changed and what's your intention?
Let the Mito engine serve the optional raw-series entrypoint, so a
per-series scan can be consumed as raw series chunks — encoded primary
keys plus Arrow-backed sample values — instead of converted record
batches. The producer uses the shared contract from #9603; this PR also
passes an explicit label binding into `scan_series_partition` and
documents typed-filter and residual-projection requirements. The trait's
existing default panic is unchanged; unsupported scanners must not be
called through that default.
### The producer
- The raw entrypoint reuses the ordinary per-series scan (same partition
pruner, key pushdown, and MVCC snapshot) and converts its scan batches
into raw series chunks: the complete encoded primary key, numeric
samples represented as `f64`, complete histogram payloads, strictly
increasing timestamps, and `is_last` for the last chunk of a series in
this stream.
- When this scanner cannot honor raw reads it returns `Ok(None)` before
consuming anything, so the caller can use the legacy record-batch scan;
`Some` selects the raw path, including a successful empty stream. Errors
are terminal and must not be replayed through the legacy path, and all
capability decisions are made before the first read, so one partition
never switches between the raw and legacy paths midway.
- It declines for explicit label sources, append mode, a series row
selector, a non-millisecond time index, and an empty or
non-numeric/non-histogram field selection.
- For non-region predicates, it declines non-immutable scalar functions,
scalar variables, compilation failures, and physical result types other
than Boolean or Null.
- A bound `PrimaryKey` label source is validated against the scanner's
metadata snapshot before any read: same metadata instance, primary-key
tags, non-reserved and non-internal names, and unique column ids and
names. An invalid binding errors instead of silently falling back.
- Field order follows the mapper's projection.
- Static residual predicates evaluate the original typed column arrays
before any conversion to the raw `f64` sample, so an `Int64` comparison
at a boundary stays exact instead of being applied after a lossy
conversion; a regression test pins that boundary.
- Inexact pushed-down predicates can still leave false-positive
candidates: callers keep the residual columns and the final projection,
and raw capability does not mean filtering is complete.
### Stream conversion
- Both internal input shapes are converted: legacy flat series batches
and plain record batches. Conversion runs inside the returned stream, so
no raw chunks are materialized when the entrypoint returns.
- Keys are zero-copy slices of the batch dictionary values, so chunks
survive the source batch; null or invalid dictionary keys error.
- Legacy input must present strictly increasing keys (terminal error
otherwise), while record-batch input need not, since keys arrive per
row.
- Timestamps must be strictly increasing within and across chunks;
duplicates or descending timestamps are terminal errors, not silent
skips. Null timestamps in an input batch are rejected before conversion.
- Ordering is declared exactly `Contiguous`: chunks of one series are
contiguous, and distinct series are unordered. It is deliberately not
`KeyThenTime` (this stream promises no series order) and not
`Interleaved`.
- Stream endpoints: an upstream error after a yielded prefix is terminal
without flushing pending state; empty input is a successful empty
stream; dropping the raw stream drops the unfinished input. Native
histogram fields are carried through as complete payloads.
### Scope and limits
- No production consumer and no public query switch: nothing outside
tests calls the raw entrypoint, so no query plan selects this path
today.
- The Mito per-series scanner is the only engine override of the raw
entrypoint; other engines keep the trait default.
`prepare_series_partition` keeps its default `None`, so there is no
prepared-source override for Mito, and explicit label sources are not
supported (the entrypoint returns `None` for them).
- This is not wired into merge-mode reads or Flight/remote reads, and it
makes no performance or optimization claim.
- Raw output is not persisted: the chunks are an in-memory read-side
representation, and storage format, wire protocol, and the ordinary
scan's pushdown, filtering, and MVCC semantics are unchanged. No
configuration switch or public option is added.
### Tests
- The added tests cover the raw candidate projection with MVCC
visibility, `Int64` predicate precision, an invalid binding rejected
before any legacy read, key ownership across batch drops, both input
shapes, stream finality, time validation, terminal errors, drop
behavior, and native histogram payloads.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
### Compatibility
- Adds raw-read support and uses the shared contract spellings
(`primary_key`, `RawTimestamps`, `SeriesOrdering`, `time.as_buffer()`);
these raw-series types exist only in this unreleased stack, the
spellings are a rename, and behavior and assertions are unchanged with
no test expectation weakened.
- No released API, schema, storage-format, or wire change, and raw
output is not persisted.
- No user-visible behavior change, so no documentation-site update is
needed, and there is no backport target.
### Verification
- Applies to this PR's exact head,
`c0eab0edbbf5ddaf975fbff762b730b0e6acca27`, a normal merge of the
actually merged `narthex` parent
`19b1c465fe33866e28a6bfb61051d7d5e832df8f` (including #9603, #9607 and
#9611) into the previous producer head
`6468d8778465570d20ed6dc0822f4796054b9ab1`. All six producer source
files are byte-identical to the previous head; public producer ordering
remains `Contiguous`.
- Commands run on this exact head with the repository's pinned
Rust/Clippy 1.98.1:
- `cargo check --locked --offline --all-targets -p datatypes -p
store-api -p common-query -p mito2 -p metric-engine` — passed.
- `cargo clippy --locked --offline --all-targets --all-features -p
datatypes -p store-api -p common-query -p mito2 -p metric-engine -- -D
warnings` — passed with the official pinned Clippy driver.
- `cargo nextest run --locked --offline -p datatypes -p store-api -p
common-query -p mito2 -p metric-engine --retries 0 --no-fail-fast` —
2370 passed, 0 skipped.
- Normal commit and push hooks (Rustfmt, TOML formatting, license
checks, typos and Snafu checks) passed; source hash checks before and
after verification and push matched.
- The passing suite includes raw candidate projection/MVCC, exact typed
`Int64` predicates, invalid binding rejection before a legacy read,
logical-metric routing/bound metadata, default-panic and unchanged
legacy-read regressions, plus finite and incremental SeriesFrame
coverage inherited from the merged parent.
- New-head remote CI is pending; this PR remains unmerged. The previous
head's Linux run failed only
`object-store::object_store_test::test_s3_backend` with a temporary
Minio/S3 transport error (`hyper::Error(IncompleteMessage)`) after four
attempts, and its failed job was normally rerun. That old-head run is
not evidence for this new commit; no source workaround or check bypass
is applied.
- Current exact-head workflow runs:
- [Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38894310912).
-
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38894310894).
- [Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38894310921).
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Depends on: #9607 (`feat/narthex-series-frame`).
- #9608 remains an independent storage producer; its implementation is
not part of this PR.
## What's changed and what's your intention?
Adds the in-process streaming boundary between raw-series streams and
DataFusion RecordBatch streams:
`pack_raw_series` and `unpack_raw_series` now move bounded multirow
Frame batches incrementally instead of
materializing one frame per union row. It is additive plumbing, not a
native-query switch, and it does not
change the frame payload or wire contract. The adapter is a new module
(`src/common/query/src/series_frame.rs`
plus its test modules), registered in `src/common/query/src/lib.rs`,
with `futures-util` promoted to a normal
dependency because the adapter implements `Stream`.
**Packing (`pack_raw_series`).** One `RawSeriesFrameBuilder` and one
`RawSeriesChunkCursor` live on the
stream, and a ready batch is yielded without polling the source again.
Rows are held across chunk boundaries
until the builder's row or byte bound is reached, until the source
returns `Pending` (the buffer is then
flushed so the pending source can still deliver), or until a valid end
of stream (the final chunk sequence
is validated and the buffer flushed). A legal row that exceeds the byte
bound alone is still emitted whole:
the bounds are a batching work limit, not a resident-memory cap.
**Pending, end of stream, errors and drop.** `Pending` never discards
buffered rows. A valid end of stream
flushes the buffer and requires every series to be finished, so an
unfinished series is a validation error and
an empty batch is not end of stream; an empty stream succeeds. Handling
is fail-fast and terminal: a validation, encode or upstream error
drops the source, the cursor and every buffered owner immediately and is
returned once — an error is never
treated as end of stream. Dropping (cancelling) the adapter mid-chunk
releases the upstream source, does not
drain the rest of the chunk, and leaves already-yielded batches fully
owned by the consumer.
**Unpacking (`unpack_raw_series`).** One `RawSeriesFrameDecoder` is
built per stream, so the expected frame
schema is prepared once, and the declared schema is preflighted with an
empty batch before the stream is
returned, so a bad declaration fails before the first poll. Every
decoded batch must match the declared stream
schema exactly, application metadata included: metadata that is stable
across the stream is accepted, and any
drift is rejected, including drift carried only by a zero-row batch.
Decoded rows are validated in order and
emitted before another batch is polled; empty batches are bounded per
poll and re-polled through a wake
instead of spinning.
**Ordering contract.** Only `SeriesOrdering::Contiguous` and
`SeriesOrdering::KeyThenTime` streams are
accepted in either direction; a `SeriesOrdering::Interleaved`
declaration is rejected before the first poll,
including for empty inputs, with no validator fallback. Validation is
adjacent-chunk only: timestamps strictly
increase within a key (including across chunks), the metric stays stable
within a key, a key must be marked
final before the key changes or recurs, and
`SeriesOrdering::KeyThenTime` additionally requires non-decreasing
key order. It does not certify nonadjacent A/B/A recurrence under
`SeriesOrdering::Contiguous`, query-global
ordering or partition ownership. `is_last` marks the final chunk of a
series in this input stream only:
splitting a final chunk marks only its final row, a non-final chunk
produces no final marker, and completion
across other input streams is not established.
**Binding and boundaries.** The binding is supplied by the caller and is
not authenticated by Frame metadata;
authorization, lazy label decoding, outer/dynamic residuals and the
final projection remain caller
responsibilities. The frame payload and wire contract are untouched: the
reserved frame metadata, the column
names and the `read_key` column literal stay as they are, and the
adapter signature changes are limited to
the shared ordering vocabulary and the renamed Rust field
(`primary_key`) of the same key buffer. No
production query-path switch, Flight/IPC network execution, distributed
or cross-region merge, spill or
exchange compaction, evaluation, prepared producer or public query route
is added.
**Tests.** The adapter tests cover round trips (bit-exact float
payloads, sliced owners and a real Arrow IPC
round trip), bounded batch splitting without re-polling the source,
flush at `Pending` and at end of stream,
oversized-but-legal rows, one terminal error without replay,
drop-mid-chunk release, empty/pending-batch poll
budgets, schema and metadata drift rejection, ordering rejection before
any poll, explicit/key label-mode
separation, and the single-row zero-copy path.
**Limitations.** This PR ships the adapter and its tests only; it is not
wired into a real Flight path, and no
consumer or planner changes are included. The local boundary
measurements below do not establish
end-to-end query performance. Earlier reviews apply to their reviewed
revisions, not automatically to
this updated adapter. The added adapter types are unreleased and no
existing
released wire or persisted format changes: the frame schema and metadata
stay on the frozen `v1` contract.
Open and unmerged; current-head CI has completed successfully.
### Bounded local Frame-boundary measurement (B baseline single-row, C
selected multirow)
Provenance: measured at the frozen local candidate
`eddd595b82888452b3c066678fabc6c1200c74f7`, whose
bounded-multirow finite codec content is
`3406a5334574a518924c67c319e2d92b6cac943b`, against the old public
baseline `36843e8cebeded1328dd7005ed668300a606f182`. The later normal
parent synchronization and the
strict-lint syntax-only fix are not separately rebenchmarked, so these
numbers describe that frozen local
content, not the pushed head; no remote CI or performance result is
implied by this local measurement. This
PR now implements the selected C direction and native A is unchanged:
the native direct path remains the
control and is not replaced. No query-speedup, Flight/IPC-network,
storage, service or integration conclusion
is drawn.
Fixture: six synthetic raw-series Frame-boundary cases in two sequential
campaigns: 100000 series in the
first campaign, 1000 series in the second, one chunk per series in both.
CPU affinity 2, seven repetitions
per group, default warmup on, plain-timing and allocator-instrumented
runs executed as separate passes
(plain first, then alloc), no retries, no outlier removal, no rescaling
and no selective repeats. The
baseline and candidate binaries were built with the same release
profile: `rustc 1.98.1 (48a229cea
2026-09-01)`, `debug=0`, `opt-level=3`, `lto=false`, `codegen-units=16`,
`incremental=0`.
Measured roles. A0 = native direct path, baseline build; A1 = native
direct path, candidate build; B = the
baseline single-row pack→unpack round trip (one frame per union row); C
= the selected multirow pack→unpack
round trip (one measured row cursor plus one bounded row builder, one
batch per flush). The two native
controls are listed separately and never pooled. C is the direction this
PR now implements, and the bounded
builder is why rows are held across chunk boundaries instead of
materializing one frame per row.
**Stage wall time (plain runner only).** Milliseconds, median [min–max];
MAD is the unscaled median absolute
deviation in milliseconds.
| Case | Series / rows per series | A0 native, baseline build | A1
native, candidate build | B baseline single-row round trip | C selected
multirow round trip |
|---|---:|---:|---:|---:|---:|
| float2_key16 | 100000 / 2 | 42.313 [41.972–44.164]; 0.341 | 43.520
[42.146–45.728]; 1.133 | 2549.316 [2435.763–2618.753]; 52.979 | 206.888
[199.977–208.500]; 1.612 |
| float3_key16 | 100000 / 3 | 59.434 [58.300–62.123]; 1.135 | 60.111
[59.460–60.870]; 0.400 | 3261.311 [3195.805–3309.910]; 10.551 | 260.933
[256.555–303.316]; 4.378 |
| float3_key128 | 100000 / 3 | 64.968 [62.159–78.136]; 1.527 | 62.999
[62.748–65.338]; 0.251 | 3240.701 [3223.537–3381.599]; 11.899 | 291.846
[263.834–328.907]; 19.877 |
| long_float1024_key16 | 1000 / 1024 | 7.556 [7.375–7.839]; 0.107 |
7.113 [6.939–7.539]; 0.051 | 26.311 [25.893–27.732]; 0.209 | 10.468
[9.849–10.832]; 0.299 |
| histogram3_key16 | 1000 / 3 | 0.452 [0.438–0.490]; 0.005 | 0.448
[0.440–0.472]; 0.008 | 17.250 [17.137–18.415]; 0.113 | 5.632
[5.528–5.992]; 0.053 |
| multifield_float_hist3_key16 | 1000 / 3 | 0.593 [0.588–0.632]; 0.005 |
0.588 [0.585–0.634]; 0.002 | 25.194 [24.633–26.472]; 0.561 | 6.328
[6.183–6.646]; 0.063 |
Ratios are deliberately not tabulated: a B/C or C/A1 ratio over one
boundary stage is not a query-level
number. The A0 and A1 ranges overlap in every case, so the native
controls establish no candidate-build
difference. The C/B ranges are disjoint in this bounded seven-round
sample, which is descriptive for these
fixtures only, not significance or generalization evidence.
**Actual frame and row counts.** Counted from the runs, not inferred
from the name of the mode; rows are
emitted frame rows, not sample positions. Both columns are invariant
across all seven rounds and both
runners; native A emits no frames.
| Case | Series × chunks | B baseline single-row batches / rows | C
selected multirow batches / rows | Logical positions |
|---|---:|---:|---:|---:|
| float2_key16 | 100000 × 1 | 100000 / 100000 | 391 / 100000 | 200000 |
| float3_key16 | 100000 × 1 | 100000 / 100000 | 391 / 100000 | 300000 |
| float3_key128 | 100000 × 1 | 100000 / 100000 | 391 / 100000 | 300000 |
| long_float1024_key16 | 1000 × 1 | 1000 / 1000 | 17 / 1000 | 1024000 |
| histogram3_key16 | 1000 × 1 | 1000 / 1000 | 4 / 1000 | 3000 |
| multifield_float_hist3_key16 | 1000 × 1 | 1000 / 1000 | 4 / 1000 |
3000 |
**Allocation (separate allocator-instrumented runner).** These counters
come only from the instrumented
runner; its timings are not used and the plain runner is not
instrumented. Deallocation calls equal
allocation calls in every run. "Requested" bytes are request sizes (the
realloc column sums full `new_size`,
not the delta) and are neither RSS nor allocator usable size. "Peak live
Δ" is the peak live-byte delta
relative to the meter baseline of that run, not whole-process peak RSS;
a larger value means more
phase-retained memory.
| Case | Role | Allocation calls | Reallocation calls | Requested alloc
B | Requested full-new realloc B | Peak live Δ B |
|---|---:|---:|---:|---:|---:|---:|
| float2_key16 | B baseline single-row | 55000553 | 400003 | 4640645622
| 198401920 | 40380 |
| float2_key16 | C selected multirow | 3690875 | 14085 | 364600734 |
17429056 | 458978 |
| float3_key16 | B baseline single-row | 68500694 | 400004 | 5597857156
| 198402320 | 46914 |
| float3_key16 | C selected multirow | 4405874 | 16432 | 482697300 |
20582480 | 585488 |
| float3_key128 | B baseline single-row | 68500694 | 400004 | 5631457268
| 198402320 | 47250 |
| float3_key128 | C selected multirow | 4405874 | 16432 | 527497412 |
20582480 | 627432 |
| long_float1024_key16 | B baseline single-row | 414412 | 4004 |
54141088 | 1986048 | 50438 |
| long_float1024_key16 | C selected multirow | 32694 | 300 | 36651101 |
131648 | 2156848 |
| histogram3_key16 | B baseline single-row | 381412 | 4006 | 39057104 |
1986084 | 36678 |
| histogram3_key16 | C selected multirow | 96233 | 437 | 12366165 |
642724 | 1336504 |
| multifield_float_hist3_key16 | B baseline single-row | 517553 | 4005 |
48814638 | 1985956 | 43188 |
| multifield_float_hist3_key16 | C selected multirow | 103926 | 460 |
13534819 | 674852 | 1454810 |
The native A controls allocate identically to each other in all six
cases (equal counters and equal
peak-live deltas; absolute peak-live bytes differ by one byte because
the meter baseline itself differs by
one byte).
**Whole-process RSS (plain runner, lifetime scope).** Kilobytes, median
[min–max], from the external
time-based maximum resident set size of each measured run; it therefore
includes startup, setup and output
and is not phase-isolated memory. The native A0/A1 lifetime readings
span 5280–6240 kB across these cases.
| Case | B baseline single-row | C selected multirow |
|---|---:|---:|
| float2_key16 | 6080 [5920–6240] | 7520 [7360–7520] |
| float3_key16 | 6240 [6080–6240] | 7520 [7520–7680] |
| float3_key128 | 6240 [6080–6240] | 7680 [7520–7680] |
| long_float1024_key16 | 6080 [5920–6080] | 9176 [9172–9336] |
| histogram3_key16 | 6080 [5920–6240] | 8320 [8160–8480] |
| multifield_float_hist3_key16 | 6080 [5920–6080] | 8640 [8480–8640] |
**Repeatability and retention.** All 336 measured records are retained
(168 per campaign): 6 cases × 4 roles
× 2 runners × 7 repetitions = 48 groups of seven. All 168 instrumented
runs pass cleanup — deallocations
equal allocations, post-phase live bytes equal that run's baseline,
reallocation failures 0 and underflow
events 0 — and every allocation, deallocation and reallocation call
count, requested byte total and
peak-live reading is identical across the seven rounds of its group (MAD
0). Only the meter's
live-baseline/cleanup value may differ by at most one byte between
processes, and it is not claimed
identical. Full logical digests, field kinds, structural and metadata
checks, CPU affinity 2, per-run
snapshots and exit statuses match the measured-scale reference on every
record, and all 1064 retained raw
artifacts verify against their recorded checksums. Nothing was rerun
after the accepted run and no round was
excluded.
**Accepted tradeoff.** The measured cost of the selected C direction is
accepted once, as part of that
direction: relative to B, C raises phase peak-live bytes and
whole-process RSS while lowering stage wall time
and allocation request counts in these six fixtures (largest observed
increase: the long-float case, peak
live Δ 2156848 B versus 50438 B and lifetime RSS 9176 kB versus 6080
kB). This is a
batching-versus-retained-memory tradeoff of this fixture, not a claim
about general or steady-state service
memory, not a veto of the direction and not an open design question.
**Reading limits.** Per-run one-minute load spans 1.06–4.29 on a shared
machine, and no host-idle, thermal or
frequency-isolation claim is made; process snapshots are scoped to the
sandbox PID namespace. Wall and CPU
values are phase clocks; the external time tool's whole-process counters
have coarse resolution and are not
used as phase clocks. The histogram and multifield native stages are
sub-millisecond, so no fine-grained
accuracy claim is made at that magnitude. Every SOURCE_KEY group uses
the identical explicit metric
`raw_bench_metric{host="bench-0"}`, which is not evidence about real
PromQL label diversity or cardinality;
the histogram table template is built outside the measured phase and
shared across series; `Samples::Mixed`
within one field is not covered. Only fixture generation, the codec
boundary, digests and phase-owned cleanup
are inside the measured phase. This is a local Frame-phase measurement
of these six synthetic cases — not a
Flight/IPC network path, not a whole-query or query-regression
benchmark, not a storage or service
measurement, and not a CI result.
<details>
<summary>All seven rounds per role (plain wall and process CPU), fixture
metadata with digests, and the acceptance checks behind the tables
above</summary>
Values are the verified local statistics, converted from nanosecond
medians to milliseconds and rounded to
three decimals; process CPU is `CLOCK_PROCESS_CPUTIME_ID`, not the
external time tool's counters. Repetitions
are listed in measurement order 1–7.
| Case | Role | Metric | Median [min–max] | MAD | Reps 1–7 |
|---|---|---|---:|---:|---|
| float2_key16 | A0 native, baseline build | wall ms | 42.313
[41.972–44.164] | 0.341 | 43.135, 43.758, 44.164, 41.972, 42.056,
42.313, 42.191 |
| float2_key16 | A0 native, baseline build | CPU ms | 42.318
[41.977–44.158] | 0.341 | 43.127, 43.754, 44.158, 41.977, 42.057,
42.318, 42.186 |
| float2_key16 | A1 native, candidate build | wall ms | 43.520
[42.146–45.728] | 1.133 | 43.520, 42.618, 45.728, 42.146, 43.789,
42.387, 44.658 |
| float2_key16 | A1 native, candidate build | CPU ms | 43.526
[42.152–45.731] | 1.128 | 43.526, 42.619, 45.731, 42.152, 43.782,
42.383, 44.654 |
| float2_key16 | B baseline single-row | wall ms | 2549.316
[2435.763–2618.753] | 52.979 | 2567.299, 2549.316, 2465.980, 2496.337,
2435.763, 2618.753, 2559.681 |
| float2_key16 | B baseline single-row | CPU ms | 2541.530
[2436.036–2618.982] | 44.918 | 2565.139, 2541.530, 2465.528, 2496.612,
2436.036, 2618.982, 2559.874 |
| float2_key16 | C selected multirow | wall ms | 206.888
[199.977–208.500] | 1.612 | 207.428, 206.888, 199.977, 201.735, 206.998,
208.500, 203.026 |
| float2_key16 | C selected multirow | CPU ms | 206.909
[200.005–208.528] | 1.618 | 207.458, 206.909, 200.005, 201.760, 207.015,
208.528, 203.033 |
| float3_key16 | A0 native, baseline build | wall ms | 59.434
[58.300–62.123] | 1.135 | 59.434, 58.442, 61.186, 62.123, 58.300,
61.205, 58.781 |
| float3_key16 | A0 native, baseline build | CPU ms | 59.438
[58.304–62.118] | 1.134 | 59.438, 58.443, 61.159, 62.118, 58.304,
61.201, 58.772 |
| float3_key16 | A1 native, candidate build | wall ms | 60.111
[59.460–60.870] | 0.400 | 60.111, 60.056, 59.460, 60.870, 59.460,
60.168, 60.511 |
| float3_key16 | A1 native, candidate build | CPU ms | 60.113
[59.466–60.868] | 0.399 | 60.113, 60.056, 59.468, 60.868, 59.466,
60.171, 60.512 |
| float3_key16 | B baseline single-row | wall ms | 3261.311
[3195.805–3309.910] | 10.551 | 3260.350, 3309.910, 3271.862, 3195.805,
3211.174, 3261.550, 3261.311 |
| float3_key16 | B baseline single-row | CPU ms | 3261.623
[3196.160–3310.088] | 10.600 | 3260.683, 3310.088, 3272.224, 3196.160,
3211.509, 3261.875, 3261.623 |
| float3_key16 | C selected multirow | wall ms | 260.933
[256.555–303.316] | 4.378 | 303.316, 275.333, 257.711, 258.074, 260.933,
256.555, 299.972 |
| float3_key16 | C selected multirow | CPU ms | 260.961
[256.565–301.641] | 4.395 | 301.641, 275.320, 257.731, 258.109, 260.961,
256.565, 299.980 |
| float3_key128 | A0 native, baseline build | wall ms | 64.968
[62.159–78.136] | 1.527 | 62.159, 78.136, 64.089, 66.290, 64.968,
66.937, 63.440 |
| float3_key128 | A0 native, baseline build | CPU ms | 64.965
[62.156–78.113] | 1.334 | 62.156, 78.113, 64.092, 66.299, 64.965,
65.967, 63.431 |
| float3_key128 | A1 native, candidate build | wall ms | 62.999
[62.748–65.338] | 0.251 | 64.904, 65.338, 62.766, 62.914, 62.999,
63.280, 62.748 |
| float3_key128 | A1 native, candidate build | CPU ms | 62.999
[62.744–65.343] | 0.255 | 64.913, 65.343, 62.755, 62.914, 62.999,
63.289, 62.744 |
| float3_key128 | B baseline single-row | wall ms | 3240.701
[3223.537–3381.599] | 11.899 | 3223.537, 3228.801, 3348.994, 3229.554,
3244.801, 3240.701, 3381.599 |
| float3_key128 | B baseline single-row | CPU ms | 3240.996
[3223.913–3381.298] | 15.381 | 3223.913, 3225.615, 3349.367, 3229.918,
3245.152, 3240.996, 3381.298 |
| float3_key128 | C selected multirow | wall ms | 291.846
[263.834–328.907] | 19.877 | 291.846, 328.907, 311.723, 263.834,
264.137, 288.462, 309.855 |
| float3_key128 | C selected multirow | CPU ms | 291.848
[263.853–328.925] | 19.915 | 291.848, 328.925, 311.763, 263.853,
264.152, 288.477, 309.890 |
| long_float1024_key16 | A0 native, baseline build | wall ms | 7.556
[7.375–7.839] | 0.107 | 7.657, 7.827, 7.449, 7.375, 7.839, 7.476, 7.556
|
| long_float1024_key16 | A0 native, baseline build | CPU ms | 7.556
[7.371–7.832] | 0.107 | 7.647, 7.828, 7.449, 7.371, 7.832, 7.477, 7.556
|
| long_float1024_key16 | A1 native, candidate build | wall ms | 7.113
[6.939–7.539] | 0.051 | 7.164, 7.539, 7.067, 6.939, 7.127, 6.997, 7.113
|
| long_float1024_key16 | A1 native, candidate build | CPU ms | 7.114
[6.940–7.533] | 0.047 | 7.160, 7.533, 7.067, 6.940, 7.116, 6.998, 7.114
|
| long_float1024_key16 | B baseline single-row | wall ms | 26.311
[25.893–27.732] | 0.209 | 26.311, 26.520, 26.225, 25.893, 27.732,
26.106, 26.924 |
| long_float1024_key16 | B baseline single-row | CPU ms | 26.304
[25.895–27.729] | 0.219 | 26.304, 26.523, 26.223, 25.895, 27.729,
26.101, 26.927 |
| long_float1024_key16 | C selected multirow | wall ms | 10.468
[9.849–10.832] | 0.299 | 10.169, 10.468, 10.542, 9.849, 10.540, 9.987,
10.832 |
| long_float1024_key16 | C selected multirow | CPU ms | 10.469
[9.850–10.833] | 0.309 | 10.160, 10.469, 10.517, 9.850, 10.541, 9.988,
10.833 |
| histogram3_key16 | A0 native, baseline build | wall ms | 0.452
[0.438–0.490] | 0.005 | 0.447, 0.475, 0.453, 0.438, 0.452, 0.451, 0.490
|
| histogram3_key16 | A0 native, baseline build | CPU ms | 0.451
[0.438–0.490] | 0.005 | 0.447, 0.475, 0.453, 0.438, 0.451, 0.451, 0.490
|
| histogram3_key16 | A1 native, candidate build | wall ms | 0.448
[0.440–0.472] | 0.008 | 0.465, 0.471, 0.444, 0.447, 0.440, 0.448, 0.472
|
| histogram3_key16 | A1 native, candidate build | CPU ms | 0.448
[0.440–0.472] | 0.008 | 0.464, 0.471, 0.444, 0.447, 0.440, 0.448, 0.472
|
| histogram3_key16 | B baseline single-row | wall ms | 17.250
[17.137–18.415] | 0.113 | 17.250, 18.324, 17.216, 17.137, 17.239,
17.619, 18.415 |
| histogram3_key16 | B baseline single-row | CPU ms | 17.235
[17.132–18.389] | 0.103 | 17.235, 18.302, 17.197, 17.132, 17.223,
17.614, 18.389 |
| histogram3_key16 | C selected multirow | wall ms | 5.632 [5.528–5.992]
| 0.053 | 5.597, 5.986, 5.632, 5.528, 5.648, 5.579, 5.992 |
| histogram3_key16 | C selected multirow | CPU ms | 5.633 [5.528–5.992]
| 0.053 | 5.598, 5.986, 5.633, 5.528, 5.643, 5.580, 5.992 |
| multifield_float_hist3_key16 | A0 native, baseline build | wall ms |
0.593 [0.588–0.632] | 0.005 | 0.588, 0.603, 0.590, 0.603, 0.593, 0.590,
0.632 |
| multifield_float_hist3_key16 | A0 native, baseline build | CPU ms |
0.593 [0.588–0.631] | 0.003 | 0.588, 0.603, 0.589, 0.593, 0.593, 0.590,
0.631 |
| multifield_float_hist3_key16 | A1 native, candidate build | wall ms |
0.588 [0.585–0.634] | 0.002 | 0.586, 0.605, 0.585, 0.617, 0.585, 0.588,
0.634 |
| multifield_float_hist3_key16 | A1 native, candidate build | CPU ms |
0.587 [0.585–0.633] | 0.002 | 0.586, 0.605, 0.585, 0.616, 0.585, 0.587,
0.633 |
| multifield_float_hist3_key16 | B baseline single-row | wall ms |
25.194 [24.633–26.472] | 0.561 | 24.676, 26.472, 24.633, 25.800, 24.725,
25.194, 26.190 |
| multifield_float_hist3_key16 | B baseline single-row | CPU ms | 25.182
[24.619–26.451] | 0.564 | 24.660, 26.451, 24.619, 25.771, 24.720,
25.182, 26.178 |
| multifield_float_hist3_key16 | C selected multirow | wall ms | 6.328
[6.183–6.646] | 0.063 | 6.308, 6.590, 6.183, 6.646, 6.318, 6.328, 6.391
|
| multifield_float_hist3_key16 | C selected multirow | CPU ms | 6.329
[6.184–6.646] | 0.063 | 6.309, 6.590, 6.184, 6.646, 6.319, 6.329, 6.392
|
The same seven-round records give the allocation state counts used
above; in every group they are identical
across all seven repetitions (MAD 0). Two examples at the two size
extremes, allocator-instrumented runner:
| Case | Role | Allocation calls | Deallocation calls | Reallocation
calls | Requested alloc B | Requested full-new realloc B | Peak live
bytes | Post-cleanup live B |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| float2_key16 | B baseline single-row | 55000553 (7/7) | 55000553 (7/7)
| 400003 (7/7) | 4640645622 (7/7) | 198401920 (7/7) | 85793 | 45413 |
| float2_key16 | C selected multirow | 3690875 (7/7) | 3690875 (7/7) |
14085 (7/7) | 364600734 (7/7) | 17429056 (7/7) | 504392 | 45414 |
| long_float1024_key16 | B baseline single-row | 414412 (7/7) | 414412
(7/7) | 4004 (7/7) | 54141088 (7/7) | 1986048 (7/7) | 95825 | 45387 |
| long_float1024_key16 | C selected multirow | 32694 (7/7) | 32694 (7/7)
| 300 (7/7) | 36651101 (7/7) | 131648 (7/7) | 2202236 | 45388 |
The remaining four cases follow the same pattern: every allocator
counter, requested-byte total and peak-live
reading equals its median in all seven repetitions of its group,
reallocation failures and underflow events
are 0, and post-cleanup live bytes equal that run's meter baseline. Only
that live-baseline/cleanup value
moves (by at most one byte) between processes.
Fixture metadata and logical digests (each digest is identical across
all 56 measured records of its case;
native A carries no frames):
| Case | Series | Chunks per series | Rows per series | Key len |
Logical positions | Logical digest |
|---|---:|---:|---:|---:|---:|---|
| float2_key16 | 100000 | 1 | 2 | 16 | 200000 | `0xdbeb81be394ac1d3` |
| float3_key16 | 100000 | 1 | 3 | 16 | 300000 | `0x731b981da38db277` |
| float3_key128 | 100000 | 1 | 3 | 128 | 300000 | `0x3be5490405031e57` |
| long_float1024_key16 | 1000 | 1 | 1024 | 16 | 1024000 |
`0x9f1da6c041555235` |
| histogram3_key16 | 1000 | 1 | 3 | 16 | 3000 | `0xaf874f7678af1d82` |
| multifield_float_hist3_key16 | 1000 | 1 | 3 | 16 | 3000 |
`0x79761914759b7b7c` |
Acceptance checks behind the tables above: 336/336 measured records
forming the complete 6 cases × 4 roles
× 2 runners × 7 repetitions matrix; 48 groups of seven repetitions;
per-record logical, field-kind,
structural and metadata agreement with the measured-scale reference; CPU
affinity 2 and exit status 0 on
every record; 168/168 instrumented runs with clean allocation
accounting; 1064/1064 retained raw artifacts
verifying against their recorded checksums. No rerun, no exclusion, and
allocator-instrumented timings were
not substituted for the plain-timing statistics.
</details>
### Verification
Pushed head: `5f0890ebeaa5a8252f77dc2d18d8456539714fc4`, normally
synchronized with #9607 at `ba88f817cdcffeae6594943c55aaced2ffa769e5`.
The three stream implementation/test files are byte-identical to the
measured candidate; subsequent changes are parent synchronization and
the finite codec's Clippy-only guard spelling.
Checks on this exact head:
- `cargo +1.98.1 check --locked -p datatypes -p store-api -p
common-query --all-targets` — passed.
- `cargo +1.98.1 clippy --locked -p datatypes -p store-api -p
common-query --all-targets --all-features -- -D warnings` — passed with
the official pinned 1.98.1 Clippy component.
- `cargo +1.98.1 nextest run --locked -p datatypes -p store-api -p
common-query --retries 0 --no-fail-fast` — **570 passed, 0 skipped**.
- `cargo +1.98.1 test --locked -p datatypes --doc` — **2 passed**,
including the `RawTimestamps` compile-fail contract and public
`Samples::segments` example.
- Rust 1.99 formatting, TOML/license/typo checks, `git diff --check`,
and normal signed-off commit/push hooks passed. Source hashes were
verified unchanged after the hooks.
The local measurements above remain attributed to their frozen candidate
rather than relabelled as a rebenchmark of this head. Current-head
remote CI completed successfully in all three workflows below. The PR
remains open and unmerged until the normal merge gates and parent-first
integration are checked.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
### Verified current-head remote CI
Exact pushed head: `5f0890ebeaa5a8252f77dc2d18d8456539714fc4`.
- [Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882957553)
— **success**.
-
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882957587)
— **success**.
- [Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882957636)
— **success**.
Linux and Windows test jobs, strict Clippy, builds, SQLness and the
executed integration/fuzz jobs succeeded. Coverage and recent-release
compatibility jobs were skipped by their workflow conditions; they are
not claimed as executed. These CI results are independent of the frozen
local experiment provenance above.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Depends on: #9603 (`feat/narthex-raw-contract`).
- Base of: #9611 (`feat/narthex-raw-frame-stream`).
- The finite raw SeriesFrame codec remains independent of the Mito
producer in #9608.
## What's changed and what's your intention?
Adds an Arrow `RecordBatch` representation for the raw series chunks
defined in the stack, so subsequent
execution and transport code can exchange typed samples without
rebuilding every sample, plus the bounded
multirow encoder that packs raw series chunks into multi-row batches
directly.
**Codec.** The new `series::frame` module provides raw-frame schema
construction, chunk packing and
whole-batch unpacking, in encoded-key and explicit-label modes,
preserving explicit timestamps,
float/histogram/mixed payloads, nulls, field order and `is_last`.
Different fields may have different chunk
boundaries: packing splits at their union and retains Arrow views rather
than densifying the inputs.
`Samples::segments()` exposes the zero-copy dense segment view the codec
walks, with a public-call doctest, so
the codec can consume existing dense segments without a new segment type
or allocation. Decode validates
schema metadata, payload lengths, indices, histogram structure and
timestamps, and any malformed row fails the
entire unpacking call rather than returning a successful prefix.
**Bounded multirow encoding.** `RawSeriesFrameBuilder` buffers rows and
emits one batch when either bound is
reached (`DEFAULT_MAX_ROWS` 256, `DEFAULT_MAX_BYTES` 1 MiB); the
effective byte cap is stored as
`min(requested, i32::MAX)`, which keeps every cumulative Arrow offset
domain representable for any admitted
group. `RawSeriesChunkCursor` walks one already validated chunk at the
union of the field segment boundaries,
holds at most one pending measured row, and measures each row exactly
once; the pending row is retried after a
flush, so no row is dropped or split, and per-field offsets advance
against each field's original segment
length, which keeps nonzero-offset (sliced) sample arrays correct. The
byte measure dominates the encoder's
offset domains: key and timestamp bytes, list element counts and bitmap
bytes, and the visible extent of each
histogram table list column read from the rebased `value_offsets` view
rather than from retained capacity. It
is an internal batching work estimate — not an exact IPC size and not a
resident-memory or RSS cap — so one
individually legal row that exceeds the requested cap is still accepted
and emitted alone, because that slice
cannot be split into smaller legal rows.
**Histogram layout and sharing.** Row-local histogram indices are never
remapped; physical tables are
concatenated under cumulative row list offsets, so each row still
addresses its own table through a nonzero
slice of the child. A row group with a single Arrow constituent reuses
it by clone, which shares buffers and
keeps the single-row zero-copy and owner-drop behavior; a group with
several constituents is concatenated,
which copies. Both cases are asserted. The `Samples::into_dense` sharing
check compares the Arrow table view
(`ArrayData` identity) in addition to `Arc` identity, and the first
histogram batch's `ArrayData` is computed
lazily and cached: that is pointer-identity caching, not content
deduplication, and distinct buffers with
equal content are never merged. Local buffer sharing does not prevent a
full table from being encoded in every
batch; any future trimming or remapping belongs at the actual
IPC/network/spill boundary, reusing Arrow
slicing and selection and the existing writer, and preserving duplicate
indices without content hashing.
**Preserved interfaces.** `pack_raw_series_chunk` keeps its
`Vec<RecordBatch>` return type and
`unpack_raw_series_frame` keeps its signature; `RawSeriesFrameDecoder`
prepares the expected frame schema once
for a stream of batches, while the wrapper still builds a decoder per
call. The raw contract's `RawTimestamps`
validates strictly increasing source timestamps once at construction
behind a private buffer. The frozen `v1`
contract is untouched: the four reserved metadata entries, the column
names (`read_key`, `is_last`, `time`,
`fields`, plus `name`/`labels` in explicit mode) and the wire column
literal `read_key` stay as they were
(`primary_key` is only the Rust-side name of that buffer).
Histogram-only rows decode through
`Samples::try_from_histogram` directly, without a duplicate bounds
check.
**Tests.** The frame tests cover schema and metadata acceptance and
rejection, bit-exact payloads (including
NaN payloads and `-0.0`), sliced-payload owner lifetimes, row and byte
bounds (bounds taken from real measured
rows, flush-before-exceed with pending retry, legal oversized row),
row-local histogram indices and complete
tables with a real Arrow IPC round trip, decoder schema caching and
label-mode separation. The `samples` tests
pin both sides of the sharing rule: distinct wrappers of one Arrow table
view are reused, while distinct
buffers, distinct table slices, and shared value buffers with different
null-bitmap offsets are never merged.
**Limitations and boundaries.** Each histogram-bearing row carries its
complete histogram table, including
entries no index references, and a large table referenced by several
segments can be repeated per row; there
is no deduplication and no global reindex, so an encoded batch can
exceed the source sample bytes. These entry
points are exercised by unit tests, and the builder, cursor and decoder
are also used by the in-process stream
adapter of #9611, which is not yet wired into a production query path;
there is no transport, persistence or
query-performance change. This PR handles finite raw frames only: it
does not add stream-wide
key/order/EOF validation (that is #9611), evaluated frames, a Flight/IPC
execution path, logical-name
authentication, a native query consumer, a production query-path switch,
distributed or cross-region merging,
or a prepared producer. The frame preserves ordered logical field names
and rejects mismatched supplied
schemas; that is not authentication of label binding or data provenance.
Existing persisted storage formats
and query entrypoints are unchanged, and this is not a cross-version
wire-compatibility claim. Open and
unmerged; current-head CI has completed successfully.
### Bounded local Frame-boundary measurement (B baseline single-row, C
selected multirow)
Provenance: measured at the frozen local candidate
`eddd595b82888452b3c066678fabc6c1200c74f7`, whose
bounded-multirow finite codec content is
`3406a5334574a518924c67c319e2d92b6cac943b`, against the old public
baseline `36843e8cebeded1328dd7005ed668300a606f182`. The later normal
parent synchronization and the
strict-lint syntax-only fix are not separately rebenchmarked, so these
numbers describe that frozen local
content, not the pushed head; no remote CI or performance result is
implied by this local measurement. This
PR now implements the selected C direction and native A is unchanged:
the native direct path remains the
control and is not replaced.
A bounded local-only experiment measured this codec's role as the
pack/unpack kernel of that direction.
Fixture: six synthetic raw-series Frame-boundary cases in two sequential
campaigns (100000 series in the
first, 1000 in the second, one chunk per series in both), CPU affinity
2, seven repetitions, default warmup,
plain-timing and allocator-instrumented runs kept separate, no retries
and no excluded rounds. Stage wall
time from the plain runner, milliseconds, median only, for the six cases
in table order:
| Case | Series / rows per series | A0 native, baseline build | A1
native, candidate build | B baseline single-row | C selected multirow |
|---|---:|---:|---:|---:|---:|
| float2_key16 | 100000 / 2 | 42.313 | 43.520 | 2549.316 | 206.888 |
| float3_key16 | 100000 / 3 | 59.434 | 60.111 | 3261.311 | 260.933 |
| float3_key128 | 100000 / 3 | 64.968 | 62.999 | 3240.701 | 291.846 |
| long_float1024_key16 | 1000 / 1024 | 7.556 | 7.113 | 26.311 | 10.468 |
| histogram3_key16 | 1000 / 3 | 0.452 | 0.448 | 17.250 | 5.632 |
| multifield_float_hist3_key16 | 1000 / 3 | 0.593 | 0.588 | 25.194 |
6.328 |
The same run's full min–max, MAD, all seven rounds per role, allocation
counters, RSS, frame counts and
acceptance checks are in #9611, which carries the canonical full results
— including the frame totals counted
from the runs (B emitted 100000 / 100000 / 100000 / 1000 / 1000 / 1000
batches and C 391 / 391 / 391 / 17 /
4 / 4, with frame rows equal to series × chunks in every case and no
frames on the native path).
Reading: the A0 and A1 native control ranges overlap in every case, so
the native controls establish no
candidate-build difference, and the C/B range disjointness in this
bounded seven-round sample is descriptive
for these synthetic fixtures only. Relative to B, C lowers stage wall
time and allocation request counts while
raising phase peak-live bytes and whole-process RSS; that
batching-versus-retained-memory tradeoff of the
selected direction is accepted once, as part of that direction, and is
not a veto or an open design question
(largest observed increase: the long-float case, peak live Δ 2156848 B
versus 50438 B and lifetime RSS
9176 kB versus 6080 kB).
Scope limits, as in #9611: every SOURCE_KEY group uses one identical
explicit metric label set, so this is
not real PromQL label diversity or cardinality evidence; the histogram
table template is built outside the
measured phase; no `Samples::Mixed` inside one field. The measured scope
is local Frame generation, the codec
boundary, digests and phase-owned cleanup — not a Flight/IPC network
path, not a whole-query or
query-regression benchmark, not a storage or service measurement, and
not a CI result. No query speedup is
claimed from this codec or from the selected direction.
### Verification
Pushed head: `ba88f817cdcffeae6594943c55aaced2ffa769e5`, synchronized
normally with #9603 at `794389688084ee97a7760d1ae993bcf1c109489e`.
Checks on the exact frozen source recorded by this head:
- `cargo +1.98.1 check --locked -p datatypes --all-targets` — passed.
- `cargo +1.98.1 clippy --locked -p datatypes --all-targets
--all-features -- -D warnings` — passed with the official pinned 1.98.1
Clippy component.
- `cargo +1.98.1 nextest run --locked -p datatypes --retries 0
--no-fail-fast` — **363 passed, 0 skipped**.
- `cargo +1.98.1 test --locked -p datatypes --doc
series::samples::Samples::segments` — **1 passed**.
- Rust 1.99 formatting, TOML/license/typo checks, `git diff --check`,
and normal signed-off commit/push hooks passed. Source hashes were
verified unchanged after the hooks.
The final closeout only collapses the full-builder flush guard into a
short-circuit-preserving let-chain for strict Clippy; it changes no
batching behavior or assertion. The local measurements above remain
attributed to their frozen candidate. Current-head remote CI completed
successfully in all three workflows below. The PR remains open and
unmerged until the normal merge gates and parent-first integration are
checked.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
### Verified current-head remote CI
Exact pushed head: `ba88f817cdcffeae6594943c55aaced2ffa769e5`.
- [Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922544)
— **success**.
-
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922504)
— **success**.
- [Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922665)
— **success**.
Linux and Windows test jobs, strict Clippy, builds, SQLness and the
executed integration/fuzz jobs succeeded. Coverage and recent-release
compatibility jobs were skipped by their workflow conditions; they are
not claimed as executed. These CI results are independent of the frozen
local experiment provenance above.
I hereby agree to the terms of the [GreptimeDB
CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md).
## Refer to a related PR or issue link (optional)
- Related: #9608, which builds on this PR's branch and adds the
engine-side raw-series producer; the contract reviewed here is this PR's
own diff.
- Base branch: `narthex`, the shared integration branch rather than
`main` (it already contains the merged Narthex PRs #9598, #9599, and
#9602).
## What's changed and what's your intention?
Define the raw-series read contract shared by storage engines and their
consumers, and separate "which series are read" from "how a reader
streams them". The contract lands on its own so its vocabulary is
reviewed once; the engine-side producer arrives in a follow-up PR.
### Raw series values and streams
- `RawSeriesChunk` is one raw source batch: the complete encoded
`primary_key` (not a TSID or hash alone), an optional `metric`, a
strictly increasing `time` axis, fields aligned to that axis, and
`is_last`. `try_new` rejects an evaluation grid, mismatched field count
or length, and an empty time axis; validation is per chunk and does not
claim ordering between chunks.
- `RawTimestamps` keeps its buffer private, so "strictly increasing, no
duplicates" is validated once at construction and cannot be bypassed by
a struct literal (pinned by a `compile_fail` doctest). Slices are
zero-copy and keep the underlying buffer alive.
- `RawSeriesStream` is the reader side: a `Stream` of
`Result<RawSeriesChunk, BoxedError>` items exposing `schema`,
`label_source`, and `ordering`. `SendableRawSeriesStream` is its pinned
boxed form.
- `RawLabelSource` carries the binding: `PrimaryKey` means labels are
encoded in the complete primary key and decoded with scan-bound
metadata; `Explicit` means the stream supplies the complete name/label
set itself, never a partial projection. Chunks under `PrimaryKey` carry
`metric: None` because labels are undecoded, not because the metric name
is missing, while `Some(metric)` is always complete.
- `PreparedSeriesSource` is the selection side and is deliberately not a
reader: `num_series()`, `series_info(ordinal)`, `label_source()`, and
`read_series(range)`. Both it and `RawSeriesStream` return the same
`label_source()`, so a holder can interpret labels before polling
anything.
### Shared ordering vocabulary
- `datatypes::series::SeriesOrdering` is a common vocabulary that raw
and evaluated series can express; in this PR the only user is
`RawSeriesStream::ordering`, and no evaluated consumer or implementation
is added.
- Modes: `Interleaved` (a series may resume after another series),
`Contiguous` (each series' chunks are contiguous), and `KeyThenTime`
(this stream's series identity, then position). Key order uses the
complete bound-compatible identity, not a TSID or hash alone.
- All modes are per-stream guarantees only: positions strictly increase
within a series, including across chunks, with no duplicates, and there
is no query-global ordering or finality claim. `is_last` marks the last
chunk of that series in this one stream; merge code must recompute
finalization across all related inputs.
### Prepared series selection
- A source describes a finite, stable selected set: `num_series()` is
exact, `series_info(ordinal)` returns identity metadata without polling
samples, `label_source()` is available without opening a reader, and
`read_series(range)` covers a half-open ordinal range.
- Ordinals are stable local indexes, not protocol identifiers, and each
ordinal's identity matches the chunks produced for it. Readers are
independent: repeated or overlapping reads do not consume shared
iteration state, dropping one reader does not cancel siblings, and a
reader owns what it needs to continue after the source is dropped.
- Invalid ordinals and ranges return errors; an empty in-bounds range is
an ordinary empty stream. `SeriesInfo::metric` follows chunk semantics
(`None` = primary-key label binding, `Some` = complete).
- The documented requirements — no sample payload loading or hidden
blocking I/O during preparation, identity lookup, or reader creation,
and binding to the partition's immutable MVCC view, predicates,
authorization, projection, and labels — are requirements for future
implementations.
- This PR adds no engine implementation, so it makes no I/O,
performance, or MVCC behavior claim.
- The store-api tests include a small in-memory `PreparedSeriesSource`
that pins these guarantees: binding available before readers, empty
ranges, invalid ordinals and ranges, independent overlapping readers
across source and reader drops, and a selection that does not change
when a later mutable selection does.
### Scanner entrypoints
- `RegionScanner::prepare_series_partition` is the optional capability
and defaults to `Ok(None)`, meaning "unsupported before execution, so
the caller may use the legacy record-batch scan". No scanner overrides
it in this PR, and the only `PreparedSeriesSource` implementation here
is the store-api test fixture.
- `RegionScanner::scan_series_partition` keeps its existing default
panic (`todo!()`), now documented as must-override; a store-api test
pins that panic and shows the legacy `scan_partition` result is
unchanged.
- Callers must use exactly one entrypoint per scanner partition and
execution, and must not replay a failed raw scan through the legacy
path.
### Compatibility
The legacy record-batch scan and existing queries are unchanged, and
nothing in planning or execution calls the new entrypoints yet. No
persisted format, wire protocol, storage format, or configuration change
is made: the added types are in-memory Rust types with no serialized
representation.
Within this stack the raw-series types are unreleased, and the branch
replaces the interim raw-only ordering enum with the shared
`SeriesOrdering` plus the source-level label binding. For released code
the change is additive: new trait methods with default implementations,
and no released method signature changes.
Note for reviewers: the malformed-range case is built with
`std::ops::Range { start: 2, end: 1 }` rather than a reversed range
literal, because `clippy::reversed_empty_ranges` rejects the literal
form; the assertion itself is unchanged and no lint allow is added.
## PR Checklist
Please convert it to a draft if some of the following conditions are not
met.
- [x] I have written the necessary rustdoc comments.
- [x] I have added the necessary unit tests and integration tests.
- [ ] This PR requires documentation updates.
- [x] API changes are backward compatible.
- [x] Schema or data changes are backward compatible.
- [ ] This PR needs to be backported to release branches, and I have
added the `backport-<target>` labels (e.g. `backport-v1.3` targets
`release/v1.3`).
### Verification
Local checks below ran on the exact source now pushed to this PR,
including the test-only range fix noted above.
- `cargo +1.98.1 check --locked -p datatypes -p store-api --all-targets`
— passed.
- `cargo nextest run --locked -p datatypes -p store-api --retries 0
--no-fail-fast` — 432 passed, 0 skipped.
- `cargo test --locked -p datatypes --doc RawTimestamps` — 1 passed.
- `cargo clippy --locked -p datatypes -p store-api --all-targets
--all-features -- -D warnings` — passed on the pinned 1.98.1 toolchain,
covering the clippy lint that the fixed test file previously tripped.
- Exact-head remote CI for `794389688084ee97a7760d1ae993bcf1c109489e`
completed successfully: [Rust
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322418)
(including workspace Clippy, Linux tests, Windows build/tests and
riscv64 build),
[Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322439),
and [Integration
CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322830)
(SQL, SDK, import/export and fuzz jobs). The workflow marked coverage
and recent-release compatibility jobs skipped; those two jobs are not
claimed as executed.
Dennis Zhuang
discord9
discord9