mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-13 22:36:57 +00:00
3caba1b7c2e433df15d237fe33a67b7899591b35
6446
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 discord9
|
3caba1b7c2 |
feat(common-recordbatch): recognize SeriesScanExec in metrics and preserve typed errors
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
 discord9
|
9ce017653e |
feat(mito2): expose scan-bound raw series candidates (#9608)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Depends on #9603, which defines the shared raw-series read contract; it is merged into `narthex` as `3fd69e312ef214d85b0fe7ef29db956cba33711c` with its CI checks green, and this PR adds the Mito-side producer on top of that contract. - Target branch: `narthex`, the shared integration branch, not `main`. ## What's changed and what's your intention? Let the Mito engine serve the optional raw-series entrypoint, so a per-series scan can be consumed as raw series chunks — encoded primary keys plus Arrow-backed sample values — instead of converted record batches. The producer uses the shared contract from #9603; this PR also passes an explicit label binding into `scan_series_partition` and documents typed-filter and residual-projection requirements. The trait's existing default panic is unchanged; unsupported scanners must not be called through that default. ### The producer - The raw entrypoint reuses the ordinary per-series scan (same partition pruner, key pushdown, and MVCC snapshot) and converts its scan batches into raw series chunks: the complete encoded primary key, numeric samples represented as `f64`, complete histogram payloads, strictly increasing timestamps, and `is_last` for the last chunk of a series in this stream. - When this scanner cannot honor raw reads it returns `Ok(None)` before consuming anything, so the caller can use the legacy record-batch scan; `Some` selects the raw path, including a successful empty stream. Errors are terminal and must not be replayed through the legacy path, and all capability decisions are made before the first read, so one partition never switches between the raw and legacy paths midway. - It declines for explicit label sources, append mode, a series row selector, a non-millisecond time index, and an empty or non-numeric/non-histogram field selection. - For non-region predicates, it declines non-immutable scalar functions, scalar variables, compilation failures, and physical result types other than Boolean or Null. - A bound `PrimaryKey` label source is validated against the scanner's metadata snapshot before any read: same metadata instance, primary-key tags, non-reserved and non-internal names, and unique column ids and names. An invalid binding errors instead of silently falling back. - Field order follows the mapper's projection. - Static residual predicates evaluate the original typed column arrays before any conversion to the raw `f64` sample, so an `Int64` comparison at a boundary stays exact instead of being applied after a lossy conversion; a regression test pins that boundary. - Inexact pushed-down predicates can still leave false-positive candidates: callers keep the residual columns and the final projection, and raw capability does not mean filtering is complete. ### Stream conversion - Both internal input shapes are converted: legacy flat series batches and plain record batches. Conversion runs inside the returned stream, so no raw chunks are materialized when the entrypoint returns. - Keys are zero-copy slices of the batch dictionary values, so chunks survive the source batch; null or invalid dictionary keys error. - Legacy input must present strictly increasing keys (terminal error otherwise), while record-batch input need not, since keys arrive per row. - Timestamps must be strictly increasing within and across chunks; duplicates or descending timestamps are terminal errors, not silent skips. Null timestamps in an input batch are rejected before conversion. - Ordering is declared exactly `Contiguous`: chunks of one series are contiguous, and distinct series are unordered. It is deliberately not `KeyThenTime` (this stream promises no series order) and not `Interleaved`. - Stream endpoints: an upstream error after a yielded prefix is terminal without flushing pending state; empty input is a successful empty stream; dropping the raw stream drops the unfinished input. Native histogram fields are carried through as complete payloads. ### Scope and limits - No production consumer and no public query switch: nothing outside tests calls the raw entrypoint, so no query plan selects this path today. - The Mito per-series scanner is the only engine override of the raw entrypoint; other engines keep the trait default. `prepare_series_partition` keeps its default `None`, so there is no prepared-source override for Mito, and explicit label sources are not supported (the entrypoint returns `None` for them). - This is not wired into merge-mode reads or Flight/remote reads, and it makes no performance or optimization claim. - Raw output is not persisted: the chunks are an in-memory read-side representation, and storage format, wire protocol, and the ordinary scan's pushdown, filtering, and MVCC semantics are unchanged. No configuration switch or public option is added. ### Tests - The added tests cover the raw candidate projection with MVCC visibility, `Int64` predicate precision, an invalid binding rejected before any legacy read, key ownership across batch drops, both input shapes, stream finality, time validation, terminal errors, drop behavior, and native histogram payloads. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ### Compatibility - Adds raw-read support and uses the shared contract spellings (`primary_key`, `RawTimestamps`, `SeriesOrdering`, `time.as_buffer()`); these raw-series types exist only in this unreleased stack, the spellings are a rename, and behavior and assertions are unchanged with no test expectation weakened. - No released API, schema, storage-format, or wire change, and raw output is not persisted. - No user-visible behavior change, so no documentation-site update is needed, and there is no backport target. ### Verification - Applies to this PR's exact head, `c0eab0edbbf5ddaf975fbff762b730b0e6acca27`, a normal merge of the actually merged `narthex` parent `19b1c465fe33866e28a6bfb61051d7d5e832df8f` (including #9603, #9607 and #9611) into the previous producer head `6468d8778465570d20ed6dc0822f4796054b9ab1`. All six producer source files are byte-identical to the previous head; public producer ordering remains `Contiguous`. - Commands run on this exact head with the repository's pinned Rust/Clippy 1.98.1: - `cargo check --locked --offline --all-targets -p datatypes -p store-api -p common-query -p mito2 -p metric-engine` — passed. - `cargo clippy --locked --offline --all-targets --all-features -p datatypes -p store-api -p common-query -p mito2 -p metric-engine -- -D warnings` — passed with the official pinned Clippy driver. - `cargo nextest run --locked --offline -p datatypes -p store-api -p common-query -p mito2 -p metric-engine --retries 0 --no-fail-fast` — 2370 passed, 0 skipped. - Normal commit and push hooks (Rustfmt, TOML formatting, license checks, typos and Snafu checks) passed; source hash checks before and after verification and push matched. - The passing suite includes raw candidate projection/MVCC, exact typed `Int64` predicates, invalid binding rejection before a legacy read, logical-metric routing/bound metadata, default-panic and unchanged legacy-read regressions, plus finite and incremental SeriesFrame coverage inherited from the merged parent. - New-head remote CI is pending; this PR remains unmerged. The previous head's Linux run failed only `object-store::object_store_test::test_s3_backend` with a temporary Minio/S3 transport error (`hyper::Error(IncompleteMessage)`) after four attempts, and its failed job was normally rerun. That old-head run is not evidence for this new commit; no source workaround or check bypass is applied. - Current exact-head workflow runs: - [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38894310912). - [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38894310894). - [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38894310921). |
||
|
discord9
|
c0eab0edbb |
chore: sync raw producer with merged series frame parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
19b1c465fe |
feat(query): add incremental raw series frame streams (#9611)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Depends on: #9607 (`feat/narthex-series-frame`). - #9608 remains an independent storage producer; its implementation is not part of this PR. ## What's changed and what's your intention? Adds the in-process streaming boundary between raw-series streams and DataFusion RecordBatch streams: `pack_raw_series` and `unpack_raw_series` now move bounded multirow Frame batches incrementally instead of materializing one frame per union row. It is additive plumbing, not a native-query switch, and it does not change the frame payload or wire contract. The adapter is a new module (`src/common/query/src/series_frame.rs` plus its test modules), registered in `src/common/query/src/lib.rs`, with `futures-util` promoted to a normal dependency because the adapter implements `Stream`. **Packing (`pack_raw_series`).** One `RawSeriesFrameBuilder` and one `RawSeriesChunkCursor` live on the stream, and a ready batch is yielded without polling the source again. Rows are held across chunk boundaries until the builder's row or byte bound is reached, until the source returns `Pending` (the buffer is then flushed so the pending source can still deliver), or until a valid end of stream (the final chunk sequence is validated and the buffer flushed). A legal row that exceeds the byte bound alone is still emitted whole: the bounds are a batching work limit, not a resident-memory cap. **Pending, end of stream, errors and drop.** `Pending` never discards buffered rows. A valid end of stream flushes the buffer and requires every series to be finished, so an unfinished series is a validation error and an empty batch is not end of stream; an empty stream succeeds. Handling is fail-fast and terminal: a validation, encode or upstream error drops the source, the cursor and every buffered owner immediately and is returned once — an error is never treated as end of stream. Dropping (cancelling) the adapter mid-chunk releases the upstream source, does not drain the rest of the chunk, and leaves already-yielded batches fully owned by the consumer. **Unpacking (`unpack_raw_series`).** One `RawSeriesFrameDecoder` is built per stream, so the expected frame schema is prepared once, and the declared schema is preflighted with an empty batch before the stream is returned, so a bad declaration fails before the first poll. Every decoded batch must match the declared stream schema exactly, application metadata included: metadata that is stable across the stream is accepted, and any drift is rejected, including drift carried only by a zero-row batch. Decoded rows are validated in order and emitted before another batch is polled; empty batches are bounded per poll and re-polled through a wake instead of spinning. **Ordering contract.** Only `SeriesOrdering::Contiguous` and `SeriesOrdering::KeyThenTime` streams are accepted in either direction; a `SeriesOrdering::Interleaved` declaration is rejected before the first poll, including for empty inputs, with no validator fallback. Validation is adjacent-chunk only: timestamps strictly increase within a key (including across chunks), the metric stays stable within a key, a key must be marked final before the key changes or recurs, and `SeriesOrdering::KeyThenTime` additionally requires non-decreasing key order. It does not certify nonadjacent A/B/A recurrence under `SeriesOrdering::Contiguous`, query-global ordering or partition ownership. `is_last` marks the final chunk of a series in this input stream only: splitting a final chunk marks only its final row, a non-final chunk produces no final marker, and completion across other input streams is not established. **Binding and boundaries.** The binding is supplied by the caller and is not authenticated by Frame metadata; authorization, lazy label decoding, outer/dynamic residuals and the final projection remain caller responsibilities. The frame payload and wire contract are untouched: the reserved frame metadata, the column names and the `read_key` column literal stay as they are, and the adapter signature changes are limited to the shared ordering vocabulary and the renamed Rust field (`primary_key`) of the same key buffer. No production query-path switch, Flight/IPC network execution, distributed or cross-region merge, spill or exchange compaction, evaluation, prepared producer or public query route is added. **Tests.** The adapter tests cover round trips (bit-exact float payloads, sliced owners and a real Arrow IPC round trip), bounded batch splitting without re-polling the source, flush at `Pending` and at end of stream, oversized-but-legal rows, one terminal error without replay, drop-mid-chunk release, empty/pending-batch poll budgets, schema and metadata drift rejection, ordering rejection before any poll, explicit/key label-mode separation, and the single-row zero-copy path. **Limitations.** This PR ships the adapter and its tests only; it is not wired into a real Flight path, and no consumer or planner changes are included. The local boundary measurements below do not establish end-to-end query performance. Earlier reviews apply to their reviewed revisions, not automatically to this updated adapter. The added adapter types are unreleased and no existing released wire or persisted format changes: the frame schema and metadata stay on the frozen `v1` contract. Open and unmerged; current-head CI has completed successfully. ### Bounded local Frame-boundary measurement (B baseline single-row, C selected multirow) Provenance: measured at the frozen local candidate `eddd595b82888452b3c066678fabc6c1200c74f7`, whose bounded-multirow finite codec content is `3406a5334574a518924c67c319e2d92b6cac943b`, against the old public baseline `36843e8cebeded1328dd7005ed668300a606f182`. The later normal parent synchronization and the strict-lint syntax-only fix are not separately rebenchmarked, so these numbers describe that frozen local content, not the pushed head; no remote CI or performance result is implied by this local measurement. This PR now implements the selected C direction and native A is unchanged: the native direct path remains the control and is not replaced. No query-speedup, Flight/IPC-network, storage, service or integration conclusion is drawn. Fixture: six synthetic raw-series Frame-boundary cases in two sequential campaigns: 100000 series in the first campaign, 1000 series in the second, one chunk per series in both. CPU affinity 2, seven repetitions per group, default warmup on, plain-timing and allocator-instrumented runs executed as separate passes (plain first, then alloc), no retries, no outlier removal, no rescaling and no selective repeats. The baseline and candidate binaries were built with the same release profile: `rustc 1.98.1 (48a229cea 2026-09-01)`, `debug=0`, `opt-level=3`, `lto=false`, `codegen-units=16`, `incremental=0`. Measured roles. A0 = native direct path, baseline build; A1 = native direct path, candidate build; B = the baseline single-row pack→unpack round trip (one frame per union row); C = the selected multirow pack→unpack round trip (one measured row cursor plus one bounded row builder, one batch per flush). The two native controls are listed separately and never pooled. C is the direction this PR now implements, and the bounded builder is why rows are held across chunk boundaries instead of materializing one frame per row. **Stage wall time (plain runner only).** Milliseconds, median [min–max]; MAD is the unscaled median absolute deviation in milliseconds. | Case | Series / rows per series | A0 native, baseline build | A1 native, candidate build | B baseline single-row round trip | C selected multirow round trip | |---|---:|---:|---:|---:|---:| | float2_key16 | 100000 / 2 | 42.313 [41.972–44.164]; 0.341 | 43.520 [42.146–45.728]; 1.133 | 2549.316 [2435.763–2618.753]; 52.979 | 206.888 [199.977–208.500]; 1.612 | | float3_key16 | 100000 / 3 | 59.434 [58.300–62.123]; 1.135 | 60.111 [59.460–60.870]; 0.400 | 3261.311 [3195.805–3309.910]; 10.551 | 260.933 [256.555–303.316]; 4.378 | | float3_key128 | 100000 / 3 | 64.968 [62.159–78.136]; 1.527 | 62.999 [62.748–65.338]; 0.251 | 3240.701 [3223.537–3381.599]; 11.899 | 291.846 [263.834–328.907]; 19.877 | | long_float1024_key16 | 1000 / 1024 | 7.556 [7.375–7.839]; 0.107 | 7.113 [6.939–7.539]; 0.051 | 26.311 [25.893–27.732]; 0.209 | 10.468 [9.849–10.832]; 0.299 | | histogram3_key16 | 1000 / 3 | 0.452 [0.438–0.490]; 0.005 | 0.448 [0.440–0.472]; 0.008 | 17.250 [17.137–18.415]; 0.113 | 5.632 [5.528–5.992]; 0.053 | | multifield_float_hist3_key16 | 1000 / 3 | 0.593 [0.588–0.632]; 0.005 | 0.588 [0.585–0.634]; 0.002 | 25.194 [24.633–26.472]; 0.561 | 6.328 [6.183–6.646]; 0.063 | Ratios are deliberately not tabulated: a B/C or C/A1 ratio over one boundary stage is not a query-level number. The A0 and A1 ranges overlap in every case, so the native controls establish no candidate-build difference. The C/B ranges are disjoint in this bounded seven-round sample, which is descriptive for these fixtures only, not significance or generalization evidence. **Actual frame and row counts.** Counted from the runs, not inferred from the name of the mode; rows are emitted frame rows, not sample positions. Both columns are invariant across all seven rounds and both runners; native A emits no frames. | Case | Series × chunks | B baseline single-row batches / rows | C selected multirow batches / rows | Logical positions | |---|---:|---:|---:|---:| | float2_key16 | 100000 × 1 | 100000 / 100000 | 391 / 100000 | 200000 | | float3_key16 | 100000 × 1 | 100000 / 100000 | 391 / 100000 | 300000 | | float3_key128 | 100000 × 1 | 100000 / 100000 | 391 / 100000 | 300000 | | long_float1024_key16 | 1000 × 1 | 1000 / 1000 | 17 / 1000 | 1024000 | | histogram3_key16 | 1000 × 1 | 1000 / 1000 | 4 / 1000 | 3000 | | multifield_float_hist3_key16 | 1000 × 1 | 1000 / 1000 | 4 / 1000 | 3000 | **Allocation (separate allocator-instrumented runner).** These counters come only from the instrumented runner; its timings are not used and the plain runner is not instrumented. Deallocation calls equal allocation calls in every run. "Requested" bytes are request sizes (the realloc column sums full `new_size`, not the delta) and are neither RSS nor allocator usable size. "Peak live Δ" is the peak live-byte delta relative to the meter baseline of that run, not whole-process peak RSS; a larger value means more phase-retained memory. | Case | Role | Allocation calls | Reallocation calls | Requested alloc B | Requested full-new realloc B | Peak live Δ B | |---|---:|---:|---:|---:|---:|---:| | float2_key16 | B baseline single-row | 55000553 | 400003 | 4640645622 | 198401920 | 40380 | | float2_key16 | C selected multirow | 3690875 | 14085 | 364600734 | 17429056 | 458978 | | float3_key16 | B baseline single-row | 68500694 | 400004 | 5597857156 | 198402320 | 46914 | | float3_key16 | C selected multirow | 4405874 | 16432 | 482697300 | 20582480 | 585488 | | float3_key128 | B baseline single-row | 68500694 | 400004 | 5631457268 | 198402320 | 47250 | | float3_key128 | C selected multirow | 4405874 | 16432 | 527497412 | 20582480 | 627432 | | long_float1024_key16 | B baseline single-row | 414412 | 4004 | 54141088 | 1986048 | 50438 | | long_float1024_key16 | C selected multirow | 32694 | 300 | 36651101 | 131648 | 2156848 | | histogram3_key16 | B baseline single-row | 381412 | 4006 | 39057104 | 1986084 | 36678 | | histogram3_key16 | C selected multirow | 96233 | 437 | 12366165 | 642724 | 1336504 | | multifield_float_hist3_key16 | B baseline single-row | 517553 | 4005 | 48814638 | 1985956 | 43188 | | multifield_float_hist3_key16 | C selected multirow | 103926 | 460 | 13534819 | 674852 | 1454810 | The native A controls allocate identically to each other in all six cases (equal counters and equal peak-live deltas; absolute peak-live bytes differ by one byte because the meter baseline itself differs by one byte). **Whole-process RSS (plain runner, lifetime scope).** Kilobytes, median [min–max], from the external time-based maximum resident set size of each measured run; it therefore includes startup, setup and output and is not phase-isolated memory. The native A0/A1 lifetime readings span 5280–6240 kB across these cases. | Case | B baseline single-row | C selected multirow | |---|---:|---:| | float2_key16 | 6080 [5920–6240] | 7520 [7360–7520] | | float3_key16 | 6240 [6080–6240] | 7520 [7520–7680] | | float3_key128 | 6240 [6080–6240] | 7680 [7520–7680] | | long_float1024_key16 | 6080 [5920–6080] | 9176 [9172–9336] | | histogram3_key16 | 6080 [5920–6240] | 8320 [8160–8480] | | multifield_float_hist3_key16 | 6080 [5920–6080] | 8640 [8480–8640] | **Repeatability and retention.** All 336 measured records are retained (168 per campaign): 6 cases × 4 roles × 2 runners × 7 repetitions = 48 groups of seven. All 168 instrumented runs pass cleanup — deallocations equal allocations, post-phase live bytes equal that run's baseline, reallocation failures 0 and underflow events 0 — and every allocation, deallocation and reallocation call count, requested byte total and peak-live reading is identical across the seven rounds of its group (MAD 0). Only the meter's live-baseline/cleanup value may differ by at most one byte between processes, and it is not claimed identical. Full logical digests, field kinds, structural and metadata checks, CPU affinity 2, per-run snapshots and exit statuses match the measured-scale reference on every record, and all 1064 retained raw artifacts verify against their recorded checksums. Nothing was rerun after the accepted run and no round was excluded. **Accepted tradeoff.** The measured cost of the selected C direction is accepted once, as part of that direction: relative to B, C raises phase peak-live bytes and whole-process RSS while lowering stage wall time and allocation request counts in these six fixtures (largest observed increase: the long-float case, peak live Δ 2156848 B versus 50438 B and lifetime RSS 9176 kB versus 6080 kB). This is a batching-versus-retained-memory tradeoff of this fixture, not a claim about general or steady-state service memory, not a veto of the direction and not an open design question. **Reading limits.** Per-run one-minute load spans 1.06–4.29 on a shared machine, and no host-idle, thermal or frequency-isolation claim is made; process snapshots are scoped to the sandbox PID namespace. Wall and CPU values are phase clocks; the external time tool's whole-process counters have coarse resolution and are not used as phase clocks. The histogram and multifield native stages are sub-millisecond, so no fine-grained accuracy claim is made at that magnitude. Every SOURCE_KEY group uses the identical explicit metric `raw_bench_metric{host="bench-0"}`, which is not evidence about real PromQL label diversity or cardinality; the histogram table template is built outside the measured phase and shared across series; `Samples::Mixed` within one field is not covered. Only fixture generation, the codec boundary, digests and phase-owned cleanup are inside the measured phase. This is a local Frame-phase measurement of these six synthetic cases — not a Flight/IPC network path, not a whole-query or query-regression benchmark, not a storage or service measurement, and not a CI result. <details> <summary>All seven rounds per role (plain wall and process CPU), fixture metadata with digests, and the acceptance checks behind the tables above</summary> Values are the verified local statistics, converted from nanosecond medians to milliseconds and rounded to three decimals; process CPU is `CLOCK_PROCESS_CPUTIME_ID`, not the external time tool's counters. Repetitions are listed in measurement order 1–7. | Case | Role | Metric | Median [min–max] | MAD | Reps 1–7 | |---|---|---|---:|---:|---| | float2_key16 | A0 native, baseline build | wall ms | 42.313 [41.972–44.164] | 0.341 | 43.135, 43.758, 44.164, 41.972, 42.056, 42.313, 42.191 | | float2_key16 | A0 native, baseline build | CPU ms | 42.318 [41.977–44.158] | 0.341 | 43.127, 43.754, 44.158, 41.977, 42.057, 42.318, 42.186 | | float2_key16 | A1 native, candidate build | wall ms | 43.520 [42.146–45.728] | 1.133 | 43.520, 42.618, 45.728, 42.146, 43.789, 42.387, 44.658 | | float2_key16 | A1 native, candidate build | CPU ms | 43.526 [42.152–45.731] | 1.128 | 43.526, 42.619, 45.731, 42.152, 43.782, 42.383, 44.654 | | float2_key16 | B baseline single-row | wall ms | 2549.316 [2435.763–2618.753] | 52.979 | 2567.299, 2549.316, 2465.980, 2496.337, 2435.763, 2618.753, 2559.681 | | float2_key16 | B baseline single-row | CPU ms | 2541.530 [2436.036–2618.982] | 44.918 | 2565.139, 2541.530, 2465.528, 2496.612, 2436.036, 2618.982, 2559.874 | | float2_key16 | C selected multirow | wall ms | 206.888 [199.977–208.500] | 1.612 | 207.428, 206.888, 199.977, 201.735, 206.998, 208.500, 203.026 | | float2_key16 | C selected multirow | CPU ms | 206.909 [200.005–208.528] | 1.618 | 207.458, 206.909, 200.005, 201.760, 207.015, 208.528, 203.033 | | float3_key16 | A0 native, baseline build | wall ms | 59.434 [58.300–62.123] | 1.135 | 59.434, 58.442, 61.186, 62.123, 58.300, 61.205, 58.781 | | float3_key16 | A0 native, baseline build | CPU ms | 59.438 [58.304–62.118] | 1.134 | 59.438, 58.443, 61.159, 62.118, 58.304, 61.201, 58.772 | | float3_key16 | A1 native, candidate build | wall ms | 60.111 [59.460–60.870] | 0.400 | 60.111, 60.056, 59.460, 60.870, 59.460, 60.168, 60.511 | | float3_key16 | A1 native, candidate build | CPU ms | 60.113 [59.466–60.868] | 0.399 | 60.113, 60.056, 59.468, 60.868, 59.466, 60.171, 60.512 | | float3_key16 | B baseline single-row | wall ms | 3261.311 [3195.805–3309.910] | 10.551 | 3260.350, 3309.910, 3271.862, 3195.805, 3211.174, 3261.550, 3261.311 | | float3_key16 | B baseline single-row | CPU ms | 3261.623 [3196.160–3310.088] | 10.600 | 3260.683, 3310.088, 3272.224, 3196.160, 3211.509, 3261.875, 3261.623 | | float3_key16 | C selected multirow | wall ms | 260.933 [256.555–303.316] | 4.378 | 303.316, 275.333, 257.711, 258.074, 260.933, 256.555, 299.972 | | float3_key16 | C selected multirow | CPU ms | 260.961 [256.565–301.641] | 4.395 | 301.641, 275.320, 257.731, 258.109, 260.961, 256.565, 299.980 | | float3_key128 | A0 native, baseline build | wall ms | 64.968 [62.159–78.136] | 1.527 | 62.159, 78.136, 64.089, 66.290, 64.968, 66.937, 63.440 | | float3_key128 | A0 native, baseline build | CPU ms | 64.965 [62.156–78.113] | 1.334 | 62.156, 78.113, 64.092, 66.299, 64.965, 65.967, 63.431 | | float3_key128 | A1 native, candidate build | wall ms | 62.999 [62.748–65.338] | 0.251 | 64.904, 65.338, 62.766, 62.914, 62.999, 63.280, 62.748 | | float3_key128 | A1 native, candidate build | CPU ms | 62.999 [62.744–65.343] | 0.255 | 64.913, 65.343, 62.755, 62.914, 62.999, 63.289, 62.744 | | float3_key128 | B baseline single-row | wall ms | 3240.701 [3223.537–3381.599] | 11.899 | 3223.537, 3228.801, 3348.994, 3229.554, 3244.801, 3240.701, 3381.599 | | float3_key128 | B baseline single-row | CPU ms | 3240.996 [3223.913–3381.298] | 15.381 | 3223.913, 3225.615, 3349.367, 3229.918, 3245.152, 3240.996, 3381.298 | | float3_key128 | C selected multirow | wall ms | 291.846 [263.834–328.907] | 19.877 | 291.846, 328.907, 311.723, 263.834, 264.137, 288.462, 309.855 | | float3_key128 | C selected multirow | CPU ms | 291.848 [263.853–328.925] | 19.915 | 291.848, 328.925, 311.763, 263.853, 264.152, 288.477, 309.890 | | long_float1024_key16 | A0 native, baseline build | wall ms | 7.556 [7.375–7.839] | 0.107 | 7.657, 7.827, 7.449, 7.375, 7.839, 7.476, 7.556 | | long_float1024_key16 | A0 native, baseline build | CPU ms | 7.556 [7.371–7.832] | 0.107 | 7.647, 7.828, 7.449, 7.371, 7.832, 7.477, 7.556 | | long_float1024_key16 | A1 native, candidate build | wall ms | 7.113 [6.939–7.539] | 0.051 | 7.164, 7.539, 7.067, 6.939, 7.127, 6.997, 7.113 | | long_float1024_key16 | A1 native, candidate build | CPU ms | 7.114 [6.940–7.533] | 0.047 | 7.160, 7.533, 7.067, 6.940, 7.116, 6.998, 7.114 | | long_float1024_key16 | B baseline single-row | wall ms | 26.311 [25.893–27.732] | 0.209 | 26.311, 26.520, 26.225, 25.893, 27.732, 26.106, 26.924 | | long_float1024_key16 | B baseline single-row | CPU ms | 26.304 [25.895–27.729] | 0.219 | 26.304, 26.523, 26.223, 25.895, 27.729, 26.101, 26.927 | | long_float1024_key16 | C selected multirow | wall ms | 10.468 [9.849–10.832] | 0.299 | 10.169, 10.468, 10.542, 9.849, 10.540, 9.987, 10.832 | | long_float1024_key16 | C selected multirow | CPU ms | 10.469 [9.850–10.833] | 0.309 | 10.160, 10.469, 10.517, 9.850, 10.541, 9.988, 10.833 | | histogram3_key16 | A0 native, baseline build | wall ms | 0.452 [0.438–0.490] | 0.005 | 0.447, 0.475, 0.453, 0.438, 0.452, 0.451, 0.490 | | histogram3_key16 | A0 native, baseline build | CPU ms | 0.451 [0.438–0.490] | 0.005 | 0.447, 0.475, 0.453, 0.438, 0.451, 0.451, 0.490 | | histogram3_key16 | A1 native, candidate build | wall ms | 0.448 [0.440–0.472] | 0.008 | 0.465, 0.471, 0.444, 0.447, 0.440, 0.448, 0.472 | | histogram3_key16 | A1 native, candidate build | CPU ms | 0.448 [0.440–0.472] | 0.008 | 0.464, 0.471, 0.444, 0.447, 0.440, 0.448, 0.472 | | histogram3_key16 | B baseline single-row | wall ms | 17.250 [17.137–18.415] | 0.113 | 17.250, 18.324, 17.216, 17.137, 17.239, 17.619, 18.415 | | histogram3_key16 | B baseline single-row | CPU ms | 17.235 [17.132–18.389] | 0.103 | 17.235, 18.302, 17.197, 17.132, 17.223, 17.614, 18.389 | | histogram3_key16 | C selected multirow | wall ms | 5.632 [5.528–5.992] | 0.053 | 5.597, 5.986, 5.632, 5.528, 5.648, 5.579, 5.992 | | histogram3_key16 | C selected multirow | CPU ms | 5.633 [5.528–5.992] | 0.053 | 5.598, 5.986, 5.633, 5.528, 5.643, 5.580, 5.992 | | multifield_float_hist3_key16 | A0 native, baseline build | wall ms | 0.593 [0.588–0.632] | 0.005 | 0.588, 0.603, 0.590, 0.603, 0.593, 0.590, 0.632 | | multifield_float_hist3_key16 | A0 native, baseline build | CPU ms | 0.593 [0.588–0.631] | 0.003 | 0.588, 0.603, 0.589, 0.593, 0.593, 0.590, 0.631 | | multifield_float_hist3_key16 | A1 native, candidate build | wall ms | 0.588 [0.585–0.634] | 0.002 | 0.586, 0.605, 0.585, 0.617, 0.585, 0.588, 0.634 | | multifield_float_hist3_key16 | A1 native, candidate build | CPU ms | 0.587 [0.585–0.633] | 0.002 | 0.586, 0.605, 0.585, 0.616, 0.585, 0.587, 0.633 | | multifield_float_hist3_key16 | B baseline single-row | wall ms | 25.194 [24.633–26.472] | 0.561 | 24.676, 26.472, 24.633, 25.800, 24.725, 25.194, 26.190 | | multifield_float_hist3_key16 | B baseline single-row | CPU ms | 25.182 [24.619–26.451] | 0.564 | 24.660, 26.451, 24.619, 25.771, 24.720, 25.182, 26.178 | | multifield_float_hist3_key16 | C selected multirow | wall ms | 6.328 [6.183–6.646] | 0.063 | 6.308, 6.590, 6.183, 6.646, 6.318, 6.328, 6.391 | | multifield_float_hist3_key16 | C selected multirow | CPU ms | 6.329 [6.184–6.646] | 0.063 | 6.309, 6.590, 6.184, 6.646, 6.319, 6.329, 6.392 | The same seven-round records give the allocation state counts used above; in every group they are identical across all seven repetitions (MAD 0). Two examples at the two size extremes, allocator-instrumented runner: | Case | Role | Allocation calls | Deallocation calls | Reallocation calls | Requested alloc B | Requested full-new realloc B | Peak live bytes | Post-cleanup live B | |---|---|---:|---:|---:|---:|---:|---:|---:| | float2_key16 | B baseline single-row | 55000553 (7/7) | 55000553 (7/7) | 400003 (7/7) | 4640645622 (7/7) | 198401920 (7/7) | 85793 | 45413 | | float2_key16 | C selected multirow | 3690875 (7/7) | 3690875 (7/7) | 14085 (7/7) | 364600734 (7/7) | 17429056 (7/7) | 504392 | 45414 | | long_float1024_key16 | B baseline single-row | 414412 (7/7) | 414412 (7/7) | 4004 (7/7) | 54141088 (7/7) | 1986048 (7/7) | 95825 | 45387 | | long_float1024_key16 | C selected multirow | 32694 (7/7) | 32694 (7/7) | 300 (7/7) | 36651101 (7/7) | 131648 (7/7) | 2202236 | 45388 | The remaining four cases follow the same pattern: every allocator counter, requested-byte total and peak-live reading equals its median in all seven repetitions of its group, reallocation failures and underflow events are 0, and post-cleanup live bytes equal that run's meter baseline. Only that live-baseline/cleanup value moves (by at most one byte) between processes. Fixture metadata and logical digests (each digest is identical across all 56 measured records of its case; native A carries no frames): | Case | Series | Chunks per series | Rows per series | Key len | Logical positions | Logical digest | |---|---:|---:|---:|---:|---:|---| | float2_key16 | 100000 | 1 | 2 | 16 | 200000 | `0xdbeb81be394ac1d3` | | float3_key16 | 100000 | 1 | 3 | 16 | 300000 | `0x731b981da38db277` | | float3_key128 | 100000 | 1 | 3 | 128 | 300000 | `0x3be5490405031e57` | | long_float1024_key16 | 1000 | 1 | 1024 | 16 | 1024000 | `0x9f1da6c041555235` | | histogram3_key16 | 1000 | 1 | 3 | 16 | 3000 | `0xaf874f7678af1d82` | | multifield_float_hist3_key16 | 1000 | 1 | 3 | 16 | 3000 | `0x79761914759b7b7c` | Acceptance checks behind the tables above: 336/336 measured records forming the complete 6 cases × 4 roles × 2 runners × 7 repetitions matrix; 48 groups of seven repetitions; per-record logical, field-kind, structural and metadata agreement with the measured-scale reference; CPU affinity 2 and exit status 0 on every record; 168/168 instrumented runs with clean allocation accounting; 1064/1064 retained raw artifacts verifying against their recorded checksums. No rerun, no exclusion, and allocator-instrumented timings were not substituted for the plain-timing statistics. </details> ### Verification Pushed head: `5f0890ebeaa5a8252f77dc2d18d8456539714fc4`, normally synchronized with #9607 at `ba88f817cdcffeae6594943c55aaced2ffa769e5`. The three stream implementation/test files are byte-identical to the measured candidate; subsequent changes are parent synchronization and the finite codec's Clippy-only guard spelling. Checks on this exact head: - `cargo +1.98.1 check --locked -p datatypes -p store-api -p common-query --all-targets` — passed. - `cargo +1.98.1 clippy --locked -p datatypes -p store-api -p common-query --all-targets --all-features -- -D warnings` — passed with the official pinned 1.98.1 Clippy component. - `cargo +1.98.1 nextest run --locked -p datatypes -p store-api -p common-query --retries 0 --no-fail-fast` — **570 passed, 0 skipped**. - `cargo +1.98.1 test --locked -p datatypes --doc` — **2 passed**, including the `RawTimestamps` compile-fail contract and public `Samples::segments` example. - Rust 1.99 formatting, TOML/license/typo checks, `git diff --check`, and normal signed-off commit/push hooks passed. Source hashes were verified unchanged after the hooks. The local measurements above remain attributed to their frozen candidate rather than relabelled as a rebenchmark of this head. Current-head remote CI completed successfully in all three workflows below. The PR remains open and unmerged until the normal merge gates and parent-first integration are checked. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ### Verified current-head remote CI Exact pushed head: `5f0890ebeaa5a8252f77dc2d18d8456539714fc4`. - [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882957553) — **success**. - [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882957587) — **success**. - [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882957636) — **success**. Linux and Windows test jobs, strict Clippy, builds, SQLness and the executed integration/fuzz jobs succeeded. Coverage and recent-release compatibility jobs were skipped by their workflow conditions; they are not claimed as executed. These CI results are independent of the frozen local experiment provenance above. |
||
|
discord9
|
7e0f6e29aa |
feat(datatypes): add finite raw series frame codec (#9607)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Depends on: #9603 (`feat/narthex-raw-contract`). - Base of: #9611 (`feat/narthex-raw-frame-stream`). - The finite raw SeriesFrame codec remains independent of the Mito producer in #9608. ## What's changed and what's your intention? Adds an Arrow `RecordBatch` representation for the raw series chunks defined in the stack, so subsequent execution and transport code can exchange typed samples without rebuilding every sample, plus the bounded multirow encoder that packs raw series chunks into multi-row batches directly. **Codec.** The new `series::frame` module provides raw-frame schema construction, chunk packing and whole-batch unpacking, in encoded-key and explicit-label modes, preserving explicit timestamps, float/histogram/mixed payloads, nulls, field order and `is_last`. Different fields may have different chunk boundaries: packing splits at their union and retains Arrow views rather than densifying the inputs. `Samples::segments()` exposes the zero-copy dense segment view the codec walks, with a public-call doctest, so the codec can consume existing dense segments without a new segment type or allocation. Decode validates schema metadata, payload lengths, indices, histogram structure and timestamps, and any malformed row fails the entire unpacking call rather than returning a successful prefix. **Bounded multirow encoding.** `RawSeriesFrameBuilder` buffers rows and emits one batch when either bound is reached (`DEFAULT_MAX_ROWS` 256, `DEFAULT_MAX_BYTES` 1 MiB); the effective byte cap is stored as `min(requested, i32::MAX)`, which keeps every cumulative Arrow offset domain representable for any admitted group. `RawSeriesChunkCursor` walks one already validated chunk at the union of the field segment boundaries, holds at most one pending measured row, and measures each row exactly once; the pending row is retried after a flush, so no row is dropped or split, and per-field offsets advance against each field's original segment length, which keeps nonzero-offset (sliced) sample arrays correct. The byte measure dominates the encoder's offset domains: key and timestamp bytes, list element counts and bitmap bytes, and the visible extent of each histogram table list column read from the rebased `value_offsets` view rather than from retained capacity. It is an internal batching work estimate — not an exact IPC size and not a resident-memory or RSS cap — so one individually legal row that exceeds the requested cap is still accepted and emitted alone, because that slice cannot be split into smaller legal rows. **Histogram layout and sharing.** Row-local histogram indices are never remapped; physical tables are concatenated under cumulative row list offsets, so each row still addresses its own table through a nonzero slice of the child. A row group with a single Arrow constituent reuses it by clone, which shares buffers and keeps the single-row zero-copy and owner-drop behavior; a group with several constituents is concatenated, which copies. Both cases are asserted. The `Samples::into_dense` sharing check compares the Arrow table view (`ArrayData` identity) in addition to `Arc` identity, and the first histogram batch's `ArrayData` is computed lazily and cached: that is pointer-identity caching, not content deduplication, and distinct buffers with equal content are never merged. Local buffer sharing does not prevent a full table from being encoded in every batch; any future trimming or remapping belongs at the actual IPC/network/spill boundary, reusing Arrow slicing and selection and the existing writer, and preserving duplicate indices without content hashing. **Preserved interfaces.** `pack_raw_series_chunk` keeps its `Vec<RecordBatch>` return type and `unpack_raw_series_frame` keeps its signature; `RawSeriesFrameDecoder` prepares the expected frame schema once for a stream of batches, while the wrapper still builds a decoder per call. The raw contract's `RawTimestamps` validates strictly increasing source timestamps once at construction behind a private buffer. The frozen `v1` contract is untouched: the four reserved metadata entries, the column names (`read_key`, `is_last`, `time`, `fields`, plus `name`/`labels` in explicit mode) and the wire column literal `read_key` stay as they were (`primary_key` is only the Rust-side name of that buffer). Histogram-only rows decode through `Samples::try_from_histogram` directly, without a duplicate bounds check. **Tests.** The frame tests cover schema and metadata acceptance and rejection, bit-exact payloads (including NaN payloads and `-0.0`), sliced-payload owner lifetimes, row and byte bounds (bounds taken from real measured rows, flush-before-exceed with pending retry, legal oversized row), row-local histogram indices and complete tables with a real Arrow IPC round trip, decoder schema caching and label-mode separation. The `samples` tests pin both sides of the sharing rule: distinct wrappers of one Arrow table view are reused, while distinct buffers, distinct table slices, and shared value buffers with different null-bitmap offsets are never merged. **Limitations and boundaries.** Each histogram-bearing row carries its complete histogram table, including entries no index references, and a large table referenced by several segments can be repeated per row; there is no deduplication and no global reindex, so an encoded batch can exceed the source sample bytes. These entry points are exercised by unit tests, and the builder, cursor and decoder are also used by the in-process stream adapter of #9611, which is not yet wired into a production query path; there is no transport, persistence or query-performance change. This PR handles finite raw frames only: it does not add stream-wide key/order/EOF validation (that is #9611), evaluated frames, a Flight/IPC execution path, logical-name authentication, a native query consumer, a production query-path switch, distributed or cross-region merging, or a prepared producer. The frame preserves ordered logical field names and rejects mismatched supplied schemas; that is not authentication of label binding or data provenance. Existing persisted storage formats and query entrypoints are unchanged, and this is not a cross-version wire-compatibility claim. Open and unmerged; current-head CI has completed successfully. ### Bounded local Frame-boundary measurement (B baseline single-row, C selected multirow) Provenance: measured at the frozen local candidate `eddd595b82888452b3c066678fabc6c1200c74f7`, whose bounded-multirow finite codec content is `3406a5334574a518924c67c319e2d92b6cac943b`, against the old public baseline `36843e8cebeded1328dd7005ed668300a606f182`. The later normal parent synchronization and the strict-lint syntax-only fix are not separately rebenchmarked, so these numbers describe that frozen local content, not the pushed head; no remote CI or performance result is implied by this local measurement. This PR now implements the selected C direction and native A is unchanged: the native direct path remains the control and is not replaced. A bounded local-only experiment measured this codec's role as the pack/unpack kernel of that direction. Fixture: six synthetic raw-series Frame-boundary cases in two sequential campaigns (100000 series in the first, 1000 in the second, one chunk per series in both), CPU affinity 2, seven repetitions, default warmup, plain-timing and allocator-instrumented runs kept separate, no retries and no excluded rounds. Stage wall time from the plain runner, milliseconds, median only, for the six cases in table order: | Case | Series / rows per series | A0 native, baseline build | A1 native, candidate build | B baseline single-row | C selected multirow | |---|---:|---:|---:|---:|---:| | float2_key16 | 100000 / 2 | 42.313 | 43.520 | 2549.316 | 206.888 | | float3_key16 | 100000 / 3 | 59.434 | 60.111 | 3261.311 | 260.933 | | float3_key128 | 100000 / 3 | 64.968 | 62.999 | 3240.701 | 291.846 | | long_float1024_key16 | 1000 / 1024 | 7.556 | 7.113 | 26.311 | 10.468 | | histogram3_key16 | 1000 / 3 | 0.452 | 0.448 | 17.250 | 5.632 | | multifield_float_hist3_key16 | 1000 / 3 | 0.593 | 0.588 | 25.194 | 6.328 | The same run's full min–max, MAD, all seven rounds per role, allocation counters, RSS, frame counts and acceptance checks are in #9611, which carries the canonical full results — including the frame totals counted from the runs (B emitted 100000 / 100000 / 100000 / 1000 / 1000 / 1000 batches and C 391 / 391 / 391 / 17 / 4 / 4, with frame rows equal to series × chunks in every case and no frames on the native path). Reading: the A0 and A1 native control ranges overlap in every case, so the native controls establish no candidate-build difference, and the C/B range disjointness in this bounded seven-round sample is descriptive for these synthetic fixtures only. Relative to B, C lowers stage wall time and allocation request counts while raising phase peak-live bytes and whole-process RSS; that batching-versus-retained-memory tradeoff of the selected direction is accepted once, as part of that direction, and is not a veto or an open design question (largest observed increase: the long-float case, peak live Δ 2156848 B versus 50438 B and lifetime RSS 9176 kB versus 6080 kB). Scope limits, as in #9611: every SOURCE_KEY group uses one identical explicit metric label set, so this is not real PromQL label diversity or cardinality evidence; the histogram table template is built outside the measured phase; no `Samples::Mixed` inside one field. The measured scope is local Frame generation, the codec boundary, digests and phase-owned cleanup — not a Flight/IPC network path, not a whole-query or query-regression benchmark, not a storage or service measurement, and not a CI result. No query speedup is claimed from this codec or from the selected direction. ### Verification Pushed head: `ba88f817cdcffeae6594943c55aaced2ffa769e5`, synchronized normally with #9603 at `794389688084ee97a7760d1ae993bcf1c109489e`. Checks on the exact frozen source recorded by this head: - `cargo +1.98.1 check --locked -p datatypes --all-targets` — passed. - `cargo +1.98.1 clippy --locked -p datatypes --all-targets --all-features -- -D warnings` — passed with the official pinned 1.98.1 Clippy component. - `cargo +1.98.1 nextest run --locked -p datatypes --retries 0 --no-fail-fast` — **363 passed, 0 skipped**. - `cargo +1.98.1 test --locked -p datatypes --doc series::samples::Samples::segments` — **1 passed**. - Rust 1.99 formatting, TOML/license/typo checks, `git diff --check`, and normal signed-off commit/push hooks passed. Source hashes were verified unchanged after the hooks. The final closeout only collapses the full-builder flush guard into a short-circuit-preserving let-chain for strict Clippy; it changes no batching behavior or assertion. The local measurements above remain attributed to their frozen candidate. Current-head remote CI completed successfully in all three workflows below. The PR remains open and unmerged until the normal merge gates and parent-first integration are checked. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ### Verified current-head remote CI Exact pushed head: `ba88f817cdcffeae6594943c55aaced2ffa769e5`. - [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922544) — **success**. - [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922504) — **success**. - [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38882922665) — **success**. Linux and Windows test jobs, strict Clippy, builds, SQLness and the executed integration/fuzz jobs succeeded. Coverage and recent-release compatibility jobs were skipped by their workflow conditions; they are not claimed as executed. These CI results are independent of the frozen local experiment provenance above. |
||
|
discord9
|
6468d87784 |
chore: synchronize merged raw series contract parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
3fd69e312e |
feat(storage): define raw series and optional scanner contract (#9603)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) - Related: #9608, which builds on this PR's branch and adds the engine-side raw-series producer; the contract reviewed here is this PR's own diff. - Base branch: `narthex`, the shared integration branch rather than `main` (it already contains the merged Narthex PRs #9598, #9599, and #9602). ## What's changed and what's your intention? Define the raw-series read contract shared by storage engines and their consumers, and separate "which series are read" from "how a reader streams them". The contract lands on its own so its vocabulary is reviewed once; the engine-side producer arrives in a follow-up PR. ### Raw series values and streams - `RawSeriesChunk` is one raw source batch: the complete encoded `primary_key` (not a TSID or hash alone), an optional `metric`, a strictly increasing `time` axis, fields aligned to that axis, and `is_last`. `try_new` rejects an evaluation grid, mismatched field count or length, and an empty time axis; validation is per chunk and does not claim ordering between chunks. - `RawTimestamps` keeps its buffer private, so "strictly increasing, no duplicates" is validated once at construction and cannot be bypassed by a struct literal (pinned by a `compile_fail` doctest). Slices are zero-copy and keep the underlying buffer alive. - `RawSeriesStream` is the reader side: a `Stream` of `Result<RawSeriesChunk, BoxedError>` items exposing `schema`, `label_source`, and `ordering`. `SendableRawSeriesStream` is its pinned boxed form. - `RawLabelSource` carries the binding: `PrimaryKey` means labels are encoded in the complete primary key and decoded with scan-bound metadata; `Explicit` means the stream supplies the complete name/label set itself, never a partial projection. Chunks under `PrimaryKey` carry `metric: None` because labels are undecoded, not because the metric name is missing, while `Some(metric)` is always complete. - `PreparedSeriesSource` is the selection side and is deliberately not a reader: `num_series()`, `series_info(ordinal)`, `label_source()`, and `read_series(range)`. Both it and `RawSeriesStream` return the same `label_source()`, so a holder can interpret labels before polling anything. ### Shared ordering vocabulary - `datatypes::series::SeriesOrdering` is a common vocabulary that raw and evaluated series can express; in this PR the only user is `RawSeriesStream::ordering`, and no evaluated consumer or implementation is added. - Modes: `Interleaved` (a series may resume after another series), `Contiguous` (each series' chunks are contiguous), and `KeyThenTime` (this stream's series identity, then position). Key order uses the complete bound-compatible identity, not a TSID or hash alone. - All modes are per-stream guarantees only: positions strictly increase within a series, including across chunks, with no duplicates, and there is no query-global ordering or finality claim. `is_last` marks the last chunk of that series in this one stream; merge code must recompute finalization across all related inputs. ### Prepared series selection - A source describes a finite, stable selected set: `num_series()` is exact, `series_info(ordinal)` returns identity metadata without polling samples, `label_source()` is available without opening a reader, and `read_series(range)` covers a half-open ordinal range. - Ordinals are stable local indexes, not protocol identifiers, and each ordinal's identity matches the chunks produced for it. Readers are independent: repeated or overlapping reads do not consume shared iteration state, dropping one reader does not cancel siblings, and a reader owns what it needs to continue after the source is dropped. - Invalid ordinals and ranges return errors; an empty in-bounds range is an ordinary empty stream. `SeriesInfo::metric` follows chunk semantics (`None` = primary-key label binding, `Some` = complete). - The documented requirements — no sample payload loading or hidden blocking I/O during preparation, identity lookup, or reader creation, and binding to the partition's immutable MVCC view, predicates, authorization, projection, and labels — are requirements for future implementations. - This PR adds no engine implementation, so it makes no I/O, performance, or MVCC behavior claim. - The store-api tests include a small in-memory `PreparedSeriesSource` that pins these guarantees: binding available before readers, empty ranges, invalid ordinals and ranges, independent overlapping readers across source and reader drops, and a selection that does not change when a later mutable selection does. ### Scanner entrypoints - `RegionScanner::prepare_series_partition` is the optional capability and defaults to `Ok(None)`, meaning "unsupported before execution, so the caller may use the legacy record-batch scan". No scanner overrides it in this PR, and the only `PreparedSeriesSource` implementation here is the store-api test fixture. - `RegionScanner::scan_series_partition` keeps its existing default panic (`todo!()`), now documented as must-override; a store-api test pins that panic and shows the legacy `scan_partition` result is unchanged. - Callers must use exactly one entrypoint per scanner partition and execution, and must not replay a failed raw scan through the legacy path. ### Compatibility The legacy record-batch scan and existing queries are unchanged, and nothing in planning or execution calls the new entrypoints yet. No persisted format, wire protocol, storage format, or configuration change is made: the added types are in-memory Rust types with no serialized representation. Within this stack the raw-series types are unreleased, and the branch replaces the interim raw-only ordering enum with the shared `SeriesOrdering` plus the source-level label binding. For released code the change is additive: new trait methods with default implementations, and no released method signature changes. Note for reviewers: the malformed-range case is built with `std::ops::Range { start: 2, end: 1 }` rather than a reversed range literal, because `clippy::reversed_empty_ranges` rejects the literal form; the assertion itself is unchanged and no lint allow is added. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). ### Verification Local checks below ran on the exact source now pushed to this PR, including the test-only range fix noted above. - `cargo +1.98.1 check --locked -p datatypes -p store-api --all-targets` — passed. - `cargo nextest run --locked -p datatypes -p store-api --retries 0 --no-fail-fast` — 432 passed, 0 skipped. - `cargo test --locked -p datatypes --doc RawTimestamps` — 1 passed. - `cargo clippy --locked -p datatypes -p store-api --all-targets --all-features -- -D warnings` — passed on the pinned 1.98.1 toolchain, covering the clippy lint that the fixed test file previously tripped. - Exact-head remote CI for `794389688084ee97a7760d1ae993bcf1c109489e` completed successfully: [Rust CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322418) (including workspace Clippy, Linux tests, Windows build/tests and riscv64 build), [Checks](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322439), and [Integration CI](https://github.com/GreptimeTeam/greptimedb/actions/runs/38878322830) (SQL, SDK, import/export and fuzz jobs). The workflow marked coverage and recent-release compatibility jobs skipped; those two jobs are not claimed as executed. |
||
|
discord9
|
ac5b74d81f |
fix(mito2): retain typed filter regression without indexed loop lint
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
5f0890ebea |
chore: synchronize clippy-clean frame codec parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
ba88f817cd |
fix(datatypes): keep frame flush guard clippy clean
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
9871adc90c |
chore: synchronize checked multirow frame parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1d710776c0 |
chore: synchronize checked raw series contract parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
7943896880 |
fix(store-api): retain reversed range test without clippy lint
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
eddd595b82 |
perf(query): batch raw series across stream chunks
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
928284bea8 | Merge branch 'fix/narthex-frame-source-followup-20261016' into fix/narthex-9607-9611-batching-20261016 | ||
|
discord9
|
3406a53345 |
perf(datatypes): encode bounded multirow series frames
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
2c04075baa |
chore: synchronize shared series contract parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> # Conflicts: # src/datatypes/src/series/raw.rs # src/store-api/src/region_engine.rs # src/store-api/src/region_engine/prepared_series_tests.rs |
||
|
discord9
|
d7fccdff18 |
chore: synchronize shared series contract parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
3ed6b5df67 |
fix(mito2): adapt raw readers to shared series contracts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
d018031823 |
feat(datatypes): share series ordering and expose source label binding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1fc0c1c7d6 |
feat(storage): separate prepared series selection from raw readers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
b0af8b3eaf |
feat(datatypes): share series ordering and expose source label binding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
650ee58c22 |
feat(datatypes): share series ordering and expose source label binding
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
1284c043e5 |
feat(storage): separate prepared series selection from raw readers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
cfc777f55c |
feat(storage): separate prepared series selection from raw readers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
d639f1569e |
feat(storage): separate prepared series selection from raw readers
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
36843e8ceb |
chore: synchronize raw frame stream parent
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
a7ca3cce4e |
fix(query): preserve raw time axis lifecycle proof across contract update
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
35ba224c18 |
chore: synchronize reviewed raw time axis contract
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
24ebc2e030 |
fix(mito2): synchronize reviewed raw scan contract
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4053026374 |
fix(mito2): share raw key buffers and validate legacy key order
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
5fa4fed582 |
fix(datatypes): share raw axis validation and expose explicit buffer
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
0428b4217d |
fix(query): verify logical frame field bindings
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
d4a3f67f65 |
fix(datatypes): preserve series frame field names
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
d47d28da2b |
style(mito): import raw series types by name
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
4ad0ce71fe |
fix(store-api): mark default raw series scan as todo
Leave the raw entrypoint explicitly unimplemented until a scanner overrides it. Document the default panic and keep the real scanner regression proving that its legacy input remains readable after the placeholder is caught. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
b774c32075 |
fix(query): bound raw frame validation history and empty polling
Keep only adjacent sequence state instead of retaining every completed key. The producer remains responsible for contiguous key runs; ordered streams still reject key regressions. Yield after 32 validated empty batches per unpack poll without losing pending rows or terminal errors. Preserve whole-batch atomicity and document the producer contract. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
cfb66f5f2e |
chore: synchronize raw frame stream stack base
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
db702dc894 |
chore: synchronize Mito raw series stack base
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
489c9453cd |
chore: synchronize series frame stack base
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
380fde19df |
chore: synchronize merged histogram foundations
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
9e2991f098 |
feat(query): bridge native histograms to columnar samples (#9602)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Target: `narthex`, which now contains merged #9598 and #9599. Their landed changes have been synchronized without rewriting feature history, so this PR remains the histogram bridge increment. ## What's changed and what's your intention? Connect the existing native-histogram representation to the shared columnar sample containers. The incremental diff is limited to `common-query/src/native_histogram.rs`. - Re-export the shared `CounterResetHint` at its existing query-module path, preserving numeric conversions and unknown values. Add an owned-histogram adapter to `HistogramBatchBuilder`. - Add a quantile entrypoint for borrowed `HistogramView` values. It shares the existing quantile calculations and preserves early-return precedence, custom/exponential bucket behavior and NaN information. Borrowed-view validation does not tighten the legacy owned histogram's malformed-layout behavior. - Normalize persisted Arrow histogram structs directly into an owning `Samples` value, without reconstructing a per-row owned `NativeHistogram`. Preserve row positions, parent nulls, complete payload bits, both bucket sides, reset/start metadata, and sliced-parent offsets. Float payloads take precedence when present; stored integer bucket counts are already absolute and are converted directly, with a missing zero count defaulting to zero. - Reject missing required scalar values, null list items and invalid span layouts. The existing `read_histogram` decoder and persisted struct schema remain unchanged. Normalization borrows source values while reading and then copies them into the canonical owning batch; integer conversion uses temporary buffers. The borrowed quantile path still derives bucket/index buffers. This is not an allocation-free or measured-performance claim. Scanner, frame/Flight, execution-operator and public HTTP/TQL/SQL integration are not included, and no persisted or network format changes. ### Validation Ten new focused tests cover borrowed quantiles and NaN information, malformed/overflow spans and early-return precedence, legacy owned-layout compatibility, complete payload bit preservation, nonzero batch/parent slices after original owners drop, empty/all-null input, integer counts and float-family selection, and required-field/list-item/span errors. - Rust 1.98.1: `cargo check --locked -p datatypes -p common-query -p promql --all-targets` — passed. - Rust 1.98.1: `cargo nextest run --locked -p datatypes -p common-query -p promql --retries 0 --no-fail-fast` — 707 passed, 0 skipped, both before and after the final test-noise cleanup. The actual listing includes all new tests and the parent buffer-capacity regression. - The three cleanup-affected payload/lifecycle/empty tests were also run explicitly — 3 passed; the complete unfiltered suite then passed all 707. - Rust 1.99.0: `rustfmt --edition 2024 --config skip_children=true --check src/common/query/src/native_histogram.rs` — passed. - Scoped `hawkeye format` and `hawkeye check` on a byte-identical source copy, plus `git diff --check` — passed. A missing-header negative control confirmed that the scoped license check inspects the file. - Rust 1.99.0: `cargo clippy --locked --no-deps -p common-query -p promql --all-targets -- -D warnings` — blocked by `async_trait`-generated `clippy::double_must_use` at unchanged `common-query/src/logical_plan.rs:129`. The exact same command on the exact parent commit `7e5acfb9a9c` reproduces that failure. Pinned 1.98.1 Clippy is unavailable in the local read-only toolchain. No lint or CI policy was weakened; this is not a passing Clippy or full-workspace validation claim. ### Diff-scoped cleanup No whole tests were deleted. Removed only an empty-result assertion that accepted every dense-storage representation after exact length was already verified; the same named test retains empty/two-null conversion, length and missing-position checks. Simplified payload assertions to explicit histogram bindings and a shared complete bitwise assertion helper, and renamed the payload test to describe its explicit expected-value oracle. Public API rustdoc, ownership rationale and compatibility/precedence contracts remain. No scanner, protocol or unrelated cleanup is bundled. ### Verified review follow-up - Current tested/pushed head: `1164877b5a5982d0cea2860e01e1c28714299e9e`, including the merged #9599 parent. The only query-file conflict resolution relocates the same enum re-export and retains the exact landed reset-hint identity regression; no bridge/helper code was dropped. Independently reviewed after resolution. - Owned `all_buckets` and borrowed-view quantile execution now share one private `assemble_bucket_sides`; no duplicate negative reversal or zero-bucket rule remains. Fallible side builders, validation and custom-bound handling are unchanged. - A focused expected-sequence regression independently checks bounds, counts and boundary rules, including insertion for nonzero count and omission for both signed zeros. - On the final synchronized head, Rust 1.98.1 all-target check and complete `common-query / promql / datatypes` nextest: **712/712, 0 skipped** (76 + 301 + 335). Strict datatypes-only Rust 1.99 Clippy passes. The earlier strict three-package run reports the existing common-query macro diagnostic; the final known-category comparison passes, not strict query lint. Normal formatting/TOML/license hooks pass. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
1164877b5a |
chore: synchronize merged series samples
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
16d478a4c3 |
feat(datatypes): add Narthex series samples and fields (#9599)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) Stack base: `narthex` (includes merged #9598). This PR targets `feat/narthex-series-foundation`; review and merge the foundation first, then retarget this increment to `narthex`. ## What's changed and what's your intention? Add the typed sample and field containers that complete the next in-memory series increment. The incremental diff is limited to the `datatypes` samples module, its focused tests, and its module export. - `Samples` retains typed Arrow float, histogram, mixed, or chunked storage. Null means missing; ordinary NaNs and float payload bits remain values. Histogram indices retain their immutable owning batch, and mixed storage rejects overlapping float/histogram validity. - Clones and checked nonzero slices share numeric buffers, validity bitmaps and histogram tables. Chunk construction flattens descriptors without flattening sample buffers; sequential iteration reports exact remaining length. - Explicit `into_dense()` concatenates only when requested. When every histogram-bearing chunk references the same `Arc<HistogramBatch>`, it reuses that complete table and joins only values and indices, including mixed and all-null histogram chunks. With different table owners, it preserves and concatenates the complete participating tables and checks/remaps their indices; it does not perform per-call owner or content deduplication in that fallback. Numeric builders use the checked local result length rather than Arrow's 1,024-slot default reservation. Pure-histogram densification does not create a float builder; a float payload is allocated only when a Float or Mixed representation requires it, including an all-null Mixed float component. - `SeriesFields` stores one field inline or multiple fields in a boxed slice, rejecting empty or unequal-length fields. `SeriesSchema` shares field descriptions and an optional evaluation grid. - `Series::try_new` validates field count, local time/field alignment and timestamp order. A grid axis requires a value-equal schema grid; explicit and explicit-chunk axes require no schema grid. A short series can hold 20 values on `[800,820)` of a 2,017-point grid without padding its fields to the full grid. This only adds in-memory types. It does not switch HTTP/TQL/SQL queries to a new execution path or change persisted data or network formats. Scanning, frame/Flight integration, execution operators and producer-side synchronized value trimming are not included. Supporting a short range here does not prove that a future computation producer avoids allocating full-grid values; that producer needs its own acceptance tests. ### Focused validation Current verified head: `cb0a75f2c0a7` (the test cleanups plus the histogram-only temporary-allocation fix). The allocation follow-up changes only builder creation, append and finish conditions; table reuse/remapping and sample semantics remain unchanged. Thirteen samples tests cover valid/missing/NaN/stale values, malformed indices and mixed validity, nested/empty chunks, nonzero shared slices, complete histogram payload and whole-table remapping, single/multiple fields, grid policy, invalid explicit ordering, and actual Arrow allocation retention/release. The shared-table regressions cover histogram-only, mixed, and all-null indices; different-owner cases check every logical value and missing position without prescribing table deduplication or index layout. - Rust 1.98.1: `cargo check --locked -p datatypes --all-targets` — passed. - Rust 1.98.1: `cargo nextest run --locked -p datatypes --retries 0 --no-fail-fast` — 335 passed, 0 skipped, both before and after this review cleanup. This includes the parent capacity regression and all thirteen samples tests; the renamed candidate tests were checked in the actual test listing. The five affected or explicitly retained samples tests also passed in a targeted run. - The added numeric-buffer assertion in `densification_preserves_values_nulls_nan_and_remaps_multitable_histograms` fails on the preceding implementation at 12,544 retained bytes and passes with the fix. A temporary measurement of the fixed fixture reported 212 bytes across its float/index buffers, excluding histogram-table storage. These are retained Arrow buffer capacities, not RSS. - A separate local allocator probe around the actual `into_dense()` call used 1,048,576 histogram positions, 64 segments, one shared table and alternating values/missing positions. Requested allocation bytes fell from **12,845,472 to 4,325,584** (10 allocation requests to 5); the one 8,388,608-byte temporary allocation and its in-call free disappeared. Histogram representation, shared table identity and all positions were unchanged. These are requested allocations during this fixture, not RSS, final retained bytes or a throughput benchmark. The probe is not added to the production or unit-test allocator. - Rust 1.99.0: `cargo fmt --all -- --check` — passed. - Rust 1.99.0: `cargo clippy --locked --no-deps -p datatypes --all-targets -- -D warnings` — passed. The pinned 1.98.1 Clippy component is unavailable in the local read-only toolchain; dependency-inclusive 1.99 linting is still blocked by pre-existing `double_must_use` diagnostics in `common-base/src/range_read.rs`. No lint level or CI rule was weakened. - `hawkeye format`, `hawkeye check`, and `git diff --check` — passed. ### Diff-scoped cleanup The earlier cleanup removed an unused crate-private chunk accessor and a single-use generic parameter from the private field container. Public API rustdoc, rationale, SAFETY comments, and all whole tests remain. This review makes only these additional local changes: | Changed test | Removed constraint or duplicate | Retained behavior and actual evidence | | --- | --- | --- | | `densification_preserves_distinct_shared_table_owner_values` (renamed) | Exact output-table cardinality, prescribed remapped indices, and forced distinct output allocation | Genuinely different histogram tables, including repeated owners and missing positions; every output value is checked by the existing complete histogram/float-bit comparator and every non-null index is in bounds. The renamed test passed. | | `short_multifield_series_uses_its_grid_range_and_checks_explicit_chunk_order` | Only the final duplicate single-axis `[2, 1]` rejection block | `fields_clone_buffers_and_series_validate_schema_alignment_and_time_order` retains that rejection with valid matching fields and an ordered positive control. Both tests passed. Short-grid, multifield and cross-chunk checks remain. | The two same-`Arc` sharing regressions retain their pointer-equality guarantees and passed. The mixed regression additionally checks that an all-null float component does not erase the Mixed representation. Owner retention/release, malformed-input, grid-boundary and histogram regressions are not removed. The earlier shared-table memory fix was also checked red/green: the pre-fix implementation retained 64 or 2 table rows where the shared-table fixtures require one; the fixed implementation passes without changing fallback semantics. ### Verified review follow-up - Retargeted to `narthex` after #9598 merged; synchronized its merge commit without rewriting feature history. Current tested/pushed head: `ad1d1ee0fb748dca664726a01508d3fdf499ffe8`. - Consolidate ordering and checked length in crate-private `TimeAxis::try_len`, used by both constructors and `Series::try_new`; remove the duplicated Series helper. Preserve direct-public-payload validation, chunk sharing, error ordering, grids and slicing. - Strengthen the existing clone-owner regression with nonempty value-buffer pointer identity, actual values, and release only after both Series and clone are dropped. - Rust 1.98.1 all-target checking for `datatypes / common-query` passes; complete nextest **400/400, 0 skipped** (335 + 65). Strict datatypes-only Rust 1.99 Clippy and ordinary fmt/TOML/license commit/push hooks pass. No expanded two-package strict-lint claim. - The previous hosted MySQL SQLness job's durable failure annotation reports that the runner lost communication; its raw log is unavailable (`BlobNotFound`). This is not presented as a successful SQLness run, and no SQL semantics or CI settings were changed to mask it. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
ad1d1ee0fb |
refactor(datatypes): share series time validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
70a6c4551f |
refactor(query): share native histogram bucket assembly
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
44b3022a73 |
chore: synchronize merged series foundation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
76c9c1c5be |
feat(datatypes): add shared series foundations for Narthex (#9598)
I hereby agree to the terms of the [GreptimeDB CLA](https://github.com/GreptimeTeam/.github/blob/main/CLA.md). ## Refer to a related PR or issue link (optional) First Narthex foundation PR, targeting the `narthex` development branch. ## What's changed and what's your intention? Introduce the first shared in-memory building blocks for native series execution, extracted from the earlier local implementation and updated to use shared evaluation grids and per-series ranges. - Add `Label` and `MetricName`: shared strings and sorted labels, content-based identity, separate metric-name handling, and copy-on-write label changes. - Add `GridSpec`, `EvalGrid`, and `GridRange`: validate numeric and range bounds, materialize a grid once, and let short results reference their absolute range without copying timestamps. - Add `TimeAxis` support for grid ranges, explicit timestamps, and explicit chunks, including checked zero-copy slicing. - Add Arrow-backed `HistogramBatch`, borrowed histogram/bucket views, and a builder with structural validation. Preserve all histogram components, float bits, and unrecognized reset-hint codes. - Cover label isolation, numeric extremes, malformed inputs, nonzero/nested slices, and real Arrow-allocation retention and release. - Start histogram primitive builders at zero capacity rather than Arrow's 1,024-slot default. A one-row retained-buffer regression rejects the old large reservation; no finish-time shrinking or new capacity API is added. This PR adds types to `datatypes` and re-exports the shared reset-hint enum from `common-query` instead of defining it twice. It does not enable a new query path or change existing HTTP/TQL/SQL results, storage formats, or network protocols. `Samples`, `SeriesFields`, and `SeriesSchema` follow in the next structural increment; producer-side synchronized value trimming, scanning, Flight integration, and execution operators are not included. ### Verification at `bdc12034e559cbabc227ce3e53405928f568a22a` - Rust 1.98.1: `cargo check --locked -p datatypes -p common-query --all-targets` — passed. - Rust 1.98.1: `cargo nextest run --locked -p datatypes -p common-query --retries 0 --no-fail-fast` — **387 passed, 0 skipped** (`datatypes` 322; `common-query` 65). - Rust 1.98.1: `cargo test --locked -p datatypes -p common-query --doc` — command passed; both crates currently have zero doctests, so this is not additional behavior coverage. - Rust 1.99.0: scoped Rust formatting, workspace formatting through ordinary commit/push hooks, TOML checks, Hawkeye format/check, and `git diff --check` — passed. - Rust 1.99.0: `cargo clippy --locked --no-deps -p datatypes -p common-query --all-targets -- -D warnings` — **not passed**: one existing `async_trait` `double_must_use` diagnostic in `common-query/src/logical_plan.rs` is emitted for both lib and lib-test targets. The identical command on immutable parent `78e94ced920` reports the same diagnostics. A diagnostic-only comparison allowing that category passes; no lint setting changed. The earlier datatypes-only strict result below does not certify the expanded two-package command. ### Review follow-up - Keep public `TimeAxis` payloads and clarify the validation boundary. The next increment #9599 already validates directly constructed explicit and chunked axes inside `Series::try_new`, including strict ordering and checked total length. Its direct-construction and cross-chunk-order regressions were run again at published head `cb0a75f2c0a7`: **2 passed**. - Remove `TimeAxis`'s unused representation-derived equality rather than add a timestamp-comparison implementation. - Remove the discarded chunk-total accumulator; ordering validation, empty-chunk removal and zero-copy storage/slicing remain unchanged. The Series boundary keeps its checked aggregate-length validation. - Use the datatypes-owned `CounterResetHint` through a public query-module re-export. Existing query imports and known/unrecognized `i32` conversions remain unchanged. A new focused test checks direct cross-crate type identity and all known codes, unrecognized codes, and both integer extremes; the existing Arrow unknown-hint round trip remains covered. ### Diff-scoped cleanup - Use ordinary `mod tests;` discovery; remove the three redundant `#[path]` attributes without moving or dropping tests. - Remove one duplicate ordinary-label miss assertion from `src/datatypes/src/series/tests.rs`. No whole tests are deleted; all 18 original new tests remain, with one additional retained-capacity regression. Keep invalid-input, boundary and allocation-lifetime coverage and the public API, invariant and SAFETY comments. | Removed assertion | Protected behavior | Surviving coverage | Actual verification | | --- | --- | --- | --- | | `one.get("__name_missing") == None` in `metric_identity_uses_contents_and_distinguishes_absent_from_empty` | Absent ordinary-label lookup | `one.get("missing") == None` in the same test; empty-label and exact `__name__` checks also remain | Named nextest run: 1 passed; full suite before and after: 321 passed, 0 skipped | At the prior checkpoints, the datatypes all-target check, formatting, license and datatypes-only strict primary-crate Clippy checks passed after cleanup and after the capacity fix (322 passed, 0 skipped). A redundant comment restating the new capacity test was also removed before the final checks. No skip, lint level, CI policy or production behavior was changed. ### Small histogram retained-buffer regression The exact new test failed against the pre-fix source: a one-row fixture retained 90,352 bytes of Arrow buffers. With the final production fix, the same fixture retained 528 bytes. These are `get_buffer_memory_size()` capacity measurements, including offsets and validity where present, not RSS measurements. The committed test requires less than 4,096 bytes. Its failure on the original builder and success on the candidate were both run locally. ## PR Checklist Please convert it to a draft if some of the following conditions are not met. - [x] I have written the necessary rustdoc comments. - [x] I have added the necessary unit tests and integration tests. - [ ] This PR requires documentation updates. - [x] API changes are backward compatible. - [x] Schema or data changes are backward compatible. - [ ] This PR needs to be backported to release branches, and I have added the `backport-<target>` labels (e.g. `backport-v1.3` targets `release/v1.3`). |
||
|
discord9
|
bdc12034e5 |
fix(datatypes): address series foundation review feedback
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
discord9
|
ac5ba11255 |
fix(mito): filter typed residuals before raw sample conversion
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |