mirror of
https://github.com/GreptimeTeam/greptimedb.git
synced 2026-11-18 00:34:11 +00:00
5517c4b76ffaae04d22fe7253adc51adfd0367de
1220
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
a1be266c3c |
fix(promql): ignore histogram samples in scalar() and share one kernel (#10426)
* fix(promql): ignore histogram samples in scalar() and share one kernel Prometheus' funcScalar counts only float samples: histogram samples are ignored silently, so they neither supply a value nor make a point hold more than one element. The select kernel rejected them instead, while the native ScalarOf runtime kept its own inline loop that ignored them, so scalar(histogram_metric) answered differently depending on the route. The kernel now ignores histogram samples and the runtime calls it, deleting the inline copy. A point with one float answers that float, with only histograms or with several floats answers the Go NaN. The kernel builds its answer as one dense run over the grid: a Vec<f64> seeded with the Go NaN, where the first float at a point writes its value and a second writes NaN back, plus the counts the caller already reserves. That buffer becomes the result's Float64Array by move, so the per-point peak is the 8 bytes of values plus 1 of counts the reservation already counts. Staging through Vec<Option<f64>> and GridSeriesBuilder would have held about 33 bytes per point against a 9-byte reservation. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): resolve the evaluation grid once in scalar() grid_points already resolves and validates the grid, so asking for the schema's grid again right after could only disagree with what was just checked. Signed-off-by: discord9 <discord9@163.com> * chore(compat): move the from-versions window to v1.3.1 v1.3.1 has published release assets, so the compatibility check asks for it and fails on every branch that still names v1.3.0. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
5194e06122 | Merge remote-tracking branch 'upstream/main' into narthex-merge-main | ||
|
|
ee9ad87fa1 |
Merge upstream/main into narthex (pre-merge rehearsal, not for push)
Rehearsal merge of main ( |
||
|
|
62dad907d0 |
fix: redact credential table options in user-visible output (#10346)
information_schema.tables.create_options (and SHOW TABLE STATUS) printed object store credentials of external tables verbatim, and SHOW CREATE TABLE masked only access_key_id and secret_access_key. Mask all four credential keys in both paths, and sort extra options in create_options by key. Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
115d85b669 |
fix(mito2): identify series by their exact sparse key, not by tsid (#10330)
* fix(mito2): identify series by their exact sparse key, not by tsid `(table_id, tsid)` is a hash, so distinct series can share it: 2000 label values already produce 20 colliding pairs and a 36k-series SST keeps only 27856 distinct tsids. The columnar series writer silently merged such pairs (first label set kept, row counts added) and the samples writer either reported time_regressed or, with increasing timestamps, attached one series' samples to another's labels. Every one of those decisions used the hash as identity. The SST already stores the exact sparse primary key, so nothing on the storage side changes: `tsid`, the hash function and the key layout are untouched. What changes is that the artifacts, their readers and the prepared read path now decide identity on the canonical sparse key bytes, and `(table_id, tsid)` is only the coarse, order-preserving prefix used for ordering, pruning and bucketing. - `ExactSeriesKey` is the new identity type: equality, ordering and hashing are defined on the key bytes alone, `id()` is documented as a derived prefix that is never compared. `from_sparse_key` validates the entire key, not just the prefix, and bounds it at 8 KiB. - Series format v2 keeps one row per exact key; the descriptor layout and the existing per-row tags already express this once the writer aggregates by key. Coarse page/block/part bounds may now touch, so readers prune conservatively. - Samples format v2 records the owning key per block, appends a `__primary_key` column, and lets a key change restart timestamps and ordinals while the same key keeps the monotonic rules. Exact selection travels in `SamplesReadRequest.keys` next to the coarse `ids`. - The prepared path keys candidates by `ExactSeriesKey`, re-encodes identity from the artifact's own tags, associates samples through `__primary_key`, and uses the exact key for the merge key and the direct-label path. - Both manifests carry a mandatory `identity_encoding` discriminator. Artifacts without it are structurally valid and perfectly ordered yet may contain an already merged row, so they are rejected and rebuilt from SSTs; maintenance rotates them out instead of reusing them, and the local cache version drops the rebuildable store. There is no dual reader and no migration. Verified: `cargo nextest run -p mito2` 1824 passed, 2 skipped (three times, identical); `cargo nextest run -p mito-codec` 57 passed; clippy `-D warnings` and rustfmt clean. Tests pin the real captured pair (`host-020078` / `host-029838`, both retained with their own labels), the silent-loss variant with globally increasing timestamps, sixteen distinct keys under one forced tsid in both PK encodings, candidate isolation end to end, and rejection of a frozen v1 fixture. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): close the exact-identity review findings - The two-phase series scan assumed candidate discovery had already enforced tag predicates through an "exact assigned-series set". An assignment is a `(table_id, tsid)` hash, so a colliding partner shares it without matching the predicate, and skipping tag filters returned its rows. Apply the predicates the precise filter already holds instead of assuming them. - `MAX_KEY_BYTES` was checked when reconstructing a key from an artifact but not when a writer built one, so the writer could publish a key the reader refused; a single 8 KiB label already encodes past the cap. Length is bounded by the containers that store a key (one key per block, descriptor, page and column limits), and long labels are real data, so drop the cap instead of enforcing it on both sides. - The `__primary_key` column copies one key per output row, but the read reservation budgeted a fixed 32 bytes per column and metadata pages did not count the per-block key heap: 8192 rows of 8 KiB keys allocated ~64 MiB against a ~1 MiB estimate. Budget the real bytes and count the heap. - `SamplesReadRequest` carried both `keys` and `ids`, which the only caller derived from the same candidates; selection is key-only now and the reader derives its coarse pruning set, so the two selectors cannot disagree. - Restore the unknown-field-codec projection coverage that the frozen v1 fixture used to carry, against a generated v2 artifact, and assert v1 rejection through the compatibility property rather than a field name. - Drop the redundant columnar increasing-timestamp test (that writer never reads timestamps) and correct the samples format doc: the five internal columns and `__primary_key` are non-null, value fields stay nullable. Verified: `cargo nextest run -p mito2` 1830 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Each new test was run with its fix reverted first: the two-phase test returned the colliding partner's rows, the long-key tests failed with "series key exceeds 8192 bytes", and the 4 MiB pool test read 8192 rows of 1 KiB keys while allocating ~8 MiB. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): budget the exact key column exactly and drop stale skip-tag claims The read reservation charged the exact-key column its logical bytes, but the column was built with `BinaryArray::from_iter_values`, whose value buffer grows geometrically: measured on a 2135-row fixture it retained 4,989,432 bytes for 2,551,325 bytes of key data (1.956x), so a pool sized from the reservation was oversubscribed. `BinaryArray::new_repeated` allocates the offsets once and reserves the key bytes once (1.003x on the same fixture), which is what the reservation already assumes. The new test asserts the pool covers what the yielded batch retains, that the column keeps no growth slack, and that the reservation is released on drop; 2135 rows sits just past a doubling step so a growing builder fails it. Also make the documentation match the mechanism and the encoder: - `read/pruner.rs` still told readers the data phase skips tag filtering because candidate discovery had applied the predicates exactly once. Assignments are `(table_id, tsid)` hashes, so a colliding partner shares the identifier without matching the predicate, and the data phase does apply tag predicates. - The initial-window fallback test anchored on `reads[5]`, which is the second payload window; it now identifies the initial window by its payload-header offset and asserts the window spans 128 blocks, so a traversal change cannot quietly point it at another read. - The samples format descriptor sizes are recomputed from the encoder (`48 + 44 * C + key_length`): `224`, `45,236`, and with the leaf prefix `45,248`, each `+ key_length`; the page-fit sentence now states the enforced constraint (descriptor and leaf prefix inside one page) instead of claiming the widest descriptor always fits. Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): budget each projected sample array's fixed overhead The read reservation charged per-row bytes plus the payload sizes, but not the fixed cost of the arrays it builds: Arrow's `get_array_memory_size` is the array struct plus its buffer capacities, so a one-row, key-only projection reserved `key_length + 72` for the 1344 bytes it actually retained. Add `ARRAY_HEADER_BYTES` (`size_of::<BinaryArray>()`) plus `BUFFER_ALIGNMENT` per projected column, as their own term, and cover the one-row key-only case in `exact_key_output_reservation_covers_the_retained_array`: the pool must cover the batch's reported memory size while it is alive and be empty after drop. Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * chore(mito2): pin the series-index cache token to 1 The branch inherits `2` from an earlier layout change. This branch is experimental and has no deployments, so it pins the token at `1`: whatever a different build wrote into the rebuildable local store is discarded on startup and rebuilt from the SSTs. The token is not the compatibility boundary on its own. Artifacts are rejected on open when they lack the mandatory `identity_encoding` discriminator, so a hash-only artifact is never served even when the reset has not run yet (a read that precedes it, or an artifact outside this cache). Signed-off-by: discord9 <discord9@163.com> * feat(mito2): keep exact series keys out of the samples descriptor page A block descriptor used to embed the whole exact key, and a descriptor shares a 65,536-byte metadata page with its leaf prefix. That put an accidental ceiling on key length: 65,344 bytes with no value fields, 65,300 with one, and 20,288 with 1,024 fields. Valid SST primary keys can be longer, so their samples builds failed with "sample descriptor exceeds leaf limit" even though the key itself was accepted. The key now lives in a per-part key area and the descriptor carries a reference: part = [header 16][column payloads][key area][leaf pages / root][trailer 120] trailer = root ref 72 | depth 4 | version 4 | file_length 8 | payload_end 8 | key_end 8 (body 104), magic 8, XXH64 over the preceding 112 bytes 8 block = table_id u32 | tsid u64 | key offset u64 | key length u32 | min i64 | max i64 | ordinal u64 | rows u32 | columns u32 (56 bytes, fixed) Descriptor size no longer depends on key length (232 bytes for one present field, 45,244 for 1,024), so the ceilings are gone. Keys are written once per series, back to back in block order and without padding, so an equal reference means "same exact series" and a key may exceed any page limit; a reference is validated against `key_end` with a checked add, an overlap between consecutive new series is rejected, and a resolved key whose `(table_id, tsid)` disagrees with its descriptor is refused before any row is emitted. Reads stay lazy: only blocks that survive coarse selection fetch a key, at most once per series, merged per page and bounded by `MAX_READ_BYTES`, reused from the part's aligned cache when one is configured. Measured on the 260-series fixture: 260 per-series key reads before, 1 merged read after, with the fetched byte count asserted to be exactly the key bytes. The read contract also stops materializing one key per row: `SamplesBatch { batch, key }` carries the block's key once, the exact-key column is refused in projections, and `prepared` attributes a whole batch with a single binary search. For an 8,192-row block with a 1 KiB key the reservation drops from 8,388,608 B to 1,120 B (~7,490x), which is what lets a 4 MiB pool read it; `Candidate` keeps one owned copy of the key instead of two, and `retained_rows` is gone in favour of the `prepare_retained_rows` metric. No version bump: the layout changes in place. Artifacts written by an earlier shape of this branch (112-byte trailer, or keys inside descriptors) are unreadable and must be rebuilt, which the cache token and the required `identity_encoding` discriminator already bring about. One user-visible cost: keys retained by a scan are charged to its memory pool, so a scan that used to fit in 16 KiB needs ~14 KB more for 128 series. The fixture that pinned 16 KiB is raised to 64 KiB with the arithmetic in a comment. Verified: `cargo test -p mito2 --lib -- series_index::samples series_scan::prepared series_index::identity` 72 passed / 0 failed / 2 ignored on six consecutive runs; clippy `-D warnings` and rustfmt clean. Failing-before evidence, both produced by reverting the relevant hunk: - restoring `48 + 44 * C + key_len` fails the >64 KiB round-trip with "sample descriptor exceeds leaf limit"; - the 260-series coalescing assertion fails against per-series reads. Signed-off-by: discord9 <discord9@163.com> * test(mito2): assert the duplicate-drop on the preparation's own rows `overlapping_buckets_retain_only_new_label_rows` compared the deltas of two process-global counters (`prepare_unique_series`, `prepare_retained_rows`). Every preparation in the process increments both, so under parallel test execution the two deltas do not pair up and the test failed intermittently - it always passed in isolation. Assert the property the pairing was standing in for, on this preparation's own rows: read every series of the prepared source and require that no series keeps two rows at the same timestamp. A surviving duplicate from the overlapping bucket would repeat a timestamp; the counters can no longer make the result depend on what else the test binary happens to be running. Signed-off-by: discord9 <discord9@163.com> * test(mito2): drop the now-unused preparation counter helper The duplicate-drop assertion no longer reads the process-global counters, so `prepared_count` has no callers and `-D warnings` rejects the dead function in `cargo check` and `cargo clippy` CI. Remove it. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): scope key references per part, bound key reads, checksum keys Review of the key-area change found three defects that ordinary data can hit and three smaller ones. All six are fixed with a test that fails beforehand. Key references are file-local offsets, and the ordering state kept them across parts: a series continuing into another part looked like an overlapping reference (or a new series with a non-zero ordinal), and two different series in structurally identical parts could be mistaken for one because their offsets and lengths coincided. Reference ordering is now scoped to a part; continuation across a boundary is decided on the exact key bytes plus time/ordinal, and the loaded keys must be nondecreasing, a repeated key must continue its series. A key prefetch fetched the whole window and only then grew the reservation, and the per-key fallback charged after its read, so a small pool could read and assemble a page of long keys before reporting exhaustion. The peak (fetched entry + owned key + alignment/assembly) is now reserved before the read; on refusal the reader drops alignment first, then halves the window, and a returned key is cloned only after the block reserved for it. A test asserts that an unpayable prefetch issues no key read at all. The key area had no integrity check, so flipping one byte of a label - leaving framing and `(table_id, tsid)` intact - could silently change which series a block belongs to. Every stored key now carries its XXH64, excluded from the reference length and verified when fetched. The manifest discriminator now names the layout (`exact-sparse-key-area` for samples; the series index keeps `exact-sparse-key`), so an artifact with inline keys is refused at open instead of being marked reusable by maintenance and failing later at a part read. No version number changes anywhere. A key copy is also counted correctly when a series spans a metadata rollover: the "first copy of this part" decision is taken after the rollover, not before. Verified: `series_index::samples` 51 passed, `series_scan::prepared` 27 passed, `series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): charge the key peak on every path and chain the order check Four follow-ups from the confirmation review of the key-area change. The per-key fallback reserved only the retained and stored bytes and then fetched through the aligned payload cache, whose aligned/assembly peak it never charged, so a leftover key could read a whole aligned block the pool had not paid for. It now reserves the same peak as the page prefetch: aligned budget first, and on refusal an explicitly cache-less reader with the unaligned peak, before any read. The key carried across a part boundary was a deep copy taken while the per-part reservation was being released, so it lived on uncharged until the next part. It is now held by its own pool consumer, charged before the copy and released when it is replaced or when the boundary falls behind the scan. The content-order check was chained over the last *traversed* block, which an unloaded block overwrote, so a block pruned by selection could hide reversed or restarted keys - and could drop the boundary key entirely when a part ended on a pruned block. It is now chained over the last *loaded* key, with adjacency taken from "the last traversed block is the last loaded one", so a gap still requires increasing order and a strictly greater ordinal but not exactly +1. Finally, the key-area read counter in the tests only saw reads fully inside the key area, missing aligned reads that start before it; it now counts reads that intersect, and the cached-read test is documented as a cache-behaviour guard rather than a failing-before proof for the budget work. Verified: `series_index::samples` 55 passed, `series_scan::prepared` 27 passed, `series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean. Failing-before evidence: a 16384-byte key-area read issued with only 15080 reserved; a boundary key with 0 of its 1195 bytes charged; reversed keys hidden behind a skipped block; and a pruned block hiding the boundary key. Signed-off-by: discord9 <discord9@163.com> * docs(mito2): qualify the ordinal rule to one exact key The samples format said a loaded key's ordinal must still grow, which reads as if ordinals had to increase across different series too. They do not: only an equal exact key continues, an adjacent block of that key requires +1, and a different key starts at zero. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
c8e0d29df2 |
feat(mito2): publish table compaction results conditionally (#10310)
* feat(mito2): publish table compaction results conditionally Publish one region from a live input snapshot using an ordinary file-only edit. Permit repeated submissions and partial success without manifest task state, while preserving sequence, GC protection and uncertain-write recovery. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): write table compaction outputs to level 0 Route split outputs into the ordinary L0 pipeline so window merges and semantic folding stay with region compaction. Update publication validation and tests accordingly. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): release verified footers before manifest publication Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): compare typed table compaction options Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): cover table compaction output validation failures Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): clarify series count verification limits Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject index declarations without output index objects Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): document table compaction outputs as L0 Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * docs(mito2): map the table compaction publication module Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
77cb39b5d1 |
feat(mito2): record pending WAL tails in region manifests (#10308)
* feat(mito2): record pending WAL tails in the region manifest Adds `pending_tails` and `handoff_floors` to the region manifest and its checkpoint, and an optional `pending_tails` / `handoff_floor` pair to `RegionEdit`. An edit without them leaves the records unchanged. A floor is dropped once the flushed entry id reaches it; a tail stays until a commit removes it. Neither is written when empty, so manifests without them encode as before. Opening a region replays its WAL after this node's tails and floor, and WAL releases at open and after a flush stay below this node's tails. Nothing writes these records yet, so every region behaves as before. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * feat(mito2): hold back reads and region changes while WAL tails are pending A region with pending WAL tails is missing data that only those tails hold. Until they are recovered, scans and memtable and SST stats fail with the retryable `RegionPendingTails` error. Truncate, entering staging, a repartition flush, a close-time flush on request, and ALTERs other than adding nullable columns without a default or changing options are rejected before they change any region state. Compaction pauses. A full-listing GC pass skips the region while it has pending tails, or while a newer manifest may exist or the manifest listing fails. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): name the WAL releases that keep pending tails Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that reopening restores the pending-tail read gate Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): pin what the pending-tail fixtures and rejections promise Signed-off-by: jeremyhi <fengjiachun@gmail.com> * refactor(mito2): name the replay start in the open log and share the pending-tail check Signed-off-by: jeremyhi <fengjiachun@gmail.com> * docs(mito2): state the per-node order of pending tails and who keeps later writes out of them Signed-off-by: jeremyhi <fengjiachun@gmail.com> * test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it. Signed-off-by: jeremyhi <fengjiachun@gmail.com> * fix(mito2): drop the pending compaction task state when WAL tails skip it Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying. Signed-off-by: jeremyhi <fengjiachun@gmail.com> --------- Signed-off-by: jeremyhi <fengjiachun@gmail.com> |
||
|
|
e2e980fe39 |
feat(mito2): add column sketch options and index metadata (#10295)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ffa0815285 |
refactor(series): hold the frame layout marker at one value
The series frame is the native PromQL path's internal transport, and that path lives on `narthex`, an experimental branch: it may break its own formats between revisions, so a frame revision number has no compatibility promise to protect. The marker becomes movable once the path reaches a release branch, where a producer and a consumer can legitimately be different revisions. The reader still refuses a frame that declares any other value, which turns a mismatched producer into an error instead of values decoded under the wrong columns. The two wire fixtures are renamed to `v1_metadata.json`, and the drift cases now use the value that is not current. |
||
|
|
f5c1020f5f |
feat(mito2): add shared SST storage and reclamation foundations (#10256)
* feat(mito2): add shared SST storage and reclamation foundations Add shared-level isolation, receipt replay and recovery, migration support, and fail-closed file protection. Keep coordinator submission and verified publication in the next stack layer. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): isolate table compaction manifest semantics Keep atomic manifest extension envelopes and lifecycle safety in the common engine. Gate table-specific receipts and remap admission with enterprise, preserving their wire format and rejecting unsupported state before replay or GC. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): remove manifest-backed compaction task state Keep file-only edits, shared-reference admission and GC safety. Task retries no longer require persisted receipts or task epochs; reject unpublished prototype task-state records rather than silently dropping them. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): guard staging manifests and reclaim shared range indexes Reject prototype task state when decoding staging blobs, matching checkpoint and delta decoding. Delete the region-scoped range index when a shared SST reference is retired, while leaving the shared object to global GC. Document the level-2 merge-split rationale and bind the delta guard's action tag with a test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): simplify GC deletion and lingering rules Require unknown files in a directory without a manifest to pass the lingering window before deletion, so an unpublished table-compaction output cannot be reclaimed out from under an active attempt. Replace the per-batch deletion loop with a single publication recheck taken before remote deletion, and drop the batching constant, the deferred-deletion accounting, and their test. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep regions open after manifest validation rejection Classify InvalidRequest from pre-write applicability checks as definitely not persisted. Cover both shared-removal and unsupported-level rejections without stopping the manifest manager, while keeping ambiguous storage errors conservative. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject ambiguous shared SST reclassification Reject edits that both add a shared SST and remove the same file ID before admission and under the manifest lock. Keep ordinary edit semantics and verify add-only L0 reclassification survives reopen without diverging from the live Version. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * test(mito2): define truncate semantics for shared SST references Keep explicit truncation applicable to shared references and document the distinction from physical reclamation. Exercise full and time-range truncation with real shared SSTs, an old query, another referencing region, and reopen. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): define shared copy results and reject metadata imports Document copied_file_ids as installed logical references, including no-op retries. Reject shared SSTs in metadata-region copies before copying any objects or changing the target, and cover shared, mixed and repeated copy behavior. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * style(mito2): unify shared SST level references and bounds docs Use module-qualified shared-level constants consistently and document MAX_LEVEL as an exclusive bound for SST version operations. No runtime behavior changes. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * refactor(mito2): move the shared SST level to a max sentinel Use the u8 maximum as the shared classification so future ordinary levels can never collide with it, and map wire levels to fixed storage slots instead of indexing by value. Shared metadata, admission and filters keep their semantics; unallocated levels stay rejected. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): keep dropped-region tombstones while files are deferred Young unknown objects and hook-protected candidates on a dropped region must report needs_retry so the meta tombstone survives until they are actually reclaimed; reporting success would let the procedure drop the tombstone and orphan the retained objects. Cover both deferral paths plus the eventual clear. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): reject adopting an existing L1 SST as shared Adopting a shared reference is L0-only: the manifest would replace an existing L1 entry while the live version keeps both handles, duplicating rows until reopen. Reject the edit at admission before any state change and cover the rejection with a real SST. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
16123d806b |
perf(query): push column projections through MergeScan (#10221)
* perf(query): push column projections through MergeScan Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(query): retain empty projections and refresh sqlness plans Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(query): update TSID join plans after projection pushdown Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
361f3595db |
fix(promql): bound scatter time filter expression depth (#10272)
Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
8bdb7d5509 |
feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with node pruning, lazy projected batches, and residual SQL filtering. Reuse inspection request and plan helpers across compaction and SST tables. Cover routing, projection, predicates, and lifecycle behavior in datanode, standalone and distributed integration, and SQL tests. Use the production information extension in the standalone test fixture. Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
c85a92ac57 |
ci: add Prometheus and align Traces benchmark artifact auditing (#10188)
* ci: add Prometheus to the long-range metrics benchmark Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: validate Prometheus with the File/S3 benchmark runtime Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): allow recorded partial Traces query results in artifact audit Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
3051cb41db |
feat(json2): support fulltext indexes on JSON paths (#10064)
* feat(json2): support fulltext indexes on JSON paths Signed-off-by: fys <fengys1996@gmail.com> * chore: use rebased proto for JSON path fulltext Signed-off-by: fys <fengys1996@gmail.com> * refactor: reuse index target memory estimate for fulltext cache Signed-off-by: fys <fengys1996@gmail.com> * fix: filter disabled fulltext options for column targets Signed-off-by: fys <fengys1996@gmail.com> * docs: clarify fulltext intermediate path uniqueness Signed-off-by: fys <fengys1996@gmail.com> * refactor: name index targets consistently Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): apply JSON fulltext indexes per compatible target Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): skip invalid JSON fulltext index targets Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): require enabled fulltext indexes for column targets Signed-off-by: fys <fengys1996@gmail.com> * refactor: simplify JSON fulltext index construction and pruning Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): unify fulltext target compatibility and creator APIs Signed-off-by: fys <fengys1996@gmail.com> * perf(function): preserve string layouts when evaluating matches Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): centralize fulltext predicate extraction errors Signed-off-by: fys <fengys1996@gmail.com> * refactor: remove enable flag from JSON fulltext proto options Signed-off-by: fys <fengys1996@gmail.com> * refactor: clarify JSON fulltext index helpers and imports Signed-off-by: fys <fengys1996@gmail.com> * fix: allow rebuilt fulltext indexes on ordinary columns Signed-off-by: fys <fengys1996@gmail.com> * test: spell out false positive rate in fulltext tests Signed-off-by: fys <fengys1996@gmail.com> * chore: update greptime-proto to upstream merged revision Signed-off-by: fys <fengys1996@gmail.com> * fix(mito2): initialize fulltext option in multi-region tests Signed-off-by: fys <fengys1996@gmail.com> --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
735401adc0 |
feat(mito2): consolidate columnar samples into bounded daily generations (#10129)
* perf(mito2): consolidate columnar samples into daily windows Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): isolate daily source coverage and version the local cache Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): use the exported index purge channel Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): exercise daily source groups across the merge limit Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): build stable daily groups from populated source windows Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): anchor daily coverage once per source group Signed-off-by: evenyag <realevenyag@gmail.com> * test(mito2): validate partial-day reads and grouped rebuild cleanup Signed-off-by: evenyag <realevenyag@gmail.com> * fix(mito2): skip daily sample readers outside query bounds Signed-off-by: evenyag <realevenyag@gmail.com> --------- Signed-off-by: evenyag <realevenyag@gmail.com> |
||
|
|
58a67a47c3 |
feat(query): execute native raw promql over real datanodes (#10150)
* feat(promql): serialize raw series source bindings through Substrait Signed-off-by: discord9 <discord9@163.com> * feat(query): transport interleaved raw series fragments Signed-off-by: discord9 <discord9@163.com> * feat(promql): reconstruct interleaved raw series at repartition Signed-off-by: discord9 <discord9@163.com> * feat(query): plan distributed native raw series Signed-off-by: discord9 <discord9@163.com> * chore: satisfy clippy for the raw series transport The workspace clippy run rejects a redundant rebinding, a constant-size `chunks_exact`, and an implicit `String` clone in the tests these commits carry. No behavior changes. Signed-off-by: discord9 <discord9@163.com> * feat(query): execute native raw promql over real datanodes Plan a whole native statement as one fused node over raw series sources when the frontend reaches regions over the wire, including single-selector statements, while keeping the in-process split and segment shapes unchanged. Validate the whole tree with the runtime constructor before committing to that plan, so unsupported shapes still fall back to the legacy route. Bound the region reads of that topology with a query-wide permit budget of target_partitions, held until each region body is drained; other plan shapes keep unbounded reads rather than risk a body budget they could deadlock. Acceptance on a real two-datanode cluster with TCP frontend clients: time-partitioned series are reconciled before the fused node and match the unpartitioned control, with the DoGet peak at the target bound; series that no region shares keep contiguous per-series order and reach the fused node without a reconciling router. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the raw series transport PR The raw series Substrait message lives in greptime-proto PR #349, which is not merged, so pin its head commit instead of carrying a local path patch. Move the rev to the merge commit once #349 lands. https://github.com/GreptimeTeam/greptime-proto/pull/349 Signed-off-by: discord9 <discord9@163.com> * fix(promql): reconcile a discovered union of raw series A nameless selector reads its matches as one `Union`, and `UnionExec` reports the frame metadata of its last child. A time-partitioned metric sorted before a host-partitioned one therefore looked contiguous, and the fused node skipped the reconciliation of the fragmented member's series, silently folding region fragments as separate series. Lower the declaration on the fused edge to the weakest ordering any member declares, before the fused node reads it. The union itself is unchanged; the fused node's existing interleaved-input path inserts the router. A projection carries the lowered declaration because the pinned physical planner ignores a union's own schema. Cover it with a planner regression for both member orders and an all-complete control, and with a real two-datanode acceptance case that mixes a time-partitioned and a host-partitioned metric and checks the extrapolated increase per series. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover aggregates over discovered metrics A nameless selector's metric comes from its own frames, so a folding aggregate is a multi-input tree even with one selector: apply the same preflight and order-dependent refusals on the remote route. Cover `sum(increase(...))` and grouping by the metric name over the mixed discovery fixture. A grouping that names only `__name__` stays declined, since the output schema cannot name that group: here the metrics also differ by `site`, so grouping by the name and `site` exercises the two groups. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep remote multi-source refusals the split made Skipping the aggregate split and `segment_tree` over the wire must not drop their admission refusals. Two sources that fold groups by metric name would report groups that differ only by name under one label set, and an order-dependent operation over a discovered selector's union would answer in arrival order. Scope both refusals to the actual multi-source inputs rather than to the logical-table-id heuristic: a discovered selector reads several metrics through one selector, and a Mito table beside a metric one carries no logical id. Cover the Mito-plus-metric grouping refusal on a real two-datanode cluster, and cover a discovered `sum by (__name__)` over a raw selector (the name survives it) and a folding `sum(increase(...))`. Signed-off-by: discord9 <discord9@163.com> * docs: drop the local proto patch prerequisite The dependency is pinned to greptime-proto PR #349's commit, so the wire fixture no longer needs a local path patch. Signed-off-by: discord9 <discord9@163.com> * chore: finish the narthex rebase resolution Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and pass the engine state through the native planner's callers, while the remote route still declines the split it cannot carry. Signed-off-by: discord9 <discord9@163.com> * test: pass the engine state to the native operand test The native operand planner now takes the engine state, which decides whether a statement is planned for remote regions. Keep the histogram operand contract test on that signature. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the split and segment plans over the wire Remote planning disabled the aggregate split and segmentation because the split's producer read a bare scan. The producer reads whatever the statement prepared, and over the wire that is the raw series source: the partial half runs per partition on the frontend, the final half folds the exported states, and no aggregate state crosses the wire. Keep the split and let segmentation build evaluated segments from the raw sources instead of bare scans, so per-series window work keeps the scan width. Admit the resulting fused chain to the query-wide read budget, since each fused node polls its input while it is drained. Remove the recovery-projection wrapper: the analyzer no longer inserts that projection, and the physical planner consumes the frame boundary as it is. Serialization is now only the node's encoding: the codec no longer decides placement, and the metric-name binding is checked against the declared frames rather than the node types below. Signed-off-by: discord9 <discord9@163.com> * refactor(query): drop the in-process encodability exception A split aggregate encodes now, so the analyzer no longer needs a special case that skipped the encoding gate for it. The gate itself stays: every plan a region receives still has to encode. Signed-off-by: discord9 <discord9@163.com> * test: cover a repeated selector over one partitioned table Two occurrences of one time-partitioned metric keep their own raw source, time bounds and identity over the wire, so a ratio of two windows and a binary expression over two identical ones stay native and reconcile each occurrence's fragments before the fused node reads them. Compare both against the unpartitioned control. Signed-off-by: discord9 <discord9@163.com> * fix(table): refuse a scan that plans interleaved ordering A scan's planned ordering is a guarantee its own partitions keep. Interleaved is the transport annotation for fragments several producers contribute, so a scanner that planned it has no raw-series read whose frame would honour that declaration. Reject it in the scan conversion, while the frame codec keeps accepting interleaved raw frames from the router that reconciles them. Signed-off-by: discord9 <discord9@163.com> * test(promql): assert the metric name beside the value `sum by (__name__)` over a discovered selector was asserted as a bare sorted list of values, so a plan that grouped by the wrong metric name - or dropped the name and folded everything into one series - could produce those same numbers. Read `__name__` and `val` by column name (the result puts `__name__` last, and a fold that drops the name drops the column rather than emptying it) and compare the `(name, value)` pairs against the fixture's own metric names. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep the hybrid route for a partitioned pair A multi-selector tree over a partitioned table was admitted as one fused node for any pair of tables, not only for scans of one table. That preempted the frontend hybrid route, which copies the matcher's labels into each operand's scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08` depends on exactly that plan: its partitioned RHS holds a duplicated `job="other"` group, so the whole-native binary reports the duplicate instead of the value the hybrid route returns. Refuse the whole-fused plan for that shape in process again; the remote raw-source admission is unchanged, since a remote multi-selector tree over a partitioned table is the point of this branch and the distributed acceptance tests read it. The hybrid route then has to behave the same way it does without native operands: `matching_filters::propagate` now runs for them too, and a rewritten operand is re-planned through `try_plan_native_operand` with its original `selector_matcher` preserved, falling back to the ordinary plan when the native route declines. The unpartitioned case, whose duplicate group the whole-native binary is expected to report, is unchanged. The two new re-plan call sites put the native route's large future inside the recursive planner frame at every nesting level (~178 KB became ~209 KB), which overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where the new branch never ran. Box the future in a non-async helper and move the re-plan into `replan_rewritten_operand`, which brings the frame back to the old size without changing the behaviour. The partitioned-pair unit test asserted the removed shape. It now asks for the remote engine state, where a cross-table pair really is one fused node over two raw sources, and asserts that the same pair in process keeps the hybrid route. Signed-off-by: discord9 <discord9@163.com> * chore(deps): pin greptime-proto to the merged raw series transport GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as `f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit, in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an unmerged ref. Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
5bcb576b49 |
ci: compare GreptimeDB File and S3 in observability benchmarks (#10187)
* ci: compare GreptimeDB File and S3 in observability benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: separate File and S3 benchmark steps and verify report artifacts Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
ec1ad1f15b |
fix(datatypes): escape object keys when rendering jsonb as text (#10112)
* fix(datatypes): escape object keys when rendering jsonb as text Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> * fix(datatypes): skip unicode normalization for escaped jsonb keys Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> * refactor(datatypes): serialize jsonb to text directly with serde_json Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that both object keys and string values are escaped by serde_json. This drops fix_unicode_point, its regex and the hand-written key escaping. Errors are propagated instead of falling back to lossy raw bytes. Characters such as U+2028 are now emitted as-is, which is still valid JSON, so the affected sqlness result lines are regenerated. Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> * fix(datatypes): accept JSON text bytes in jsonb_to_string Some write paths (COPY FROM csv/json/parquet, JSON bound as text via MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode such bytes as jsonb before serializing with serde_json, so they still render (properly escaped) instead of failing with InvalidJsonb. Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> --------- Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com> |
||
|
|
a995ad0951 |
test(compat): exercise the bloom-token case only below v1.3.0 (#10170)
`fulltext_bloom_tokens` reads a fulltext bloom SST in the first format and
asserts that rows written by the older binary stay visible next to a current
one. Its `from_range` started at v1.0.0, but the released v1.3.0 binary
already writes `bloom_version: 2`, so the setup produces two current-format
SSTs and the expectation ("one old, one new") can never hold:
expected: | false | 1 | actual: | true | 2 |
expected: | true | 1 |
Narrow the range to the releases that still write the first format. The
constraint language has `>=`, `<=`, `==` and `=` only, and several entries
are alternatives rather than a conjunction, so `<=v1.2.99` is the bound
that excludes v1.3.0 while keeping v1.2.x and older - including the
nightly window.
Verified with the compat runner: `--from-version v1.2.1 --dry-run` selects
the case, `--from-version v1.3.0 --dry-run` skips it with
`from_range ["<=v1.2.99"] does not match version 'v1.3.0'`, and
`update-compat-versions.py --check --published-only --check-anchors`
passes.
Signed-off-by: discord9 <discord9@163.com>
|
||
|
|
c02e1f58af |
fix(ci): prepare Ubuntu build runners and default EC2 disks to 500 GiB (#10156)
* ci: share Linux runner dependency setup for builds and benchmarks Signed-off-by: WenyXu <wenymedia@gmail.com> * ci: configure EC2 build runner volumes with a 500 GiB default Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
d750d25965 |
perf: push down COUNT and SUM for SST manifest queries (#9949)
* feat: execute logical plans through distributed inspection Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: push down SST manifest scalar aggregates Signed-off-by: WenyXu <wenymedia@gmail.com> * docs: remove redundant SST manifest inspection guide Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: generalize SST manifest aggregate pushdown Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: retain frontend aggregation for unsupported inspection states Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: expose remote SST aggregate plans in EXPLAIN Signed-off-by: WenyXu <wenymedia@gmail.com> * test: normalize SST analyzer in TQL explain snapshots Signed-off-by: WenyXu <wenymedia@gmail.com> * fix: guard SST predicate types before remote inspection Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
b30557960b |
feat(promql): preserve scan parallelism through native frame boundaries (#10055)
* fix(promql): let a closed chain keep its charge until the packer is done Four places released the frame reservation: the early-publish path handed it to the packer, but the queue drain and the end-of-input path freed it in place. Those two run while the packer still holds rows it has not turned into batches, so a series spanning several batches went unbilled once its last piece was handed over. Every emit path now hands the charge over, and the packed stream frees what it holds: what is left when it is polled and pending is the one identity it is turning into batches, and nothing survives its end or its drop. The identity guard is also widened: within one source every series shares its metric name, so a node that drops `__name__`/`__type__`/`__unit__` -- which is what a range function, a math map and the timestamp shape do -- still maps distinct series to distinct labels. Only rewriting an ordinary label can make two equal, and that is what the guard refuses. Ordinary `rate` and arithmetic frame output therefore publish as their chains close instead of buffering a partition. Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep a cross-source tree's partitions, and gather a raw one only when it must Three defects in how this node decided a tree's execution layout. A match whose two operands derive from the same source still needs its match keys co-located, but only the two-source case was recognized. `x - 2*x` with `on(host)` therefore fell through to single-partition execution and lost the parallelism the scan gave it. A raw input never received a matching router: the normalization branch returned it unchanged unless the whole tree was global, while the builder requires a direct matching router for any multi-partition matching-local input. A two-source raw query was rejected at planning. Nothing reconciled the partition counts of different sources either, though the builder requires one common positive count. A single count is now computed for the matching-local case and every router under it uses it, gathering when that count is one. The window restriction also ran before normalization decided the layout, so a tree about to be gathered could be refused for a limit that only applies to partitioned windows. Two tests changed with the behaviour rather than around it: hash routing does not promise which partition holds a given series, so that test compares results instead of placement while keeping its partition count, per-partition execution and operand-order assertions; and a test that pinned the old refusal of a multi-source frame now pins its acceptance. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): keep a raw source's scan partitions instead of gathering them `RawSeriesSourceExec` declared one partition and read every child partition inside `execute(0)`, so a cross-table node lost the scan's parallelism before its inputs reached it. It now declares the child's own count, executes exactly the partition it is asked for, and re-labels that partition's frames. That leaves `SeriesGather` with no consumer: it existed to interleave several partitions into one stream. The module is gone rather than kept as a hidden N-to-one path a later plan could quietly fall back on. Signed-off-by: discord9 <discord9@163.com> * fix(promql): publish a raw source's frames under the ordering it declared `pack_raw_series` derives the frame schema from the stream's own ordering, and `RawSeriesSourceExec` fixes that schema at plan time from a constant. Passing the scan's stream through unchanged broke that agreement: a columnar scan promises key-then-time, so the region published key-then-time frames while the frontend still expected the schema it had built, and `MergeScan` refused them. The forwarding adapter restates the ordering this node declares. It is the weaker promise, so a scan that keeps a finer one still satisfies it, and every other stream property is forwarded unchanged. Signed-off-by: discord9 <discord9@163.com> * fix(promql): route a raw frame that names its labels by its key A region cannot hand out complete labels: it has no metric-name binding, so `prepare_raw_scan` refuses an explicit binding outright. What it hands out is the stored key plus the description a decoder needs, which is the normal input to this node rather than a case to refuse — every single-table selector is planned that way. The node's constructor refused exactly those frames, so any plan that inserted a router under a single-table selector failed before it ran: the matching policies this branch adds, and the one-partition normalization of an evaluated tree. The node publishes labels, because that is what it groups and closes identities by: `finish_raw_group` reads `chunk.metric` and the merge needs it too. So the input is decoded once, at the unpack boundary this node already has, and an already-labelled stream passes through it untouched. The frame it declares is rebuilt from the input's own declaration instead of being compared against it, which keeps the check that a frame this node cannot re-declare is refused while letting input and output differ in how they name the same series. `frame_schema_with` becomes public for that rebuild, since it is the one place that decides what a declaration is. `FusedSeries` now describes the router's output rather than the scan's when it wraps a raw input, because the binding it hands `build` has to match the frame the router actually publishes. Tested through the node directly: a key-labelled input routes at all; matching puts both pieces of one label set in one owner and closes it once; and `RawPrimaryKey` still hashes the key rather than the decoded labels. Signed-off-by: discord9 <discord9@163.com> * test(promql): measure a routed pair by the width it routes to Four assertions still described the layout this branch replaced: a node with two inputs used to read every partition of both into one and answer one partition, and that gather is gone. The pair is now matched in the partitions it was scanned in, so the fused node answers as many partitions as its inputs route to. Each of these fixtures scans with four partitions, and on the base of this branch all five tests passed, so this is a behaviour change this branch made and did not re-verify rather than a defect: the value assertions behind these shape assertions pass unchanged, in both the native and the legacy comparison. The width is read from the routed children instead of being written as four, and the scan is found by walking down through the nodes that only pass its frames along, since a routed input is no longer a bare `SeriesScanExec`. Signed-off-by: discord9 <discord9@163.com> * fix(query): normalize a multi-partition series-frame root A query engine hands its plan to one stream, and for a plan with several partitions that means `CoalescePartitionsExec` - ordinary batch coalescing. A series frame cannot survive it: one stored series may be split across partitions, and a chunk's `is_last` closes that producer rather than the series, so interleaving two partitions' batches produces a stream whose decoder refuses it (`raw series key changed before final chunk`). Evaluated frames have the same shape of problem: their validator keeps one open chain, and a batch from another partition interrupts it. The engine's root is also a region's root, which is how a cross-table native statement reaches this: its per-region sub-plan is rooted at a partitioned `RawSeriesSourceExec`. So the rule belongs at the physical optimizer's end, before the final sanity check, and it normalizes the root and nothing else - `FusedSeries` already inserts its own owner wherever a consumer needs one. A single-partition root is left alone: one producer is the whole answer. A key-labelled raw root is refused rather than coalesced, because the router checks that one stored key decodes to one label set and it needs the labels to do that. No production plan reaches that rejection today - the only producer of a key-labelled frame is the region's series scan, and every plan that executes it puts `RawSeriesSourceExec` or `FusedSeries` above it, both of which publish explicit labels - and the rejection is what keeps it that way. `schema_check` stays enabled: for the roots this rule accepts, the router re-declares its input's frame with the same content, label mode, decode description and ordering, so the schema it publishes is the one it was given. Signed-off-by: discord9 <discord9@163.com> * feat(promql): admit an evaluated frame as a fused input The extension planner hands a child straight to the fused executor when that child already publishes a series frame, and until now it only recognised a raw one. A segment's output is an evaluated frame, so a statement planned as two segments could not be bound above the first one. The check it replaces asked for a raw frame; the property that matters is that the frame names its own series, which is what `labels() == Explicit` says - evaluated frames always declare it and carry no decode description. Everything else stays with `FusedSeriesExec::build`, which already refuses a mixed-content node, an evaluated input that is not contiguous, and a frame whose declaration does not match the binding it was given. Signed-off-by: discord9 <discord9@163.com> * feat(query): plan a global topk under an aggregate as two segments `sum(topk(1, m))` needs the whole vector for its ranking, which is a property of the tree rather than of the statement, and the routes that existed could only answer a whole-vector tree in one partition: the statement fell back to the existing planner. Splitting it lets the per-series prefix keep the scan's partitions while the ranking sees every candidate. The cut goes below the `topk`, not below the `sum`. A segment holding a `topk` reads across series, so it would gather its own input before evaluating the prefix - correct numbers, no parallelism, which is the thing the split is for. Below the `topk`, the prefix runs on the scan's partitions and the ranking and the fold above it run once, on the child's evaluated frame. Both segments are built from one compiled tree: the child is the prefix subtree, the parent is the original tree with that subtree replaced by an input bound to the child's frame. Nothing is re-parsed, so the grid, the matching rules and the bound scans are the ones the compiler produced. The parent needs its output columns without asking the existing planner to name a computed column after its expression, so `fused_output_tags` now derives the grouping labels of an `Aggregate` and passes a selecting `Vector`'s input labels through. It derives output *columns*, not a complete runtime label set: the metric name is bound separately from the tag columns, and a grouping by `__name__` keeps that name in the labels while contributing no column. `fused_root_schema` is that derivation, shared with the ordinary route. The segmenter is its own method, boxed at the call site. Inlined, its future made every frame of the planner's recursion large enough to overflow the stack on a deeply nested statement - `test_parse_or_operator` aborted before this. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the two-segment topk plan and what it keeps The statement's two segments do not land in one physical tree: the prefix has one scan for a child, so the categorizer classifies it commutative and pushes it into the region, while the aggregate above it stays at the frontend. The observer therefore records every plan a query engine wraps, not only the first, so the test can look at the region's own root. What the assertions pin, and why each one is worth its brittleness: - the answer is the global maximum, not a sum of per-partition winners; - the frontend segment holds the complete sum and the global topk over an evaluated input, with no instant evaluation left in it; - the region's segment holds the prefix and its output is an evaluated, explicitly labelled, contiguous frame; - that segment keeps more than one output partition, which is the whole point of cutting below the topk; - the region's root is the series-aware router, one output, and it reads that segment. Column names are not compared: this route names a computed column after its compiled root while the existing planner names it after the expression, which is older than this split and not what this test is about. Signed-off-by: discord9 <discord9@163.com> * feat(promql): share the input-locality classification A tree's distribution requirement is one property, and two layers now ask it. The physical planner already decided with `input_locality` what a tree needs from its inputs; the logical planner has to ask the same question when it splits a statement, or the two can disagree about what a subtree requires. `InputLocality` and `input_locality` are therefore public within the workspace. The classification describes the external inputs a tree needs - not its output distribution, and not which neighbouring operators may share a node - so a caller that reads it as either of those will cut where it should not. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a statement where a gather would swallow its prefixes The previous commit planned `sum(topk(k, prefix))` by matching that shape. A planner that grows one matcher per query shape grows one per benchmark entry, and the shape is not what decides where a statement divides: the tree already says what each subtree needs from its inputs, and `input_locality` answers it. So a segment is now a maximal subtree with one distribution requirement. Inside a tree that must gather, any child that (a) can cross a boundary - it produces evaluated samples rather than raw samples, aggregate state or a scalar - and (b) does not itself need a gather, becomes a segment of its own and keeps the partitions it was scanned with. Nothing else is cut, and a tree with nothing to cut keeps the plan it had. Every segment rebinds its own inputs: the scans it reads directly first, in the order it first names them, then the segments cut out of it, because `FusedSeries` requires dense source indices over its children. `fused_output_tags` derives the parent's columns from those inputs rather than from the statement's first selector. Three gates keep this from answering what the routes before it would not. Each one is a failure this change produced before it was added: - the existing planner must still be able to plan the statement. Deriving tags locally had cut the route off from that refusal, so `sum(clamp(m, 2, 1))` - an empty range, which the existing planner rejects - was answered instead of failing; - every segment's tree must construct a `NativeExecutionState`, because nothing below can fall back once physical planning starts, and a grouping by a value field is a shape the runtime refuses; - the tree must contain a prefix that a boundary could carry, so a whole-vector tree with no per-series work inside it is not split for nothing. The scope is one selector. A statement over two tables binds each scan as a `RawSeriesSource`, and a segment whose child is that wrapper cannot be placed in a region, so its prefix would not gain the partitions the cut is for; those statements keep the plan they had, which the acceptance test pins. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the segment boundaries and their scope One statement per shape the cut can meet, asserting what the rule says rather than what one query happens to produce: a per-series tree stays one node, a matching tree routes instead of gathering, a global fold over per-series work cuts that work out, a constant stays embedded because a boundary cannot carry one, and a statement over two tables keeps the plan it had because a segment over `RawSeriesSource` cannot be placed in a region. Answers are compared as sets of rows with the value column renamed. Three things differ between the routes and none of them is the answer: this one names a value column after its compiled root while the existing planner names it after the expression, the two emit columns in a different order, and this one is partitioned so its rows arrive by partition. Comparing them as what they are leaves the values as the only thing an assertion can fail on. Signed-off-by: discord9 <discord9@163.com> * fix(query): keep a selected segment whole A segment is chosen because its subtree needs one distribution; cutting inside it again contradicts the choice. The inner builder was still applying the cut predicate, and that predicate is "exportable and not `SinglePartition`" - which every node inside a per-series subtree satisfies, since a per-series tree has no `SinglePartition` node in it to begin with. `sum(topk(1, abs(m)))` showed it: the prefix was cut out of the `topk` as one segment, then `abs` was cut out of that segment, leaving a middle segment whose child is a node rather than a scan. That middle segment cannot be placed in a region, so it answered one partition while the scan had several - the prefix lost exactly the parallelism the cut is for. The inner builder now collects the segment's `Input` leaves and nothing else. Signed-off-by: discord9 <discord9@163.com> * refactor(query): bind a segment to its own scan A cut segment carries one source's identity: its scan, its metric name, its time index, and its labels. Building it from the statement's first selector only happens to be the same thing while a statement has one selector. `build_inner_segment` takes the two lists the planner already has - each occurrence's filtered scan, and each occurrence's metric, time index and tags - and binds the segment's single source from them. The parent keeps the wrapped children and the statement-level binding it has now, because a source that reaches the parent directly still has to cross a region as a raw frame. A segment that collects more than one source is declined. `MatchingLocal` satisfies the cut predicate while reading two sources, and one segment cannot carry both sources' identity. No behaviour changes: the entry is still one selector, so no scan is wrapped and the statement-level values are already that selector's. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a statement that reads more than one table A whole-vector statement over two sources is now cut into one per-series prefix segment per source, with the fold above them in the frontend. Each segment binds its own source's scan, metric, time index and labels, so it can be placed in the region that holds that source, and each prefix keeps the scan partitions it was read with. Three things had to change for the existing segmenter to reach that shape: The cut sat inside `!per_partition && !multi_input`, so a statement over two metric tables - which is `multi_input` - never reached it. The cut now sits under `!per_partition` alone. The aggregate split keeps its own conditions, including `!multi_input`, so a multi-input tree is still not split that way. The single-selector gate was doing two jobs. A statement naming several selectors declines unless those selectors read more than one table: the count alone says nothing about whether the tree has more than one input to place, and the shapes it would otherwise admit - one metric named twice under a fold, such as `sum(m) / m` or `m + sum(m)` - are a plan the existing path already answers. `cross_table` rather than `multi_input` decides this, because two Mito tables carry no metric logical id and are still two sources. A segment that collects more than one source is declined. A matching subtree over two sources is a cut candidate, and one segment cannot carry both sources' identity, so `sum(topk(1, m / n))` keeps the plan it had. The segmented parent reports the value column the compiled tree names, not the one the existing planner names; `sum(rate(a[5m])) / sum(b)` reports `v` where the existing plan reports an expression-shaped name. The values are identical and both names are pinned in the test, so a change to either rule shows up there. Restoring the existing name needs a rename above the parent - the node's own schema must keep the compiled names its output columns are classified by - and is left to a follow-up. Signed-off-by: discord9 <discord9@163.com> * test(query): pin that two sources keep their own scan bounds A segmented statement binds each prefix to its own source's scan, and A1 made that binding: `bare_scans` is derived from the prepared scans after each selector's time filter has been pushed into its own `TableScan`. Nothing tested that the two segments actually read `their` bounds rather than one window applied twice - the existing two-source case has no offset, so both sides read the same range and a crossed binding would answer the same numbers. `sum(off_a) / sum(off_b offset 60s)` differs from `sum(off_a) / sum(off_b)` only in the right side's window, so the two queries must disagree: 20/1 and 30/2 against 20/2 and 30/3. The test asserts they do, which is what fails if both segments were given the left scan's bounds. The shape is read from `MergeScanExec`'s `Debug`, because its `DisplayAs` prints only the peers: the debug form carries the table each segment reads and the embedded `FusedSeries` segment. An in-process region hands the plan observer no separately wrapped region tree, so the per-partition evidence for segments is measured in `native_aggregate_test.rs::check_two_source_segregation` instead. Signed-off-by: discord9 <discord9@163.com> * fix(query): decline a multi-source tree that groups by the metric name The gate that lets a multi-source statement reach the segmenter is `selectors.len() != 1 && !cross_table`, but the metric-name grouping guard above it is keyed on `multi_input`, which is `selectors.len() > 1 && logical_ids_differ`. Two ordinary Mito tables differ in table name and carry no metric logical id, so they are `cross_table && !multi_input`: the guard did not apply and the statement reached the segmenter. `fused_output_tags` keeps a grouping by `__name__` out of the declared columns on purpose - the name travels in the chunk's labels, not as a tag column - so `sum by (__name__)` over two sources derives no tag column while the runtime holds two groups that differ only by metric name. Both are then reported under one label set, as two rows a reader cannot tell apart. That is the failure the metric-logical-table test already documents; it could not catch this path because its tables make `multi_input` true. The check is scoped to a tree that reads more than one source. One source folds one metric, so its groups carry the same name and naming that name or dropping it cannot merge two of them. Signed-off-by: discord9 <discord9@163.com> * test(query): cut a range function on each side of a division The two-source cases so far cut an `Instant` prefix: one sample per grid point. A window function's prefix computes its window inside the region over that source's own samples instead, so the segment carries a `Window` rather than an `Instant` and nothing tested that such a prefix is still a cut candidate. `sum(increase(a[2m])) / sum(increase(b[2m]))` over two tables: at the first step the increases are 60 and 30, at the second 120 and 90, so the ratio is 2 and then 4/3. Two different step values are what let a shared window bound change an answer, which is asserted before the result is compared. The shape is read from `MergeScanExec`'s `Debug` for the same reason as the offset case: an in-process region hands the plan observer no separately wrapped region tree, so the per-partition evidence lives in `native_aggregate_test.rs`. This is the shape of the throttling query in the benchmark corpus: a window under a fold, on each side of a division. Signed-off-by: discord9 <discord9@163.com> * test(query): plan the benchmark's throttling and namespace-ratio shapes Every two-source case so far divides two ungrouped scalars. The corpus queries that motivate this work group by a label on each side, so each segment reports several groups and the parent matches the two sides by that label. Nothing tested that path. `container_cpu_cfs_throttled_periods_total` over `container_cpu_cfs_periods_total` is the corpus query verbatim: two metric logical tables on one physical table, `sum(increase(...)) by (container)` on each side. It plans as one frontend `FusedSeries` over two `MergeScan` children, each holding a segment bound to its own metric, and reports the same labelled rows as the existing plan. The namespace-CPU-usage-over-requests shape differs from the corpus text only in the metric names, because the corpus names carry recording-rule colons that a table name cannot: it exercises the nested `sum(max by (...) (...)) by (...)` form and is asserted as that shape rather than as the query verbatim. Both tests assert the native route answered, so a decline fails here instead of passing on the legacy rows. The value column still differs from the existing plan's - the same known gap recorded for the ungrouped case - and both names are pinned. The fixture helper records the three setup traps the hard way: `INSERT` needs explicit column lists because the physical table's column order differs from the logical table's, SQL keywords used as tags need quoting, and no sample may sit at timestamp zero, which both the scan's lower bound and PromQL staleness drop. Signed-off-by: discord9 <discord9@163.com> * test(query): record which native route every corpus query takes The corpus is 29 queries and nothing said which of them the native route answers, or by which of its plans. One parameterised test now runs all of them over generated fixtures and records, per query, the route and the numbers. Four routes, and the frontend tree alone cannot tell them apart: - `segmented` - the frontend runs a `FusedSeries` over `MergeScan` children, one per source; the prefix work happens in the regions. - `aggregate-split` - the frontend runs `AggregateFinalizeExec` over a region `FusedSeries` that reports `partial state:`, so what crosses the boundary is aggregate state. - `region-local` - the frontend holds no native node at all: a per-series tree is commutative, so the whole `FusedSeries` is pushed into the region. A classifier that only looks at the frontend calls these legacy, which is what happened before this test existed. - `legacy` - no native node on either side. All 29 take a native route: sixteen `aggregate-split`, nine `segmented`, two `region-local`, none legacy. The two `region-local` ones are the corpus queries that fold nothing - a bare selector and a `rate` - which is also why pushing the whole tree into the region costs nothing. The test asserts only that the existing plan answers each query with rows, and that a native route's labelled rows agree to within `1e-10` relative. It records the route rather than asserting it, because which route a shape takes is a property of the planner that this test exists to report, not to legislate. Ten queries differ from the existing plan in the last bits; the largest relative difference is 2.9e-16, which is summation order, not arithmetic. The table prints that figure per query so a future equality failure can be told apart from rounding without re-measuring. Every fixture here is generated: explicit `INSERT` column lists, quoted SQL keywords, and no sample at timestamp zero. They establish the plans and the values; they say nothing about speed, memory, or how many partitions actually produced rows. Signed-off-by: discord9 <discord9@163.com> * test(query): record how many partitions each corpus query's nodes used The route matrix said which plan answers each query. It did not say whether that plan produced anything, so a node that ran and emitted nothing looked the same as one that worked. `output_values_per_partition` reads `native.root.output_values` off a `FusedSeriesExec` and returns the counts per partition, and the matrix prints them as `partitions used`. `assert_four_native_partitions` keeps its own assertions and now builds on the same helper. What the column does not establish, and says so where it is printed: these fixtures hold a handful of series each, so a scan puts them all in one partition - every vector here is `[n,0,0,0]`. That is a property of the fixture, not of the plan. The one place a fixture is built to fill the scan's partitions is `native_aggregate_test.rs::check_two_source_segregation`, which asserts all four produce values for both segments of a two-source statement. This matrix records a single assertion instead: at least one partition produced values, so a node that ran and emitted nothing fails. Four cases report `no metric` - L05, L06, L08 and R03, the same-table pointwise divisions. Their region side has no `FusedSeriesExec` at all, because both scans are matched inside one region: that is the route, not a missing measurement, and the table says `no metric` rather than implying zero. Signed-off-by: discord9 <discord9@163.com> * test(query): record the new rule in the explain stages `TQL EXPLAIN` lists the plan after every physical optimizer rule, so registering `SeriesFrameOutput` adds one line to that listing whether or not the rule changes the plan. The three `TQL EXPLAIN VERBOSE` statements in this case use ordinary tables, whose schemas carry no series frame declaration, so the rule returns the plan untouched - the new line says `SAME TEXT AS ABOVE`, which is what the existing lines say for the same reason. Regenerated by the sqlness runner; the diff is those three lines and nothing else. Signed-off-by: discord9 <discord9@163.com> * test(query): make the corpus matrix prove what it claims Three gaps, all in the test the "every corpus query answers natively" claim rests on. The partition helper returned early on a plan that reported no `native.root.output_values`, so a producer that registered nothing passed on the declared partition count alone. It now expects the metric: the point of the assertion is that every partition executed, and an absent counter says nothing about that. `segmented` was decided by the frontend tree containing a `MergeScan`, which also matches a frontend native node consuming raw frames from its regions - a different placement that says nothing about a prefix running in a region. The merge scan's `Debug` carries the plan its region runs, so the check now reads that for a segment; the raw case gets its own `frontend-raw` route. Four queries move there: L05, L06, L08 and R03. The matrix held 23 of the 29 queries: Q05, Q08, Q09 and Q10 had only ever been probed in a throwaway run. They are entries now, with the join labels Q08 and Q09 match on and real `le` buckets for Q10's histogram. A fallback is no longer a route to record: a case that reaches the existing plan fails, because that is the regression this table exists to catch. Twenty-seven cases here plus the separate Q04 and Q07 tests cover all 29. Signed-off-by: discord9 <discord9@163.com> * refactor(query): let the compiler decide which calls this route answers A second list of function names stood beside the compiler's own capability check, and the two disagreed. `admitted_selector` named five calls; the compiler folds `ceil`, `floor`, `sqrt`, `irate`, `avg_over_time`, `sort`, `absent` and more. So `ceil(m)` was refused while `sum(ceil(m))` was admitted, and the same call meant two different things depending on what wrapped it. The compiler already answers this question: `compile_call` refuses a name it has no kernel for with `NotImplemented`, and the planner turns that into a decline. With the list gone there is one place that decides, and a call the compiler folds reaches the route whether or not it is wrapped in an aggregate. `sort` and `absent` read across a vector rather than one series at a time, which is why the old list named them out. They compile, so they are planned; the tests that asserted the refusal now assert the admission. Only the shapes the compiler genuinely cannot express keep the existing plan - a subquery evaluates its body on a grid of its own, which this route does not offer. Signed-off-by: discord9 <discord9@163.com> * refactor(query): admit an operand by its shape, not by call name An operand of a hybrid binary was admitted only if it was a selector under one of five named calls. The compiler folds many more than those five, so `ceil(m)`, `irate(m[5m])`, `sort(m)` and `absent(m)` were refused while the same call under an aggregate was admitted - the name list disagreed with the compiler about what this route can answer. What the list was really protecting is a layout assumption, and the shape states it directly: `try_plan_native_operand` hands back a selector's planner context, and the hybrid caller reads field columns and tags off that context, so an operand whose output layout is not its selector's cannot be one. `sum(A) / B` is refused for that reason, not because `sum` is a name. `single_selector_spine` states it on the expression: a selector, parentheses, unary, a call carrying exactly one vector-or-matrix argument, or a binary whose other side is a compile-time constant. Calls are followed by the type of their arguments rather than by `constant_value`, because a call's string parameters are constants that `constant_value` does not fold and would otherwise look like data. Which calls this route answers is now the compiler's answer alone: a name it has no kernel for comes back as `NotImplemented`, which the planner turns into a decline. Signed-off-by: discord9 <discord9@163.com> * feat(promql): mix raw and evaluated inputs in one fused tree `build` refused a tree whose children were not all raw or all evaluated, but nothing below that point assumed uniformity: `execute` picks the decoder from each source's own schema, `FusedFeed` carries the source index, and `collect_source_schemas` already validates a repeated source against its own grid. A frame stream has one format per stream, not one per consumer, so one segment can decode the raw samples of a rate and compare the result against a vector a previous stage already evaluated. Remove the refusal and make the remaining per-input decisions per source: `evaluated_global` now asks about the source it orders, and `window_sources` keeps only the evaluated sources, since a raw source is answered by the raw path it always used. Cover it by executing `rate(x[5m]) + y` where `x` arrives raw and `y` arrives evaluated on the statement grid. The test asserts all ten (host, timestamp) rows and fails on the removed refusal if the refusal is restored. Signed-off-by: discord9 <discord9@163.com> * feat(promql): bind a segment to every source its subtree reads `cut_as_segment` accepts any exportable subtree that is not `SinglePartition`, so a matching subtree over two sources - `m / n` - is a legal cut point. `build_inner_segment` then required exactly one source and returned `None` for it, and `segment_tree` propagates that `None` as "the statement keeps the plan it had", so `sum(topk(1, m / n))` fell back to the legacy path even though its cut was legal. Build the segment from every source the subtree reads. Each child gets the `RawSeriesSource` boundary already used for separate metric tables, which carries that source's own metric, tags and fields across the boundary and makes the child placeable; a segment over two tables cannot sit inside either region, so it runs above them. The single-source case keeps its existing direct-scan path unchanged, which is what lets a one-table segment still be pushed into its region. Field names come from `tree_source_fields` on the rewritten tree, whose source indexes are already dense in the order the children are built. That helper also validates a repeated source against its own schema and refuses a non-dense index set, which a second traversal would have to re-derive. It becomes `pub` and is re-exported for the query crate to call. Take the segment's time column from its own first source rather than from the statement's first: a cut whose sources are `[1, 2]` would otherwise name the time column of a table it does not read. Cover the construction with a planner test and with a real-Mito integration test that asserts both the exact values and the route: one segment over the two tables, each child reading its own table through its own raw series source. Without the route assertion a silent fallback would pass on equal values. Signed-off-by: discord9 <discord9@163.com> * feat(promql): run a matching tree at the configured width A matching tree's width came from its inputs: `matching_partitions` took the selected source's count, or the maximum of every input's. In standalone a cross-table statement arrives as one `MergeScanExec` per table with one partition each, so `m / n` matched on a single worker while the scans under it ran in parallel. The engine's `execution.target_partitions` was reachable - `plan_fused_series` already uses it for `series_scan` - but never reached this decision. Add `to_execution_plan_with_parallelism(inputs, target_partitions)`, which resolves a matching tree's width as `target_partitions.max(1)` for both the same-source and the multi-input case. `to_execution_plan` keeps its signature and its input-derived width, so a caller that has no configured parallelism is unchanged. `PartitionLocal` and `SinglePartition` trees do not consult the value at all: a per-series tree runs on whatever partitions it was scanned with, and asking for a router there would only add a shuffle. Forward the value through `FusedSeriesExec` because `with_new_children` rebuilds from the node's own children, which are the ones below the router; rebuilding without it would resolve the width again from the pre-router count and silently fall back to one. Verified on a real standalone: the two-table segment now reports four partitions with a `SeriesRepartitionExec` under each one-partition region scan, where it reported one before, and the rows are identical. Signed-off-by: discord9 <discord9@163.com> * test(promql): declare a keyed test frame's ordering, and order its keys `plan_keyed` built its schema with `datatypes::series::frame::series_frame_schema`, which hardcodes the frame metadata `ORDERING` to `contiguous`. The stream it returns packs what `RawScript::ordering` reports, which is now the key-then-time order this helper models. The node reads the ordering off the *plan* schema, so the two disagreeing meant the router scheduled these fixtures as `Contiguous` while unpacking validated them under the stricter ordered contract. Build the schema through `frame_schema_with` with the ordering the stream actually produces, so the plan and the runtime frames state the same contract. That mismatch was invisible until `RawScript` stopped hardcoding `Contiguous`: with both sides claiming `contiguous`, the ordered validation never ran. Running it showed the matching fixture feeding key `b` before `a` within one partition, which its declared ordering forbids. Reorder the partition; the two pieces of `host="a"` still arrive in different partitions, which is what the test is about. Signed-off-by: discord9 <discord9@163.com> * fix(mito2): declare the key order a two-phase raw scan actually keeps `SeriesScan::raw_ordering` reported `SeriesOrdering::Contiguous` for a `TwoPhase` scan that is not reading columnar artifacts. The data that path produces is ordered by primary key: - candidates are merged and sorted ascending by the encoded primary-key column, then decoded to ids by walking that ordered stream without reordering; - partition assignment is a deterministic TSID-to-partition map that appends in arrival order, so one series never spans two partitions; - each assignment is drained completely before the next is taken, rather than emitted in completion order; - the raw adapter's predicate buffering preserves input order and cannot let a later key overtake a held one, and two-phase records always report `ends_package: false`, so it never engages the package-tail machinery that legacy relies on. Declaring the weaker order costs two things that are available: the frame codec's `SequenceValidator` never ran its increasing-key check, and a repartition owner could not use a key watermark to release an identity before another input ended. It also contradicts the scanner's own documented contract, which states the output is always ordered by (primary key, time index) within a partition. Report the order the path actually keeps, and reword the two ordering errors that said "legacy" now that both modes can reach them. Cover it with a test that forces two-phase mode and the non-prepared fallback, asserts which of the two it ran, and then checks per-partition key monotonicity, per-series timestamp monotonicity, single-partition ownership, and the ordering the scan reports. The fixture writes three rounds, uses two output partitions and a residual predicate, and keeps one series above a batch so the multi-batch path is exercised. The three existing two-phase raw tests asserted the old declaration. They check that a series stays in one run and never recurs non-adjacently, which the stronger order still guarantees; only their expected declaration and their names said "contiguous", so those change and the substantive assertions stay. Signed-off-by: discord9 <discord9@163.com> * fix(promql): route a raw match only by a key the computation keeps `same_source` answers "this subtree reads one source", and a `MatchingLocal` classification was taking that answer as licence to insert the router below the fused computations, hashing the labels as they are at the raw stage. Those are different facts. A math map drops the metric name and the `__type__`/`__unit__` metadata, `timestamp` does the same, and every range function drops them except `last_over_time` and `first_over_time` — so `on(__name__)` over `abs(...)` routes by a key the very next node deletes. Concretely, `abs(x) + on(__name__) abs(y)` with one series on each side routes `x` and `y` to different owners; both become the empty key after `abs`, nothing re-partitions between the map and the match, and the pair is silently lost. Decide preservation separately from source identity. A node keeps the key when it cannot remove one of the labels the key is built from; a `Window` consults `RangeFunction::drops_metric_name` rather than assuming, so the two functions that keep the name still route early. Anything not proven to keep the key — a label rewrite, a histogram map, an unmodelled node — does not establish preservation, and the tree is classified `SinglePartition`. That is not a fallback that trades parallelism for correctness. When the computation removes the labels the key is built from, every series on one side reduces to the same empty key, so every left series matches every right one: the match is whole-vector by construction and has no partitioned form that preserves its result. One partition is the only placement that answers it. Where the key does survive, the match stays partitioned. Signed-off-by: discord9 <discord9@163.com> * fix(query): describe an admitted operand by what it produces `try_plan_native_operand` handed the binary matcher the context saved during scanning, which describes the operand's *input* layout. An admitted call can change that layout: `histogram_quantile` folds a classic histogram and drops `le`, so the scan context still names a tag column the operand no longer produces, and default matching then builds a join key over a column that is not there. Concretely, `histogram_quantile(0.9, m) / histogram_quantile(0.5, m)` declines whole-tree planning (two selectors over one table) and is then admitted operand by operand; each operand drops `le`, and planning fails building the join with `FieldNotFound { field: "le" }` against a schema of `timestamp, field_0, host`. Publish only the tag and field columns that the operand's own plan schema actually contains, and drop `use_tsid`, which the produced output does not carry. The check is against the schema the node returns rather than against a list of functions that happen to shrink it, so a call added later cannot reintroduce the gap by not being on a list. Covered by a classic-histogram fixture with `host` and `le` and a float bucket field, asserting each operand's context reports `["host"]` and that the planned join key omits `le`. Reverting the change fails that test with the `FieldNotFound` above. Signed-off-by: discord9 <discord9@163.com> * test(promql): pin the partitioned duplicate-group answer, with the gap named `native_hybrid_binary_05_06_08` asserted that the right side's two `job="other"` series are refused as a duplicate match group. On the partitioned route they are not: the check is gated on `lhs.presence && rhs.presence` computed from the partition's own slices, so the partition holding `other` sees no left-hand data and filters every point out. Pin what the route actually answers, and state the three things that constrain the fix, because each has already been tried and rejected here: - the gate cannot be dropped. `prometheus_cases.txt:1065-1075` is a Prometheus-generated case where the left is absent at one point and the right carries three samples of one group there, and the answer is a value with no error; - the scan cannot run unconditionally, for the same reason; - the match cannot be collapsed to one partition to make both operands whole, because that gives up the parallelism this route exists for. What remains is to let the verdict see whole-operand presence while the partitions keep answering — a deferred check rather than a different placement. This test now makes any change to the current answer visible instead of silent. Signed-off-by: discord9 <discord9@163.com> * feat(query): segment a nested match instead of gathering it A nested one-to-one match — `x + y + z`, or `(x - y) / on(host) z` — passes the logical admission check, because `independent_scan_pair` recurses through `scan_operand`. The physical locality classification does not go that far: it recognizes a match only when both operands read one source each, so `same_source` fails on an operand that is itself a match and the whole tree is classified `SinglePartition`. The fused computation then ran on one partition while the scans under it stayed parallel, which the existing planner does not do: it builds ordinary equijoins that DataFusion can execute as partitioned hash joins. Admission and placement are different questions, so they now get different predicates. The tree is segmented when it is cross-table, not multi-input, classified `SinglePartition`, and its root is a vector match with a nested vector match directly under it. The inner match becomes a segment and the outer one repartitions the evaluated operands by its own key, which is what the segmenter already does for any exportable evaluated prefix; a nested match whose two levels use different keys is routed twice, once per key, rather than pretending one routing serves both. The predicate stays narrow on purpose. Three-selector trees over distinct metric logical ids are `multi_input` and keep their existing construction, including the parent binding that construction produces. A same-table nested match does not enter this branch either, so its prior admission is unchanged. The plan test's collector walks each fused node's native tree in expression-child order and resolves each input's `source_index` against that node's inputs, so a segmented plan is checked for the same property as before — every selector occurrence bound exactly once, in expression order — rather than being checked more weakly. It panics on a shape it does not model instead of returning an empty list. Signed-off-by: discord9 <discord9@163.com> * fix(promql): prove a raw match key survives the computation `preserves_matching_key` decides whether a subtree keeps the labels a vector match routes by. Two arms answered yes without looking at the field that decides it. A binary with a scalar operand recursed into the vector side alone, but arithmetic and a `bool` comparison drop the metric name while a filtering comparison keeps it — the distinction execution already makes. And an `Instant` recursed unconditionally, while `timestamp(x)` compiles to `InstantValue::Timestamp`, which drops the name too; the check written for the separate `Timestamp` node never covered that shape. Both now read the field that decides: the operator together with `return_bool`, and `InstantValue`. The labels checked are the whole computed set — name, type and unit — because `drop_metric_name` removes all three and default matching does not ignore the last two by itself. Without this, `(x * 2) + on(__name__) (y * 2)` was classified matching-local, the router hashed the raw names before the maps dropped them, and a valid pair could land in different partitions with nothing re-partitioning afterwards. The regression asserts the value is 10 and that the tree is `SinglePartition`, computing the destinations the router would have used (`x0` to partition 0, `y0` to partition 1 of two). With the old arm restored the value assertion fails with `left: [10.0], right: []`. Signed-off-by: discord9 <discord9@163.com> * feat(query): find a nested match through the wrappers around it Segmentation was gated on the root itself being a vector match with a nested match beneath it, so `abs(x + y + z)` and `2 * (x + y + z)` never entered: the wrapper or the scalar broadcast sat where the match was expected. Meanwhile the segmenter already walks children and leaves a wrapper in the parent while cutting what it can carry, so no per-wrapper case is needed — the search just has to look through them. Replacing the root check with `carries_prefix` was too wide: almost any multi-source tree contains something a boundary could carry, so ordinary shapes like `some_metric unless on(tag_0) other_metric` were cut for no gain and their inputs stopped being raw sources. So keep requiring a nested vector match, and search for it anywhere in the tree rather than at the root. Wrappers and scalar broadcasts are transparent because the walk visits every child; the nested match itself is still required, which is what keeps the plain multi-source shapes on the plan they had. `cross_table`, `!multi_input` and the `SinglePartition` locality requirement stay, as before: they are what keeps distinct metric logical ids and the same-table shapes out of this branch. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): let a router publish an identity whose chain is complete `SeriesRepartitionExec` published an identity only once every input had contributed it or ended, because an identity may be split across inputs. A per-series region scan assigns each identity to one partition, so on that shape the rule holds every identity until the other partitions reach EOF: a region's whole raw selection is retained before the first series leaves. The node now carries an `IdentityCompletion`, defaulting to the existing merge. `CompleteDisjoint` publishes an identity as soon as its chain closes and keeps the published identities charged in the pool, so a second chain for the same identity fails the execution instead of emitting a partial series as final. The declaration is the producer's, not something a consumer infers from a node's type: publishing early against a producer that does not own its identities would be wrong, so the conservative mode stays the default and every existing constructor keeps it. Signed-off-by: discord9 <discord9@163.com> * fix(query): choose a series-frame root's router from declared provenance `SeriesFrameOutput` normalized every multi-partition frame root through a router that merges identities, which is wrong for a root whose identities are already disjoint: it waits for every scan partition to end before the first series leaves. The rule now selects `CompleteDisjoint` only for a root whose producer declared that property. A raw root qualifies when `RawSeriesSourceExec` was built through `to_per_series_execution_plan` - which also checks the child is a `SeriesScanExec`, so the type alone is not taken as proof - and a direct `SeriesRepartitionExec` root qualifies because it merged the identities itself. Everything else, including a fused frame root, keeps the merging router: a native label transformation can map two source series onto one output label set, so its output identities are not exclusive. The rejection of a key-only raw frame is unchanged and its reason is restated: coalescing cannot produce such a frame, and a router needs explicit labels to check that one key has one binding. Signed-off-by: discord9 <discord9@163.com> * fix(promql): report the labels `absent` names, not the table's columns `fused_output_tags` answered `None` for `absent`, so the caller fell back to the existing planner's column names. Those are the table's tag columns, which is not what `absent` reports: `absent(m{host="x"})` carries the labels its matchers name, and the compiled tree already knows them in `BoundVectorOp::Absent { labels }`. The label set is now read from that field: the named labels are kept in the input's column order, so `absent(native_shapes{host="missing"})` over a table with `host` and `le` reports `host` rather than `ts, val, host, le`. A label the matchers name that is not an input column still answers `None`, because a frame has no column to carry it; `count_values` and `histogram` are unchanged. Caught by the native-versus-legacy value comparison: the extra column made `absent` differ from the existing plan while every value matched. Signed-off-by: discord9 <discord9@163.com> * fix(promql): describe a frame root's emission from what it does `FusedSeriesExec` declared `EmissionType::Final` for a frame output, but `poll_output` hands a frame's evaluated chunk over while its input is still arriving, so the declaration overstated the node to the optimizer. Only a partial aggregate's state is emitted after EOF; every other output keeps its input's emission type. Signed-off-by: discord9 <discord9@163.com> * test(promql): compare the shapes the native route now admits with the existing plan Removing the shapes a name list used to refuse put single-selector `sort` and `sort_desc`, `absent`, and a bare `histogram_quantile` on the segmented route. Nothing compared their answers with the existing plan, so the widening was unverified. The new case runs each shape both ways over one metric table. `sort`, `sort_desc` and both `absent` cases compare the whole printed result, which covers column names and values together. `histogram_quantile` compares values row by row and asserts the one known difference instead of hiding it: a segmented statement names its value column from the compiled tree (`val`) where the existing plan names it after the expression that produced it. That case is given a window long enough to produce a row, so it cannot pass by comparing two empty results. The `absent` cases caught a real difference: the native route reported the table's tag columns rather than the labels the matchers name, which is fixed in the preceding commit. Signed-off-by: discord9 <discord9@163.com> * fix(query): publish a fused root's identities early when the tree keeps them The root rule publishes an identity as soon as its chain closes only for a root whose producer declared that each identity lives in one partition, and it knew two such producers: a per-series raw scan, and a router that had already merged its identities. A fused root was always treated as needing a merge, so a region whose segment is a plain instant selection over one per-series scan still held every identity until the other scan partitions reached EOF - `sum(topk(1, m))` is that shape, and its segment is already multi-partition with no label rewrite. `FusedSeriesExec` now answers whether its output identities are disjoint, and the rule asks it. The answer is narrow on purpose: one input, and a tree of nothing but inputs and sample instant selections, over a per-series scan or another fused node that already answered yes. Anything that computes labels is excluded, because a timestamp or a pointwise map can map two source series onto one output identity, and publishing early would then hand over one identity's pieces as if they were different ones. A vector-to-vector binary reads across series and is excluded too. Signed-off-by: discord9 <discord9@163.com> * fix(query): segment a computed matching key's operands instead of gathering them A vector match whose operands compute away one of the matched labels was classified `SinglePartition` and executed on one partition. That is correct for the bare tree - the raw names cannot route a match whose operands drop them - but it also gave up the parallelism the operands still have: `(x * 2) + on(__name__, host) (y * 2)` loses `__name__` and keeps `host`, so the computed sides can be routed by what they actually carry. The planner now cuts those operands into segments of their own and lets the parent route their evaluated outputs by the unchanged match key. The key itself is not rewritten: `group_key` reads the labels present at that point, so `On(__name__, host)` degrades to the `host` a computed operand still has. Which shapes take this route is decided by one predicate shared with `input_locality`, so the admission rule cannot drift from the locality judgement: a matching site whose two operands are each proven single-source computations, at least one of which does not preserve the key. A failed preservation proof on its own is not enough - it can also mean the proof gave up, as with a parameter that is a compound expression or an operand that is itself a vector match - and the shared doc says so. The same table read twice is admitted for this shape only. The rule that declines several selectors on one table exists to keep folds such as `sum(m) / m` off this route, not because one table cannot be read twice; the compiler already gives each occurrence its own source index, and each cut gets its own input. Distinct partitions are what the two operands need, and they get them. The regression executes both the same-table and the cross-table shape over a four-host fixture and asserts that at least two output partitions produce rows, so the result is not silently produced by one of them. Signed-off-by: discord9 <discord9@163.com> * fix(query): take the early-publication guarantee from the scan's binding The root rule publishes an identity as soon as its chain closes only when the producer can promise that each identity lives in one partition, and the fused answer to that question asked whether its input was a `SeriesScanExec`. The node's type is not that promise: a scan binds a set of logical tags and decodes those into the series' labels, and `try_to_named_series_scan` requires the caller to pass complete tag names but does not check it. A binding that names only some primary-key columns maps two stored series onto one output identity, and early publication would hand one identity's pieces over as different ones. The answer now reads the binding: the label source has to be `PrimaryKey`, and every primary-key column has to be among the bound tags. A source that carries its labels explicitly is not this shape and answers no, as before. Covered by a new `bound_tags_cover_primary_key` unit test - partial coverage is false, full coverage is true, an explicit source is false - and by tests for the predicate's two other parts: a tree that computes labels and a tree that is not one instant selection over one input both answer no, and a gated execution shows a completed identity leaving a fused node while the other input is still held. Signed-off-by: discord9 <discord9@163.com> * refactor(promql): drop the checks the types and decoders already guarantee Two re-checks ran inside the repartition node over facts its inputs had already established, and both cost real work on every query. A raw merge scanned each chunk's timestamps for strict increase, but `RawTimestamps` cannot hold an unordered buffer: its field is private, its only constructor validates, and the accessors are read-only or order-preserving, so no safe caller can build the value this check defends against. The check is gone; the identity, label and shape checks around it stay, because those compare different producers' pieces rather than one chunk's own timestamps. An owner rebuilt each chunk's identity to prove a chain does not change it, but the decoder that hands the chain over already refuses that: `SequenceValidator` holds one open key and rejects a key or metric change before the chain closes, `EvaluatedSequenceValidator` does the same for labels, and the input task that assembles chains only reads from those decoders. What stays is the identity the owner needs anyway to pick a destination, now built by the named functions rather than inline. Where a schema was needed, the node built an empty stream of its own, wrapped it in a packer and read the packer's schema back. The frame module already answers that question from the declaration, so the fake stream is deleted along with its half of the roundtrip. Signed-off-by: discord9 <discord9@163.com> * fix(promql): recursively segment native matching plans Signed-off-by: discord9 <discord9@163.com> * fix(promql): derive native schemas without legacy probes Signed-off-by: discord9 <discord9@163.com> * fix(promql): recognize complete metric scan identities Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 8f752a02dbb9dc7858618c1d5cc0c41e2d4efb23) * fix(promql): release completed disjoint identity history Signed-off-by: discord9 <discord9@163.com> (cherry picked from commit 9975fa6c5c68786e18af230f62041d2aa5e71f6c) * test(promql): assert native aggregate schemas independently Signed-off-by: discord9 <discord9@163.com> * fix(promql): identify internal metric keys by reserved id Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep parallel sources and derive group match labels Signed-off-by: discord9 <discord9@163.com> * test(promql): observe the frontend fused root in scan-parallelism checks Signed-off-by: discord9 <discord9@163.com> * fix(promql): release duplicate evidence immediately on abort Signed-off-by: discord9 <discord9@163.com> * test(promql): preserve operand identity across matching repartition Signed-off-by: discord9 <discord9@163.com> * fix(promql): charge the final duplicate verdict until it is reported Signed-off-by: discord9 <discord9@163.com> * test(promql): observe native scans across frontend and region plans Signed-off-by: discord9 <discord9@163.com> * chore(promql): drop dead test helper and clippy lints Signed-off-by: discord9 <discord9@163.com> * fix(promql): select limitk by canonical label order Signed-off-by: discord9 <discord9@163.com> * fix(promql): keep data-free subtrees in the parent segment Signed-off-by: discord9 <discord9@163.com> * test(promql): prove the duplicate error crosses matching partitions Signed-off-by: discord9 <discord9@163.com> * test(promql): drop debug prints from the segmentation test Signed-off-by: discord9 <discord9@163.com> * revert(promql): keep limitk's Prometheus input-order semantics Signed-off-by: discord9 <discord9@163.com> * chore(promql): satisfy clippy in fused root schema Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
5fe9bdc814 |
feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision. Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * feat(exemplar): persist source metric identities Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): bound exemplar write memory Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): correct batching with exemplars Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(exemplar): accept mixed-case sparse encoding Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(exemplar): reduce write preparation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(prom): reuse exemplar source targets Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: clippy Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(prometheus): defer exemplar series expansion Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
01b0076f3e |
fix(promql): collapse duplicate samples at one timestamp before range functions (#10069)
* fix(promql): collapse duplicate samples at one timestamp before range functions A read path that does not deduplicate, such as an append-mode region, can hand PromQL two rows for one series at one timestamp. Every range function reads its samples from RangeManipulate, so the duplicate entries were treated as independent samples: count_over_time counted both, idelta and irate returned NULL because the last two entries shared a timestamp, and delta/increase/rate deflated because the average sampling interval divides the observed span by sample_count - 1. Prometheus sample identity is one sample per series per timestamp, so its range functions can never see this input. Restore that identity once, at the operator every range function shares, instead of special-casing each function's arithmetic. The collapse compares native ticks, never the millisecond payload, so distinct sub-millisecond samples that share a millisecond stay distinct as calculate_range documents and tests. Batches without duplicate timestamps take a single adjacency scan and are returned untouched; only when duplicates exist is a take built, keeping tag and value columns aligned. Keeping the last row of a group matches MergeMode::LastRow. The normalize test that fed three rows at one timestamp now uses distinct timestamps: it exercises histogram start-timestamp offsetting, and its repeated timestamp was incidental, not asserted behavior. Signed-off-by: discord9 <discord9@163.com> * test(promql): cover duplicate samples end to end and keep the newer value The operator now keeps the first row of a tied timestamp group, which is the row a deduplicating read keeps: storage orders equal primary key and timestamp by sequence descending, and FlatDedupReader takes the first row of each group. The end-to-end case pins that choice by writing 100 and then 101 at the same timestamp: the window reports increase 6, while keeping the stale 100 would report 8. The sqlness case writes duplicate rows into an append_mode region, which is the read path that does not deduplicate, and asserts every range function plus the unaffected ordinary table in both the standalone and distributed environments. Signed-off-by: discord9 <discord9@163.com> * docs(promql): qualify duplicate-timestamp conflict selection Signed-off-by: discord9 <discord9@163.com> * perf(promql): avoid copying tags when collapsing duplicate samples Signed-off-by: discord9 <discord9@163.com> --------- Signed-off-by: discord9 <discord9@163.com> |
||
|
|
ec59131d39 |
feat(ci): support optional Docker targets for query regression (#10028)
* feat(ci): support optional Docker targets for query regression Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): enable Docker when provisioning image regression runners Signed-off-by: WenyXu <wenymedia@gmail.com> * feat(ci): publish precompiled query regression helpers with pin update PRs Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): retry helper image publication on registry timeouts Signed-off-by: WenyXu <wenymedia@gmail.com> * fix(ci): publish helper artifacts with hosted Skopeo job Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor(ci): update helper image through repository variable Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
d33c664183 |
feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints Signed-off-by: fys <fengys1996@gmail.com> * chore: remove unused code files Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): pass build type to bloom filter indexer Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito): share bloom filter field and JSON update loop Signed-off-by: fys <fengys1996@gmail.com> * refactor: better name * refactor(mito2): simplify JSON path bloom index handling Signed-off-by: fys <fengys1996@gmail.com> * refactor(mito2): expose index targets from creators Signed-off-by: fys <fengys1996@gmail.com> * docs(mito2): note JSON path index rebuild risk Signed-off-by: fys <fengys1996@gmail.com> * chore: update greptime-proto * chore: update greptime-proto --------- Signed-off-by: fys <fengys1996@gmail.com> |
||
|
|
7d42feb023 |
chore(ci): update compatibility versions (#10051)
* chore(ci): update compatibility versions
Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
* test(compat): bound the fulltext bloom case below v1.3.0
Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit
|
||
|
|
ec240ad7e6 |
feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions. Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint. Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): preserve reserved sparse key semantics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix(mito): clarify sparse errors and avoid panics Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito): reduce sparse key validation overhead Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: sqlness Signed-off-by: shuiyisong <xixing.sys@gmail.com> * perf(mito-codec): reduce sparse validation cost Signed-off-by: shuiyisong <xixing.sys@gmail.com> * refactor(mito): remove ordinary sparse support Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
eb8e4316ec |
ci: backport scoped heavy CI to narthex (#10053)
* ci: backport scoped heavy CI to narthex Backport #9927 and the CI-only changes from #10043. Adapt script references to narthex, preserve manual platform checks, and document the default-branch schedule limitation. Upstream-Commit: |
||
|
|
efed847ca6 |
fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: enforce sparse primary key schema Signed-off-by: shuiyisong <xixing.sys@gmail.com> * test: fix Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
b21cd16f1d |
feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup Signed-off-by: lczllx <2181719471@qq.com> * fix(datanode): harden the startup object storage self-test Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): cover the File store skip in the object store self-test Signed-off-by: lczllx <2181719471@qq.com> * refactor(datanode): rename the object storage self-test to probe Signed-off-by: lczllx <2181719471@qq.com> * test(datanode): disable object store self-test in sqlness and integration tests Signed-off-by: lczllx <2181719471@qq.com> * docs(datanode): correct what the object storage probe claims to cover Signed-off-by: lczllx <2181719471@qq.com> --------- Signed-off-by: lczllx <2181719471@qq.com> |
||
|
|
045cf74c3f |
perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task Compaction read, merged and encoded each output on a single task, so parquet encoding waited for the merge and the merge waited for encoding. The merge now runs on its own compaction runtime task and hands merged batches to the SST writer through a bounded channel. The channel also closes when that task panics, so the reading side awaits the task before reporting the end of the input. A panic fails the compaction instead of producing a truncated SST. One output of 4.8M rows took 1.28s instead of 1.65s. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * perf(mito2): parallelize decoding and encoding of compaction outputs Parquet encoding took about two thirds of a compaction output's time and ran on a single thread. When `experimental_compaction_output_parallelism` is above 1, the SST writer encodes the columns of each row group on up to that many blocking threads of the compaction runtime while the next row group is buffered, and the merge decodes up to that many input files ahead of it on compaction runtime tasks. The parallel writer feeds every column writer the same input as `ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group reach the object store writer one row group later, so `max_file_size` may split a file one row group later. The option defaults to 1, which keeps today's CPU usage. One output of 4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): cancel compaction producers before consumer polling Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * feat(mito2): support per-table compaction output parallelism Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoding tasks before returning failures Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime. Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain encoders on SST write errors Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error. Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> * fix(mito2): drain compaction outputs on cancellation Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures. Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts. Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> --------- Signed-off-by: Lei, HUANG <ratuthomm@gmail.com> |
||
|
|
4135e65a76 |
fix(query): keep requested sort when filtering on a lossy cast (#10026)
fix(query): pin DataFusion cast constraint propagation fix Update the pinned GreptimeTeam/datafusion revision from e3221b8 to cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger regression test). CastExpr::propagate_constraints used to cast a filter's output interval straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so FilterExec advertised k as constant, the equivalence classes dropped the ordering required by ORDER BY k, and rows returned in scan order. The sqlness case keeps every #9600 assertion and the explicit ORDER BY, and adds ascending and descending checks for the partitioned table, the control table and a table whose matching rows are stored in neither ascending nor descending k order. Cargo.lock only rewrites the 35 datafusion source lines. Signed-off-by: discord9 <discord9@163.com> |
||
|
|
b369d35d8b |
feat: enable JSON2 columns on non-append-only tables (#9985)
feat(mito2): support JSON2 in non-append-only tables Signed-off-by: luofucong <luofc@foxmail.com> |
||
|
|
984f80e277 |
fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit- or timezone-changing time casts A cast on the time index column keeps the windowed-sort optimization eligible only when it preserves the underlying timestamp values. Casts that change the time unit (e.g. ms -> s) or that add a timezone to a naive timestamp re-interpret the values, so the working ranges derived from the time index no longer describe the sort column and the scan fails with a working-range error or silently wrong order. Reject both cast shapes so such queries fall back to the normal SortExec path. The integration fixture uses two rows 1000 seconds apart so the cast output stays distinct and the ordering assertions are not vacuous; the descending arm gets its own expectation. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): stop windowed sort on unit-changing timestamp wrappers to_timestamp_seconds/millis/... cast their argument to their own unit inside the function, so the hidden cast bypassed the explicit-cast guard and kept the windowed-sort optimization eligible on a sort column whose values no longer match the working ranges. The wrapper branch now applies the same unit- and value-preserving predicate as the cast branch, with the predicate extracted into preserves_timestamp_values and shared by both. The integration regression gains to_timestamp_seconds ASC/DESC arms and a to_timestamp_millis value-preserving control; unit tests cover unit-changing, tz-adding, and non-timestamp wrapper inputs. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test: regen windowed sort sqlness after wrapper guard The windowed_sort_nanos and windowed_sort_advance cases sort by to_timestamp_nanos over a millisecond index and to_timestamp_millis over a nanosecond index. Both wrappers change the time unit inside the function, so the windowed-sort rule no longer applies and the plans render a plain SortExec. Query results are unchanged; only the EXPLAIN plan line differs. Regenerated by the sqlness runner. Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): simplify windowed sort cast regressions Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
2e84833ac8 |
feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): reject partition column type changes and quote rule identifiers Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(table): keep time index widening for partition columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * refactor(partition): replace PartitionFunction with IcebergHash Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * docs(partition): describe iceberg_hash in the function module Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test: quote partition rule identifiers in SHOW CREATE TABLE expectations Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * test(partition): check hash rules in information_schema and repartition target columns Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): coerce hash partition rules against the evaluation schema Signed-off-by: Dennis Zhuang <killme2008@gmail.com> * fix(partition): keep quotes in partition column names when building filters Signed-off-by: Dennis Zhuang <killme2008@gmail.com> --------- Signed-off-by: Dennis Zhuang <killme2008@gmail.com> |
||
|
|
2117f8b059 |
feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID Signed-off-by: hahaha <a310608318@gmail.com> * feat: Fix and add tests according to review comments Signed-off-by: hahaha <a310608318@gmail.com> * fix: the zero-argument signature in ADMIN swallows explicit arguments Signed-off-by: hahaha <a310608318@gmail.com> * fix: fix the CI failures caused by SQL version issues Signed-off-by: hahaha <a310608318@gmail.com> * fix: apply the changes based on the review Signed-off-by: hahaha <a310608318@gmail.com> --------- Signed-off-by: hahaha <a310608318@gmail.com> |
||
|
|
5e9bb3c689 |
feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read Signed-off-by: shuiyisong <xixing.sys@gmail.com> * fix: align native histogram ingestion with remote read validation Signed-off-by: shuiyisong <xixing.sys@gmail.com> --------- Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
0dab3aef21 |
fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)
* fix(query): skip LastRow pushdown for IGNORE NULLS Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): keep IGNORE NULLS aggregates above remote scans Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): record IGNORE NULLS results before and after flush Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): share IGNORE NULLS coverage across deployments Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh last-value plans after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
df8f6848cc |
fix(query): avoid lossy casts in partition pruning (#9600)
* fix(query): keep lossy casts out of partition pruning Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): align cast pruning expectations with conservative fallback Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): record lossy cast partition pruning regression Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * fix(query): retain safe pruning conjuncts beside unsupported casts Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> * test(query): refresh cast pruning plans after rebase Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> --------- Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
8170d0458a |
fix(ci): scope Aliyun janitor sweeps to the owning repository (#9966)
Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
6f5c064f54 |
fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process
Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.
The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.
Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache load and pusher updates coherent
A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.
A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.
The epoch check moves into the heartbeat service.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): keep the leader cache coherent with writes while it is disabled
A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* test(meta): pin the epoch 0 exemption and pusher order, and update docs
The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): register a datanode pusher only after its first request passes
A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.
Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): check a datanode epoch again before replying
A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
* fix(meta): handle a datanode's requests one at a time
A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.
The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
---------
Signed-off-by: jeremyhi <fengjiachun@gmail.com>
|
||
|
|
c96cdd0026 |
perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(mito2): stream and prune manifest SST metadata Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(datanode): stream inspection queries through reusable providers Signed-off-by: WenyXu <wenymedia@gmail.com> * perf(catalog): push manifest filters through inspection queries Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: align manifest column constants with module conventions Signed-off-by: WenyXu <wenymedia@gmail.com> * perf: reduce manifest inspection allocations Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: narrow manifest inspection API visibility Signed-off-by: WenyXu <wenymedia@gmail.com> * refactor: retain residual filters for manifest inspection Signed-off-by: WenyXu <wenymedia@gmail.com> --------- Signed-off-by: WenyXu <wenymedia@gmail.com> |
||
|
|
513b9cee58 |
feat(catalog): add public-global information schema scope (#9957)
Signed-off-by: shuiyisong <xixing.sys@gmail.com> |
||
|
|
e87573e286 |
test(sqlness): cover query options in merge-scan regressions (#9956)
* test(sqlness): cover query options on persistent SQL sessions Signed-off-by: Byte-Naut <Byte-Naut@proton.me> * test(query): cover merge-scan joins with query option controls Signed-off-by: Byte-Naut <Byte-Naut@proton.me> --------- Signed-off-by: Byte-Naut <Byte-Naut@proton.me> |
||
|
|
2b67d0d60c |
test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
62652ec62e |
fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |
||
|
|
b20e22b426 |
refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.
What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.
Three combinations are legal and every other one is refused in both
directions:
raw + key + timestamps a storage raw-series scan
raw + explicit + timestamps a raw input whose labels are decoded
evaluated + explicit + grid_spans a compute node's output
An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.
An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.
Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.
Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit
|
||
|
|
6eff9f5804 |
test(promql): verify scalar routes and four-partition broadcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com> |