Commit Graph
1219 Commits
Author SHA1 Message Date
discord9 5194e06122 Merge remote-tracking branch 'upstream/main' into narthex-merge-main 2026-11-12 16:00:40 +08:00
discord9 ee9ad87fa1 Merge upstream/main into narthex (pre-merge rehearsal, not for push)
Rehearsal merge of main (f8d8fe1071) into narthex (115d85b669) on a throwaway
local branch. 31 semantic conflicts resolved; workspace and all test targets
compile; promql 1215/1215 and mito2 series_index 185/185 pass.

Pre-merge pins and follow-ups are marked with TODO(narthex-merge):
- greptime-proto is temporarily pinned to the fork branch carrying
  GreptimeTeam/greptime-proto#355; replace with the upstream rev once it lands.
- native payload byte cache is disabled (main removed PageRangeCache); two
  samples tests are #[ignore]d until it gets a dedicated page-cache budget.
- columnar metadata now has its own CacheManager tier (a quarter of
  sst_meta_cache_size); the unified decoded tier was not kept.
- query.experimental_native_promql is registered as a query option, otherwise
  the native route is silently rejected by the option snapshot.

Signed-off-by: discord9 <discord9@163.com>
2026-11-12 15:45:04 +08:00
dennis zhuang 62dad907d0 fix: redact credential table options in user-visible output (#10346)
information_schema.tables.create_options (and SHOW TABLE STATUS) printed
object store credentials of external tables verbatim, and SHOW CREATE TABLE
masked only access_key_id and secret_access_key. Mask all four credential
keys in both paths, and sort extra options in create_options by key.

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-12 05:49:19 +00:00
discord9 115d85b669 fix(mito2): identify series by their exact sparse key, not by tsid (#10330)
* fix(mito2): identify series by their exact sparse key, not by tsid

`(table_id, tsid)` is a hash, so distinct series can share it: 2000 label
values already produce 20 colliding pairs and a 36k-series SST keeps only 27856
distinct tsids. The columnar series writer silently merged such pairs (first
label set kept, row counts added) and the samples writer either reported
time_regressed or, with increasing timestamps, attached one series' samples to
another's labels. Every one of those decisions used the hash as identity.

The SST already stores the exact sparse primary key, so nothing on the storage
side changes: `tsid`, the hash function and the key layout are untouched. What
changes is that the artifacts, their readers and the prepared read path now
decide identity on the canonical sparse key bytes, and `(table_id, tsid)` is
only the coarse, order-preserving prefix used for ordering, pruning and
bucketing.

- `ExactSeriesKey` is the new identity type: equality, ordering and hashing are
  defined on the key bytes alone, `id()` is documented as a derived prefix that
  is never compared. `from_sparse_key` validates the entire key, not just the
  prefix, and bounds it at 8 KiB.
- Series format v2 keeps one row per exact key; the descriptor layout and the
  existing per-row tags already express this once the writer aggregates by key.
  Coarse page/block/part bounds may now touch, so readers prune conservatively.
- Samples format v2 records the owning key per block, appends a `__primary_key`
  column, and lets a key change restart timestamps and ordinals while the same
  key keeps the monotonic rules. Exact selection travels in
  `SamplesReadRequest.keys` next to the coarse `ids`.
- The prepared path keys candidates by `ExactSeriesKey`, re-encodes identity from
  the artifact's own tags, associates samples through `__primary_key`, and uses
  the exact key for the merge key and the direct-label path.
- Both manifests carry a mandatory `identity_encoding` discriminator. Artifacts
  without it are structurally valid and perfectly ordered yet may contain an
  already merged row, so they are rejected and rebuilt from SSTs; maintenance
  rotates them out instead of reusing them, and the local cache version drops
  the rebuildable store. There is no dual reader and no migration.

Verified: `cargo nextest run -p mito2` 1824 passed, 2 skipped (three times,
identical); `cargo nextest run -p mito-codec` 57 passed; clippy
`-D warnings` and rustfmt clean. Tests pin the real captured pair
(`host-020078` / `host-029838`, both retained with their own labels), the
silent-loss variant with globally increasing timestamps, sixteen distinct keys
under one forced tsid in both PK encodings, candidate isolation end to end, and
rejection of a frozen v1 fixture.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): close the exact-identity review findings

- The two-phase series scan assumed candidate discovery had already enforced
  tag predicates through an "exact assigned-series set". An assignment is a
  `(table_id, tsid)` hash, so a colliding partner shares it without matching
  the predicate, and skipping tag filters returned its rows. Apply the
  predicates the precise filter already holds instead of assuming them.
- `MAX_KEY_BYTES` was checked when reconstructing a key from an artifact but
  not when a writer built one, so the writer could publish a key the reader
  refused; a single 8 KiB label already encodes past the cap. Length is
  bounded by the containers that store a key (one key per block, descriptor,
  page and column limits), and long labels are real data, so drop the cap
  instead of enforcing it on both sides.
- The `__primary_key` column copies one key per output row, but the read
  reservation budgeted a fixed 32 bytes per column and metadata pages did not
  count the per-block key heap: 8192 rows of 8 KiB keys allocated ~64 MiB
  against a ~1 MiB estimate. Budget the real bytes and count the heap.
- `SamplesReadRequest` carried both `keys` and `ids`, which the only caller
  derived from the same candidates; selection is key-only now and the reader
  derives its coarse pruning set, so the two selectors cannot disagree.
- Restore the unknown-field-codec projection coverage that the frozen v1
  fixture used to carry, against a generated v2 artifact, and assert v1
  rejection through the compatibility property rather than a field name.
- Drop the redundant columnar increasing-timestamp test (that writer never
  reads timestamps) and correct the samples format doc: the five internal
  columns and `__primary_key` are non-null, value fields stay nullable.

Verified: `cargo nextest run -p mito2` 1830 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean. Each new test was run with its fix reverted
first: the two-phase test returned the colliding partner's rows, the long-key
tests failed with "series key exceeds 8192 bytes", and the 4 MiB pool test read
8192 rows of 1 KiB keys while allocating ~8 MiB.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): budget the exact key column exactly and drop stale skip-tag claims

The read reservation charged the exact-key column its logical bytes, but the
column was built with `BinaryArray::from_iter_values`, whose value buffer grows
geometrically: measured on a 2135-row fixture it retained 4,989,432 bytes for
2,551,325 bytes of key data (1.956x), so a pool sized from the reservation was
oversubscribed. `BinaryArray::new_repeated` allocates the offsets once and
reserves the key bytes once (1.003x on the same fixture), which is what the
reservation already assumes. The new test asserts the pool covers what the
yielded batch retains, that the column keeps no growth slack, and that the
reservation is released on drop; 2135 rows sits just past a doubling step so a
growing builder fails it.

Also make the documentation match the mechanism and the encoder:

- `read/pruner.rs` still told readers the data phase skips tag filtering because
  candidate discovery had applied the predicates exactly once. Assignments are
  `(table_id, tsid)` hashes, so a colliding partner shares the identifier
  without matching the predicate, and the data phase does apply tag predicates.
- The initial-window fallback test anchored on `reads[5]`, which is the second
  payload window; it now identifies the initial window by its payload-header
  offset and asserts the window spans 128 blocks, so a traversal change cannot
  quietly point it at another read.
- The samples format descriptor sizes are recomputed from the encoder
  (`48 + 44 * C + key_length`): `224`, `45,236`, and with the leaf prefix
  `45,248`, each `+ key_length`; the page-fit sentence now states the enforced
  constraint (descriptor and leaf prefix inside one page) instead of claiming
  the widest descriptor always fits.

Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): budget each projected sample array's fixed overhead

The read reservation charged per-row bytes plus the payload sizes, but not
the fixed cost of the arrays it builds: Arrow's `get_array_memory_size` is the
array struct plus its buffer capacities, so a one-row, key-only projection
reserved `key_length + 72` for the 1344 bytes it actually retained. Add
`ARRAY_HEADER_BYTES` (`size_of::<BinaryArray>()`) plus `BUFFER_ALIGNMENT` per
projected column, as their own term, and cover the one-row key-only case in
`exact_key_output_reservation_covers_the_retained_array`: the pool must cover
the batch's reported memory size while it is alive and be empty after drop.

Verified: `cargo nextest run -p mito2` 1831 passed, 2 skipped; clippy
`-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* chore(mito2): pin the series-index cache token to 1

The branch inherits `2` from an earlier layout change. This branch is
experimental and has no deployments, so it pins the token at `1`: whatever a
different build wrote into the rebuildable local store is discarded on startup
and rebuilt from the SSTs.

The token is not the compatibility boundary on its own. Artifacts are rejected
on open when they lack the mandatory `identity_encoding` discriminator, so a
hash-only artifact is never served even when the reset has not run yet (a read
that precedes it, or an artifact outside this cache).

Signed-off-by: discord9 <discord9@163.com>

* feat(mito2): keep exact series keys out of the samples descriptor page

A block descriptor used to embed the whole exact key, and a descriptor shares a
65,536-byte metadata page with its leaf prefix. That put an accidental ceiling on
key length: 65,344 bytes with no value fields, 65,300 with one, and 20,288 with
1,024 fields. Valid SST primary keys can be longer, so their samples builds failed
with "sample descriptor exceeds leaf limit" even though the key itself was
accepted.

The key now lives in a per-part key area and the descriptor carries a reference:

  part    = [header 16][column payloads][key area][leaf pages / root][trailer 120]
  trailer = root ref 72 | depth 4 | version 4 | file_length 8 | payload_end 8 |
            key_end 8 (body 104), magic 8, XXH64 over the preceding 112 bytes 8
  block   = table_id u32 | tsid u64 | key offset u64 | key length u32 | min i64 |
            max i64 | ordinal u64 | rows u32 | columns u32  (56 bytes, fixed)

Descriptor size no longer depends on key length (232 bytes for one present field,
45,244 for 1,024), so the ceilings are gone. Keys are written once per series,
back to back in block order and without padding, so an equal reference means
"same exact series" and a key may exceed any page limit; a reference is validated
against `key_end` with a checked add, an overlap between consecutive new series is
rejected, and a resolved key whose `(table_id, tsid)` disagrees with its descriptor
is refused before any row is emitted.

Reads stay lazy: only blocks that survive coarse selection fetch a key, at most
once per series, merged per page and bounded by `MAX_READ_BYTES`, reused from the
part's aligned cache when one is configured. Measured on the 260-series fixture:
260 per-series key reads before, 1 merged read after, with the fetched byte count
asserted to be exactly the key bytes.

The read contract also stops materializing one key per row: `SamplesBatch
{ batch, key }` carries the block's key once, the exact-key column is refused in
projections, and `prepared` attributes a whole batch with a single binary search.
For an 8,192-row block with a 1 KiB key the reservation drops from 8,388,608 B to
1,120 B (~7,490x), which is what lets a 4 MiB pool read it; `Candidate` keeps one
owned copy of the key instead of two, and `retained_rows` is gone in favour of the
`prepare_retained_rows` metric.

No version bump: the layout changes in place. Artifacts written by an earlier
shape of this branch (112-byte trailer, or keys inside descriptors) are
unreadable and must be rebuilt, which the cache token and the required
`identity_encoding` discriminator already bring about.

One user-visible cost: keys retained by a scan are charged to its memory pool, so
a scan that used to fit in 16 KiB needs ~14 KB more for 128 series. The fixture
that pinned 16 KiB is raised to 64 KiB with the arithmetic in a comment.

Verified: `cargo test -p mito2 --lib -- series_index::samples series_scan::prepared
series_index::identity` 72 passed / 0 failed / 2 ignored on six consecutive runs;
clippy `-D warnings` and rustfmt clean. Failing-before evidence, both produced by
reverting the relevant hunk:
- restoring `48 + 44 * C + key_len` fails the >64 KiB round-trip with
  "sample descriptor exceeds leaf limit";
- the 260-series coalescing assertion fails against per-series reads.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): assert the duplicate-drop on the preparation's own rows

`overlapping_buckets_retain_only_new_label_rows` compared the deltas of two
process-global counters (`prepare_unique_series`, `prepare_retained_rows`). Every
preparation in the process increments both, so under parallel test execution the
two deltas do not pair up and the test failed intermittently - it always passed in
isolation.

Assert the property the pairing was standing in for, on this preparation's own
rows: read every series of the prepared source and require that no series keeps
two rows at the same timestamp. A surviving duplicate from the overlapping bucket
would repeat a timestamp; the counters can no longer make the result depend on
what else the test binary happens to be running.

Signed-off-by: discord9 <discord9@163.com>

* test(mito2): drop the now-unused preparation counter helper

The duplicate-drop assertion no longer reads the process-global counters, so
`prepared_count` has no callers and `-D warnings` rejects the dead function in
`cargo check` and `cargo clippy` CI. Remove it.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): scope key references per part, bound key reads, checksum keys

Review of the key-area change found three defects that ordinary data can hit and
three smaller ones. All six are fixed with a test that fails beforehand.

Key references are file-local offsets, and the ordering state kept them across
parts: a series continuing into another part looked like an overlapping
reference (or a new series with a non-zero ordinal), and two different series in
structurally identical parts could be mistaken for one because their offsets and
lengths coincided. Reference ordering is now scoped to a part; continuation
across a boundary is decided on the exact key bytes plus time/ordinal, and the
loaded keys must be nondecreasing, a repeated key must continue its series.

A key prefetch fetched the whole window and only then grew the reservation, and
the per-key fallback charged after its read, so a small pool could read and
assemble a page of long keys before reporting exhaustion. The peak (fetched
entry + owned key + alignment/assembly) is now reserved before the read; on
refusal the reader drops alignment first, then halves the window, and a returned
key is cloned only after the block reserved for it. A test asserts that an
unpayable prefetch issues no key read at all.

The key area had no integrity check, so flipping one byte of a label - leaving
framing and `(table_id, tsid)` intact - could silently change which series a
block belongs to. Every stored key now carries its XXH64, excluded from the
reference length and verified when fetched.

The manifest discriminator now names the layout
(`exact-sparse-key-area` for samples; the series index keeps
`exact-sparse-key`), so an artifact with inline keys is refused at open instead
of being marked reusable by maintenance and failing later at a part read. No
version number changes anywhere. A key copy is also counted correctly when a
series spans a metadata rollover: the "first copy of this part" decision is
taken after the rollover, not before.

Verified: `series_index::samples` 51 passed, `series_scan::prepared` 27 passed,
`series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): charge the key peak on every path and chain the order check

Four follow-ups from the confirmation review of the key-area change.

The per-key fallback reserved only the retained and stored bytes and then
fetched through the aligned payload cache, whose aligned/assembly peak it never
charged, so a leftover key could read a whole aligned block the pool had not
paid for. It now reserves the same peak as the page prefetch: aligned budget
first, and on refusal an explicitly cache-less reader with the unaligned peak,
before any read.

The key carried across a part boundary was a deep copy taken while the per-part
reservation was being released, so it lived on uncharged until the next part.
It is now held by its own pool consumer, charged before the copy and released
when it is replaced or when the boundary falls behind the scan.

The content-order check was chained over the last *traversed* block, which an
unloaded block overwrote, so a block pruned by selection could hide reversed or
restarted keys - and could drop the boundary key entirely when a part ended on a
pruned block. It is now chained over the last *loaded* key, with adjacency taken
from "the last traversed block is the last loaded one", so a gap still requires
increasing order and a strictly greater ordinal but not exactly +1.

Finally, the key-area read counter in the tests only saw reads fully inside the
key area, missing aligned reads that start before it; it now counts reads that
intersect, and the cached-read test is documented as a cache-behaviour guard
rather than a failing-before proof for the budget work.

Verified: `series_index::samples` 55 passed, `series_scan::prepared` 27 passed,
`series_index::identity` 4 passed; clippy `-D warnings` and rustfmt clean.
Failing-before evidence: a 16384-byte key-area read issued with only 15080
reserved; a boundary key with 0 of its 1195 bytes charged; reversed keys hidden
behind a skipped block; and a pruned block hiding the boundary key.

Signed-off-by: discord9 <discord9@163.com>

* docs(mito2): qualify the ordinal rule to one exact key

The samples format said a loaded key's ordinal must still grow, which reads as if
ordinals had to increase across different series too. They do not: only an equal
exact key continues, an adjacent block of that key requires +1, and a different
key starts at zero.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-12 12:10:51 +08:00
Lei, HUANG c8e0d29df2 feat(mito2): publish table compaction results conditionally (#10310)
* feat(mito2): publish table compaction results conditionally

Publish one region from a live input snapshot using an ordinary file-only edit. Permit repeated submissions and partial success without manifest task state, while preserving sequence, GC protection and uncertain-write recovery.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): write table compaction outputs to level 0

Route split outputs into the ordinary L0 pipeline so window merges and semantic folding stay with region compaction. Update publication validation and tests accordingly.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): release verified footers before manifest publication

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): compare typed table compaction options

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): cover table compaction output validation failures

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): clarify series count verification limits

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject index declarations without output index objects

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): document table compaction outputs as L0

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* docs(mito2): map the table compaction publication module

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-11 05:42:51 +00:00
jeremyhi 77cb39b5d1 feat(mito2): record pending WAL tails in region manifests (#10308)
* feat(mito2): record pending WAL tails in the region manifest

Adds `pending_tails` and `handoff_floors` to the region manifest and its
checkpoint, and an optional `pending_tails` / `handoff_floor` pair to
`RegionEdit`. An edit without them leaves the records unchanged. A floor is
dropped once the flushed entry id reaches it; a tail stays until a commit
removes it. Neither is written when empty, so manifests without them encode
as before.

Opening a region replays its WAL after this node's tails and floor, and WAL
releases at open and after a flush stay below this node's tails. Nothing
writes these records yet, so every region behaves as before.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* feat(mito2): hold back reads and region changes while WAL tails are pending

A region with pending WAL tails is missing data that only those tails hold.
Until they are recovered, scans and memtable and SST stats fail with the
retryable `RegionPendingTails` error. Truncate, entering staging, a
repartition flush, a close-time flush on request, and ALTERs other than
adding nullable columns without a default or changing options are rejected
before they change any region state. Compaction pauses. A full-listing GC
pass skips the region while it has pending tails, or while a newer manifest
may exist or the manifest listing fails.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): name the WAL releases that keep pending tails

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): reject no-op ALTERs of restricted kinds while WAL tails are pending

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that reopening restores the pending-tail read gate

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): pin what the pending-tail fixtures and rejections promise

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* refactor(mito2): name the replay start in the open log and share the pending-tail check

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* docs(mito2): state the per-node order of pending tails and who keeps later writes out of them

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(mito2): check that a cancelled compaction starts nothing while WAL tails are pending

When a compaction cancelled by task id finishes, the queued manual request goes through handle_pending_compaction_request, whose pending-tails check replies Ok(0). The reply is read with try_recv so that a dispatched request fails the test at once instead of hanging it.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(mito2): drop the pending compaction task state when WAL tails skip it

Skipping a pending manual compaction while WAL tails are pending replied Ok(0) but left its task state in place, so task inspection kept listing it as queued and cancelling it by id still succeeded. Clear the pending task state and publish before replying.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-11-10 12:21:17 +00:00
Weny Xu e2e980fe39 feat(mito2): add column sketch options and index metadata (#10295)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-10 09:55:20 +00:00
discord9 ffa0815285 refactor(series): hold the frame layout marker at one value
The series frame is the native PromQL path's internal transport, and that path lives on `narthex`, an experimental branch: it may break its own formats between revisions, so a frame revision number has no compatibility promise to protect. The marker becomes movable once the path reaches a release branch, where a producer and a consumer can legitimately be different revisions.

The reader still refuses a frame that declares any other value, which turns a mismatched producer into an error instead of values decoded under the wrong columns. The two wire fixtures are renamed to `v1_metadata.json`, and the drift cases now use the value that is not current.
2026-11-10 16:40:03 +08:00
Lei, HUANG f5c1020f5f feat(mito2): add shared SST storage and reclamation foundations (#10256)
* feat(mito2): add shared SST storage and reclamation foundations

Add shared-level isolation, receipt replay and recovery, migration support, and fail-closed file protection. Keep coordinator submission and verified publication in the next stack layer.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): isolate table compaction manifest semantics

Keep atomic manifest extension envelopes and lifecycle safety in the common engine. Gate table-specific receipts and remap admission with enterprise, preserving their wire format and rejecting unsupported state before replay or GC.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): remove manifest-backed compaction task state

Keep file-only edits, shared-reference admission and GC safety. Task retries no longer require persisted receipts or task epochs; reject unpublished prototype task-state records rather than silently dropping them.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): guard staging manifests and reclaim shared range indexes

Reject prototype task state when decoding staging blobs, matching checkpoint and delta decoding. Delete the region-scoped range index when a shared SST reference is retired, while leaving the shared object to global GC. Document the level-2 merge-split rationale and bind the delta guard's action tag with a test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): simplify GC deletion and lingering rules

Require unknown files in a directory without a manifest to pass the lingering window before deletion, so an unpublished table-compaction output cannot be reclaimed out from under an active attempt. Replace the per-batch deletion loop with a single publication recheck taken before remote deletion, and drop the batching constant, the deferred-deletion accounting, and their test.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): keep regions open after manifest validation rejection

Classify InvalidRequest from pre-write applicability checks as definitely not persisted. Cover both shared-removal and unsupported-level rejections without stopping the manifest manager, while keeping ambiguous storage errors conservative.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject ambiguous shared SST reclassification

Reject edits that both add a shared SST and remove the same file ID before admission and under the manifest lock. Keep ordinary edit semantics and verify add-only L0 reclassification survives reopen without diverging from the live Version.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* test(mito2): define truncate semantics for shared SST references

Keep explicit truncation applicable to shared references and document the distinction from physical reclamation. Exercise full and time-range truncation with real shared SSTs, an old query, another referencing region, and reopen.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): define shared copy results and reject metadata imports

Document copied_file_ids as installed logical references, including no-op retries. Reject shared SSTs in metadata-region copies before copying any objects or changing the target, and cover shared, mixed and repeated copy behavior.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* style(mito2): unify shared SST level references and bounds docs

Use module-qualified shared-level constants consistently and document MAX_LEVEL as an exclusive bound for SST version operations. No runtime behavior changes.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* refactor(mito2): move the shared SST level to a max sentinel

Use the u8 maximum as the shared classification so future ordinary levels can never collide with it, and map wire levels to fixed storage slots instead of indexing by value. Shared metadata, admission and filters keep their semantics; unallocated levels stay rejected.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): keep dropped-region tombstones while files are deferred

Young unknown objects and hook-protected candidates on a dropped region must report needs_retry so the meta tombstone survives until they are actually reclaimed; reporting success would let the procedure drop the tombstone and orphan the retained objects. Cover both deferral paths plus the eventual clear.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): reject adopting an existing L1 SST as shared

Adopting a shared reference is L0-only: the manifest would replace an existing L1 entry while the live version keeps both handles, duplicating rows until reopen. Reject the edit at admission before any state change and cover the rejection with a real SST.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-11-10 06:39:15 +00:00
dennis zhuang 16123d806b perf(query): push column projections through MergeScan (#10221)
* perf(query): push column projections through MergeScan

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(query): retain empty projections and refresh sqlness plans

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(query): update TSID join plans after projection pushdown

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-11-09 22:14:09 +00:00
Yingwen 361f3595db fix(promql): bound scatter time filter expression depth (#10272)
Signed-off-by: evenyag <realevenyag@gmail.com>
2026-11-09 09:03:25 +00:00
Weny Xu 8bdb7d5509 feat: add information_schema.compaction_tasks (#10173)
Expose live compaction tasks through the datanode inspection path with
node pruning, lazy projected batches, and residual SQL filtering. Reuse
inspection request and plan helpers across compaction and SST tables.

Cover routing, projection, predicates, and lifecycle behavior in datanode,
standalone and distributed integration, and SQL tests. Use the production
information extension in the standalone test fixture.

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-09 03:17:48 +00:00
Weny Xu c85a92ac57 ci: add Prometheus and align Traces benchmark artifact auditing (#10188)
* ci: add Prometheus to the long-range metrics benchmark

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: validate Prometheus with the File/S3 benchmark runtime

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): allow recorded partial Traces query results in artifact audit

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-06 18:19:49 +00:00
fys 3051cb41db feat(json2): support fulltext indexes on JSON paths (#10064)
* feat(json2): support fulltext indexes on JSON paths

Signed-off-by: fys <fengys1996@gmail.com>

* chore: use rebased proto for JSON path fulltext

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: reuse index target memory estimate for fulltext cache

Signed-off-by: fys <fengys1996@gmail.com>

* fix: filter disabled fulltext options for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* docs: clarify fulltext intermediate path uniqueness

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: name index targets consistently

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): apply JSON fulltext indexes per compatible target

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): skip invalid JSON fulltext index targets

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): require enabled fulltext indexes for column targets

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: simplify JSON fulltext index construction and pruning

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): unify fulltext target compatibility and creator APIs

Signed-off-by: fys <fengys1996@gmail.com>

* perf(function): preserve string layouts when evaluating matches

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): centralize fulltext predicate extraction errors

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: remove enable flag from JSON fulltext proto options

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: clarify JSON fulltext index helpers and imports

Signed-off-by: fys <fengys1996@gmail.com>

* fix: allow rebuilt fulltext indexes on ordinary columns

Signed-off-by: fys <fengys1996@gmail.com>

* test: spell out false positive rate in fulltext tests

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto to upstream merged revision

Signed-off-by: fys <fengys1996@gmail.com>

* fix(mito2): initialize fulltext option in multi-region tests

Signed-off-by: fys <fengys1996@gmail.com>

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-11-06 10:33:10 +00:00
Yingwen 735401adc0 feat(mito2): consolidate columnar samples into bounded daily generations (#10129)
* perf(mito2): consolidate columnar samples into daily windows

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): isolate daily source coverage and version the local cache

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): use the exported index purge channel

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): exercise daily source groups across the merge limit

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): build stable daily groups from populated source windows

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): anchor daily coverage once per source group

Signed-off-by: evenyag <realevenyag@gmail.com>

* test(mito2): validate partial-day reads and grouped rebuild cleanup

Signed-off-by: evenyag <realevenyag@gmail.com>

* fix(mito2): skip daily sample readers outside query bounds

Signed-off-by: evenyag <realevenyag@gmail.com>

---------

Signed-off-by: evenyag <realevenyag@gmail.com>
2026-11-06 16:00:22 +08:00
discord9 58a67a47c3 feat(query): execute native raw promql over real datanodes (#10150)
* feat(promql): serialize raw series source bindings through Substrait

Signed-off-by: discord9 <discord9@163.com>

* feat(query): transport interleaved raw series fragments

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): reconstruct interleaved raw series at repartition

Signed-off-by: discord9 <discord9@163.com>

* feat(query): plan distributed native raw series

Signed-off-by: discord9 <discord9@163.com>

* chore: satisfy clippy for the raw series transport

The workspace clippy run rejects a redundant rebinding, a constant-size
`chunks_exact`, and an implicit `String` clone in the tests these
commits carry. No behavior changes.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): execute native raw promql over real datanodes

Plan a whole native statement as one fused node over raw series
sources when the frontend reaches regions over the wire, including
single-selector statements, while keeping the in-process split and
segment shapes unchanged. Validate the whole tree with the runtime
constructor before committing to that plan, so unsupported shapes
still fall back to the legacy route.

Bound the region reads of that topology with a query-wide permit
budget of target_partitions, held until each region body is drained;
other plan shapes keep unbounded reads rather than risk a body budget
they could deadlock.

Acceptance on a real two-datanode cluster with TCP frontend clients:
time-partitioned series are reconciled before the fused node and match
the unpartitioned control, with the DoGet peak at the target bound;
series that no region shares keep contiguous per-series order and
reach the fused node without a reconciling router.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): pin greptime-proto to the raw series transport PR

The raw series Substrait message lives in greptime-proto PR #349, which
is not merged, so pin its head commit instead of carrying a local path
patch. Move the rev to the merge commit once #349 lands.

    https://github.com/GreptimeTeam/greptime-proto/pull/349

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): reconcile a discovered union of raw series

A nameless selector reads its matches as one `Union`, and `UnionExec`
reports the frame metadata of its last child. A time-partitioned metric
sorted before a host-partitioned one therefore looked contiguous, and
the fused node skipped the reconciliation of the fragmented member's
series, silently folding region fragments as separate series.

Lower the declaration on the fused edge to the weakest ordering any
member declares, before the fused node reads it. The union itself is
unchanged; the fused node's existing interleaved-input path inserts the
router. A projection carries the lowered declaration because the
pinned physical planner ignores a union's own schema.

Cover it with a planner regression for both member orders and an
all-complete control, and with a real two-datanode acceptance case that
mixes a time-partitioned and a host-partitioned metric and checks the
extrapolated increase per series.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover aggregates over discovered metrics

A nameless selector's metric comes from its own frames, so a folding
aggregate is a multi-input tree even with one selector: apply the same
preflight and order-dependent refusals on the remote route. Cover
`sum(increase(...))` and grouping by the metric name over the mixed
discovery fixture.

A grouping that names only `__name__` stays declined, since the output
schema cannot name that group: here the metrics also differ by `site`,
so grouping by the name and `site` exercises the two groups.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep remote multi-source refusals the split made

Skipping the aggregate split and `segment_tree` over the wire must not
drop their admission refusals. Two sources that fold groups by metric
name would report groups that differ only by name under one label set,
and an order-dependent operation over a discovered selector's union
would answer in arrival order.

Scope both refusals to the actual multi-source inputs rather than to
the logical-table-id heuristic: a discovered selector reads several
metrics through one selector, and a Mito table beside a metric one
carries no logical id.

Cover the Mito-plus-metric grouping refusal on a real two-datanode
cluster, and cover a discovered `sum by (__name__)` over a raw selector
(the name survives it) and a folding `sum(increase(...))`.

Signed-off-by: discord9 <discord9@163.com>

* docs: drop the local proto patch prerequisite

The dependency is pinned to greptime-proto PR #349's commit, so the wire
fixture no longer needs a local path patch.

Signed-off-by: discord9 <discord9@163.com>

* chore: finish the narthex rebase resolution

Keep the upstream aggregate split (`FusedSeries::split_aggregate`) and
pass the engine state through the native planner's callers, while the
remote route still declines the split it cannot carry.

Signed-off-by: discord9 <discord9@163.com>

* test: pass the engine state to the native operand test

The native operand planner now takes the engine state, which decides
whether a statement is planned for remote regions. Keep the histogram
operand contract test on that signature.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep the split and segment plans over the wire

Remote planning disabled the aggregate split and segmentation because
the split's producer read a bare scan. The producer reads whatever the
statement prepared, and over the wire that is the raw series source:
the partial half runs per partition on the frontend, the final half
folds the exported states, and no aggregate state crosses the wire.

Keep the split and let segmentation build evaluated segments from the
raw sources instead of bare scans, so per-series window work keeps the
scan width. Admit the resulting fused chain to the query-wide read
budget, since each fused node polls its input while it is drained.

Remove the recovery-projection wrapper: the analyzer no longer inserts
that projection, and the physical planner consumes the frame boundary
as it is. Serialization is now only the node's encoding: the codec no
longer decides placement, and the metric-name binding is checked
against the declared frames rather than the node types below.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): drop the in-process encodability exception

A split aggregate encodes now, so the analyzer no longer needs a
special case that skipped the encoding gate for it. The gate itself
stays: every plan a region receives still has to encode.

Signed-off-by: discord9 <discord9@163.com>

* test: cover a repeated selector over one partitioned table

Two occurrences of one time-partitioned metric keep their own raw
source, time bounds and identity over the wire, so a ratio of two
windows and a binary expression over two identical ones stay native and
reconcile each occurrence's fragments before the fused node reads them.
Compare both against the unpartitioned control.

Signed-off-by: discord9 <discord9@163.com>

* fix(table): refuse a scan that plans interleaved ordering

A scan's planned ordering is a guarantee its own partitions keep.
Interleaved is the transport annotation for fragments several producers
contribute, so a scanner that planned it has no raw-series read whose
frame would honour that declaration. Reject it in the scan conversion,
while the frame codec keeps accepting interleaved raw frames from the
router that reconciles them.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): assert the metric name beside the value

`sum by (__name__)` over a discovered selector was asserted as a bare sorted
list of values, so a plan that grouped by the wrong metric name - or dropped
the name and folded everything into one series - could produce those same
numbers. Read `__name__` and `val` by column name (the result puts `__name__`
last, and a fold that drops the name drops the column rather than emptying it)
and compare the `(name, value)` pairs against the fixture's own metric names.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep the hybrid route for a partitioned pair

A multi-selector tree over a partitioned table was admitted as one fused node
for any pair of tables, not only for scans of one table. That preempted the
frontend hybrid route, which copies the matcher's labels into each operand's
scan and checks uniqueness above the join. `native_hybrid_binary_05_06_08`
depends on exactly that plan: its partitioned RHS holds a duplicated
`job="other"` group, so the whole-native binary reports the duplicate instead
of the value the hybrid route returns. Refuse the whole-fused plan for that
shape in process again; the remote raw-source admission is unchanged, since a
remote multi-selector tree over a partitioned table is the point of this
branch and the distributed acceptance tests read it.

The hybrid route then has to behave the same way it does without native
operands: `matching_filters::propagate` now runs for them too, and a rewritten
operand is re-planned through `try_plan_native_operand` with its original
`selector_matcher` preserved, falling back to the ordinary plan when the
native route declines. The unpartitioned case, whose duplicate group the
whole-native binary is expected to report, is unchanged.

The two new re-plan call sites put the native route's large future inside the
recursive planner frame at every nesting level (~178 KB became ~209 KB), which
overflowed the 2 MiB test-thread stack in `test_parse_or_operator` even where
the new branch never ran. Box the future in a non-async helper and move the
re-plan into `replan_rewritten_operand`, which brings the frame back to the
old size without changing the behaviour.

The partitioned-pair unit test asserted the removed shape. It now asks for the
remote engine state, where a cross-table pair really is one fused node over
two raw sources, and asserts that the same pair in process keeps the hybrid
route.

Signed-off-by: discord9 <discord9@163.com>

* chore(deps): pin greptime-proto to the merged raw series transport

GreptimeTeam/greptime-proto#349 has been merged into `narthex-proto` as
`f51cb5312bf7`. Move the rev from that PR's head commit to its merge commit,
in `Cargo.toml` and `Cargo.lock` together, so the pin no longer points at an
unmerged ref.

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-06 10:31:52 +08:00
Weny Xu 5bcb576b49 ci: compare GreptimeDB File and S3 in observability benchmarks (#10187)
* ci: compare GreptimeDB File and S3 in observability benchmarks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: separate File and S3 benchmark steps and verify report artifacts

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 18:19:56 +00:00
Yvan Wang ec1ad1f15b fix(datatypes): escape object keys when rendering jsonb as text (#10112)
* fix(datatypes): escape object keys when rendering jsonb as text

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* fix(datatypes): skip unicode normalization for escaped jsonb keys

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* refactor(datatypes): serialize jsonb to text directly with serde_json

Serialize the binary jsonb with serde_json::to_string(&RawJsonb) so that
both object keys and string values are escaped by serde_json. This drops
fix_unicode_point, its regex and the hand-written key escaping. Errors are
propagated instead of falling back to lossy raw bytes.

Characters such as U+2028 are now emitted as-is, which is still valid
JSON, so the affected sqlness result lines are regenerated.

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

* fix(datatypes): accept JSON text bytes in jsonb_to_string

Some write paths (COPY FROM csv/json/parquet, JSON bound as text via
MySQL) store JSON text instead of binary jsonb in JSON columns. Re-encode
such bytes as jsonb before serializing with serde_json, so they still
render (properly escaped) instead of failing with InvalidJsonb.

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>

---------

Signed-off-by: Yvan Wang <131545713+BootstrapperSBL@users.noreply.github.com>
2026-11-05 09:42:33 +00:00
discord9 a995ad0951 test(compat): exercise the bloom-token case only below v1.3.0 (#10170)
`fulltext_bloom_tokens` reads a fulltext bloom SST in the first format and
asserts that rows written by the older binary stay visible next to a current
one. Its `from_range` started at v1.0.0, but the released v1.3.0 binary
already writes `bloom_version: 2`, so the setup produces two current-format
SSTs and the expectation ("one old, one new") can never hold:

  expected: | false | 1 |   actual: | true | 2 |
  expected: | true  | 1 |

Narrow the range to the releases that still write the first format. The
constraint language has `>=`, `<=`, `==` and `=` only, and several entries
are alternatives rather than a conjunction, so `<=v1.2.99` is the bound
that excludes v1.3.0 while keeping v1.2.x and older - including the
nightly window.

Verified with the compat runner: `--from-version v1.2.1 --dry-run` selects
the case, `--from-version v1.3.0 --dry-run` skips it with
`from_range ["<=v1.2.99"] does not match version 'v1.3.0'`, and
`update-compat-versions.py --check --published-only --check-anchors`
passes.

Signed-off-by: discord9 <discord9@163.com>
2026-11-05 12:52:23 +08:00
Weny Xu c02e1f58af fix(ci): prepare Ubuntu build runners and default EC2 disks to 500 GiB (#10156)
* ci: share Linux runner dependency setup for builds and benchmarks

Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: configure EC2 build runner volumes with a 500 GiB default

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:40:14 +00:00
Weny Xu d750d25965 perf: push down COUNT and SUM for SST manifest queries (#9949)
* feat: execute logical plans through distributed inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: push down SST manifest scalar aggregates

Signed-off-by: WenyXu <wenymedia@gmail.com>

* docs: remove redundant SST manifest inspection guide

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: generalize SST manifest aggregate pushdown

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: retain frontend aggregation for unsupported inspection states

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: expose remote SST aggregate plans in EXPLAIN

Signed-off-by: WenyXu <wenymedia@gmail.com>

* test: normalize SST analyzer in TQL explain snapshots

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix: guard SST predicate types before remote inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-11-05 02:37:00 +00:00
discord9 b30557960b feat(promql): preserve scan parallelism through native frame boundaries (#10055)
* fix(promql): let a closed chain keep its charge until the packer is done

Four places released the frame reservation: the early-publish path handed it to
the packer, but the queue drain and the end-of-input path freed it in place. Those
two run while the packer still holds rows it has not turned into batches, so a
series spanning several batches went unbilled once its last piece was handed over.

Every emit path now hands the charge over, and the packed stream frees what it
holds: what is left when it is polled and pending is the one identity it is
turning into batches, and nothing survives its end or its drop.

The identity guard is also widened: within one source every series shares its
metric name, so a node that drops `__name__`/`__type__`/`__unit__` -- which is
what a range function, a math map and the timestamp shape do -- still maps
distinct series to distinct labels. Only rewriting an ordinary label can make two
equal, and that is what the guard refuses. Ordinary `rate` and arithmetic frame
output therefore publish as their chains close instead of buffering a partition.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep a cross-source tree's partitions, and gather a raw one only when it must

Three defects in how this node decided a tree's execution layout.

A match whose two operands derive from the same source still needs its match keys
co-located, but only the two-source case was recognized. `x - 2*x` with `on(host)`
therefore fell through to single-partition execution and lost the parallelism the
scan gave it.

A raw input never received a matching router: the normalization branch returned it
unchanged unless the whole tree was global, while the builder requires a direct
matching router for any multi-partition matching-local input. A two-source raw
query was rejected at planning.

Nothing reconciled the partition counts of different sources either, though the
builder requires one common positive count. A single count is now computed for the
matching-local case and every router under it uses it, gathering when that count is
one.

The window restriction also ran before normalization decided the layout, so a tree
about to be gathered could be refused for a limit that only applies to partitioned
windows.

Two tests changed with the behaviour rather than around it: hash routing does not
promise which partition holds a given series, so that test compares results
instead of placement while keeping its partition count, per-partition execution
and operand-order assertions; and a test that pinned the old refusal of a
multi-source frame now pins its acceptance.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): keep a raw source's scan partitions instead of gathering them

`RawSeriesSourceExec` declared one partition and read every child partition
inside `execute(0)`, so a cross-table node lost the scan's parallelism before
its inputs reached it. It now declares the child's own count, executes exactly
the partition it is asked for, and re-labels that partition's frames.

That leaves `SeriesGather` with no consumer: it existed to interleave several
partitions into one stream. The module is gone rather than kept as a hidden
N-to-one path a later plan could quietly fall back on.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): publish a raw source's frames under the ordering it declared

`pack_raw_series` derives the frame schema from the stream's own ordering, and
`RawSeriesSourceExec` fixes that schema at plan time from a constant. Passing
the scan's stream through unchanged broke that agreement: a columnar scan
promises key-then-time, so the region published key-then-time frames while the
frontend still expected the schema it had built, and `MergeScan` refused them.

The forwarding adapter restates the ordering this node declares. It is the
weaker promise, so a scan that keeps a finer one still satisfies it, and every
other stream property is forwarded unchanged.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): route a raw frame that names its labels by its key

A region cannot hand out complete labels: it has no metric-name binding, so
`prepare_raw_scan` refuses an explicit binding outright. What it hands out is
the stored key plus the description a decoder needs, which is the normal input
to this node rather than a case to refuse — every single-table selector is
planned that way. The node's constructor refused exactly those frames, so any
plan that inserted a router under a single-table selector failed before it ran:
the matching policies this branch adds, and the one-partition normalization of
an evaluated tree.

The node publishes labels, because that is what it groups and closes identities
by: `finish_raw_group` reads `chunk.metric` and the merge needs it too. So the
input is decoded once, at the unpack boundary this node already has, and an
already-labelled stream passes through it untouched. The frame it declares is
rebuilt from the input's own declaration instead of being compared against it,
which keeps the check that a frame this node cannot re-declare is refused while
letting input and output differ in how they name the same series. `frame_schema_with`
becomes public for that rebuild, since it is the one place that decides what a
declaration is.

`FusedSeries` now describes the router's output rather than the scan's when it
wraps a raw input, because the binding it hands `build` has to match the frame
the router actually publishes.

Tested through the node directly: a key-labelled input routes at all; matching
puts both pieces of one label set in one owner and closes it once; and
`RawPrimaryKey` still hashes the key rather than the decoded labels.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): measure a routed pair by the width it routes to

Four assertions still described the layout this branch replaced: a node with two
inputs used to read every partition of both into one and answer one partition,
and that gather is gone. The pair is now matched in the partitions it was
scanned in, so the fused node answers as many partitions as its inputs route to.

Each of these fixtures scans with four partitions, and on the base of this branch
all five tests passed, so this is a behaviour change this branch made and did not
re-verify rather than a defect: the value assertions behind these shape
assertions pass unchanged, in both the native and the legacy comparison.

The width is read from the routed children instead of being written as four, and
the scan is found by walking down through the nodes that only pass its frames
along, since a routed input is no longer a bare `SeriesScanExec`.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): normalize a multi-partition series-frame root

A query engine hands its plan to one stream, and for a plan with several
partitions that means `CoalescePartitionsExec` - ordinary batch coalescing.
A series frame cannot survive it: one stored series may be split across
partitions, and a chunk's `is_last` closes that producer rather than the
series, so interleaving two partitions' batches produces a stream whose
decoder refuses it (`raw series key changed before final chunk`). Evaluated
frames have the same shape of problem: their validator keeps one open chain,
and a batch from another partition interrupts it.

The engine's root is also a region's root, which is how a cross-table native
statement reaches this: its per-region sub-plan is rooted at a partitioned
`RawSeriesSourceExec`. So the rule belongs at the physical optimizer's end,
before the final sanity check, and it normalizes the root and nothing else -
`FusedSeries` already inserts its own owner wherever a consumer needs one.

A single-partition root is left alone: one producer is the whole answer. A
key-labelled raw root is refused rather than coalesced, because the router
checks that one stored key decodes to one label set and it needs the labels to
do that. No production plan reaches that rejection today - the only producer of
a key-labelled frame is the region's series scan, and every plan that executes
it puts `RawSeriesSourceExec` or `FusedSeries` above it, both of which publish
explicit labels - and the rejection is what keeps it that way.

`schema_check` stays enabled: for the roots this rule accepts, the router
re-declares its input's frame with the same content, label mode, decode
description and ordering, so the schema it publishes is the one it was given.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): admit an evaluated frame as a fused input

The extension planner hands a child straight to the fused executor when that
child already publishes a series frame, and until now it only recognised a raw
one. A segment's output is an evaluated frame, so a statement planned as two
segments could not be bound above the first one.

The check it replaces asked for a raw frame; the property that matters is that
the frame names its own series, which is what `labels() == Explicit` says -
evaluated frames always declare it and carry no decode description. Everything
else stays with `FusedSeriesExec::build`, which already refuses a mixed-content
node, an evaluated input that is not contiguous, and a frame whose declaration
does not match the binding it was given.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): plan a global topk under an aggregate as two segments

`sum(topk(1, m))` needs the whole vector for its ranking, which is a property of
the tree rather than of the statement, and the routes that existed could only
answer a whole-vector tree in one partition: the statement fell back to the
existing planner. Splitting it lets the per-series prefix keep the scan's
partitions while the ranking sees every candidate.

The cut goes below the `topk`, not below the `sum`. A segment holding a `topk`
reads across series, so it would gather its own input before evaluating the
prefix - correct numbers, no parallelism, which is the thing the split is for.
Below the `topk`, the prefix runs on the scan's partitions and the ranking and
the fold above it run once, on the child's evaluated frame.

Both segments are built from one compiled tree: the child is the prefix subtree,
the parent is the original tree with that subtree replaced by an input bound to
the child's frame. Nothing is re-parsed, so the grid, the matching rules and the
bound scans are the ones the compiler produced.

The parent needs its output columns without asking the existing planner to name
a computed column after its expression, so `fused_output_tags` now derives the
grouping labels of an `Aggregate` and passes a selecting `Vector`'s input labels
through. It derives output *columns*, not a complete runtime label set: the
metric name is bound separately from the tag columns, and a grouping by
`__name__` keeps that name in the labels while contributing no column.
`fused_root_schema` is that derivation, shared with the ordinary route.

The segmenter is its own method, boxed at the call site. Inlined, its future
made every frame of the planner's recursion large enough to overflow the stack
on a deeply nested statement - `test_parse_or_operator` aborted before this.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the two-segment topk plan and what it keeps

The statement's two segments do not land in one physical tree: the prefix has
one scan for a child, so the categorizer classifies it commutative and pushes it
into the region, while the aggregate above it stays at the frontend. The
observer therefore records every plan a query engine wraps, not only the first,
so the test can look at the region's own root.

What the assertions pin, and why each one is worth its brittleness:

- the answer is the global maximum, not a sum of per-partition winners;
- the frontend segment holds the complete sum and the global topk over an
  evaluated input, with no instant evaluation left in it;
- the region's segment holds the prefix and its output is an evaluated,
  explicitly labelled, contiguous frame;
- that segment keeps more than one output partition, which is the whole point of
  cutting below the topk;
- the region's root is the series-aware router, one output, and it reads that
  segment.

Column names are not compared: this route names a computed column after its
compiled root while the existing planner names it after the expression, which is
older than this split and not what this test is about.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): share the input-locality classification

A tree's distribution requirement is one property, and two layers now ask it.
The physical planner already decided with `input_locality` what a tree needs
from its inputs; the logical planner has to ask the same question when it splits
a statement, or the two can disagree about what a subtree requires.

`InputLocality` and `input_locality` are therefore public within the workspace.
The classification describes the external inputs a tree needs - not its output
distribution, and not which neighbouring operators may share a node - so a
caller that reads it as either of those will cut where it should not.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a statement where a gather would swallow its prefixes

The previous commit planned `sum(topk(k, prefix))` by matching that shape. A
planner that grows one matcher per query shape grows one per benchmark entry,
and the shape is not what decides where a statement divides: the tree already
says what each subtree needs from its inputs, and `input_locality` answers it.

So a segment is now a maximal subtree with one distribution requirement. Inside
a tree that must gather, any child that (a) can cross a boundary - it produces
evaluated samples rather than raw samples, aggregate state or a scalar - and
(b) does not itself need a gather, becomes a segment of its own and keeps the
partitions it was scanned with. Nothing else is cut, and a tree with nothing to
cut keeps the plan it had.

Every segment rebinds its own inputs: the scans it reads directly first, in the
order it first names them, then the segments cut out of it, because
`FusedSeries` requires dense source indices over its children. `fused_output_tags`
derives the parent's columns from those inputs rather than from the statement's
first selector.

Three gates keep this from answering what the routes before it would not. Each
one is a failure this change produced before it was added:

- the existing planner must still be able to plan the statement. Deriving tags
  locally had cut the route off from that refusal, so `sum(clamp(m, 2, 1))` -
  an empty range, which the existing planner rejects - was answered instead of
  failing;
- every segment's tree must construct a `NativeExecutionState`, because nothing
  below can fall back once physical planning starts, and a grouping by a value
  field is a shape the runtime refuses;
- the tree must contain a prefix that a boundary could carry, so a whole-vector
  tree with no per-series work inside it is not split for nothing.

The scope is one selector. A statement over two tables binds each scan as a
`RawSeriesSource`, and a segment whose child is that wrapper cannot be placed in
a region, so its prefix would not gain the partitions the cut is for; those
statements keep the plan they had, which the acceptance test pins.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the segment boundaries and their scope

One statement per shape the cut can meet, asserting what the rule says rather
than what one query happens to produce: a per-series tree stays one node, a
matching tree routes instead of gathering, a global fold over per-series work
cuts that work out, a constant stays embedded because a boundary cannot carry
one, and a statement over two tables keeps the plan it had because a segment
over `RawSeriesSource` cannot be placed in a region.

Answers are compared as sets of rows with the value column renamed. Three
things differ between the routes and none of them is the answer: this one names
a value column after its compiled root while the existing planner names it after
the expression, the two emit columns in a different order, and this one is
partitioned so its rows arrive by partition. Comparing them as what they are
leaves the values as the only thing an assertion can fail on.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): keep a selected segment whole

A segment is chosen because its subtree needs one distribution; cutting inside
it again contradicts the choice. The inner builder was still applying the cut
predicate, and that predicate is "exportable and not `SinglePartition`" - which
every node inside a per-series subtree satisfies, since a per-series tree has no
`SinglePartition` node in it to begin with.

`sum(topk(1, abs(m)))` showed it: the prefix was cut out of the `topk` as one
segment, then `abs` was cut out of that segment, leaving a middle segment whose
child is a node rather than a scan. That middle segment cannot be placed in a
region, so it answered one partition while the scan had several - the prefix
lost exactly the parallelism the cut is for.

The inner builder now collects the segment's `Input` leaves and nothing else.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): bind a segment to its own scan

A cut segment carries one source's identity: its scan, its metric name, its
time index, and its labels. Building it from the statement's first selector
only happens to be the same thing while a statement has one selector.

`build_inner_segment` takes the two lists the planner already has - each
occurrence's filtered scan, and each occurrence's metric, time index and tags -
and binds the segment's single source from them. The parent keeps the wrapped
children and the statement-level binding it has now, because a source that
reaches the parent directly still has to cross a region as a raw frame.

A segment that collects more than one source is declined. `MatchingLocal`
satisfies the cut predicate while reading two sources, and one segment cannot
carry both sources' identity.

No behaviour changes: the entry is still one selector, so no scan is wrapped
and the statement-level values are already that selector's.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a statement that reads more than one table

A whole-vector statement over two sources is now cut into one per-series
prefix segment per source, with the fold above them in the frontend. Each
segment binds its own source's scan, metric, time index and labels, so it can
be placed in the region that holds that source, and each prefix keeps the scan
partitions it was read with.

Three things had to change for the existing segmenter to reach that shape:

The cut sat inside `!per_partition && !multi_input`, so a statement over two
metric tables - which is `multi_input` - never reached it. The cut now sits
under `!per_partition` alone. The aggregate split keeps its own conditions,
including `!multi_input`, so a multi-input tree is still not split that way.

The single-selector gate was doing two jobs. A statement naming several
selectors declines unless those selectors read more than one table: the count
alone says nothing about whether the tree has more than one input to place, and
the shapes it would otherwise admit - one metric named twice under a fold, such
as `sum(m) / m` or `m + sum(m)` - are a plan the existing path already answers.
`cross_table` rather than `multi_input` decides this, because two Mito tables
carry no metric logical id and are still two sources.

A segment that collects more than one source is declined. A matching subtree
over two sources is a cut candidate, and one segment cannot carry both sources'
identity, so `sum(topk(1, m / n))` keeps the plan it had.

The segmented parent reports the value column the compiled tree names, not the
one the existing planner names; `sum(rate(a[5m])) / sum(b)` reports `v` where
the existing plan reports an expression-shaped name. The values are identical
and both names are pinned in the test, so a change to either rule shows up
there. Restoring the existing name needs a rename above the parent - the node's
own schema must keep the compiled names its output columns are classified by -
and is left to a follow-up.

Signed-off-by: discord9 <discord9@163.com>

* test(query): pin that two sources keep their own scan bounds

A segmented statement binds each prefix to its own source's scan, and A1 made
that binding: `bare_scans` is derived from the prepared scans after each
selector's time filter has been pushed into its own `TableScan`. Nothing tested
that the two segments actually read `their` bounds rather than one window applied
twice - the existing two-source case has no offset, so both sides read the same
range and a crossed binding would answer the same numbers.

`sum(off_a) / sum(off_b offset 60s)` differs from `sum(off_a) / sum(off_b)` only
in the right side's window, so the two queries must disagree: 20/1 and 30/2
against 20/2 and 30/3. The test asserts they do, which is what fails if both
segments were given the left scan's bounds.

The shape is read from `MergeScanExec`'s `Debug`, because its `DisplayAs` prints
only the peers: the debug form carries the table each segment reads and the
embedded `FusedSeries` segment. An in-process region hands the plan observer no
separately wrapped region tree, so the per-partition evidence for segments is
measured in `native_aggregate_test.rs::check_two_source_segregation` instead.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): decline a multi-source tree that groups by the metric name

The gate that lets a multi-source statement reach the segmenter is
`selectors.len() != 1 && !cross_table`, but the metric-name grouping guard above
it is keyed on `multi_input`, which is `selectors.len() > 1 && logical_ids_differ`.
Two ordinary Mito tables differ in table name and carry no metric logical id, so
they are `cross_table && !multi_input`: the guard did not apply and the statement
reached the segmenter.

`fused_output_tags` keeps a grouping by `__name__` out of the declared columns on
purpose - the name travels in the chunk's labels, not as a tag column - so
`sum by (__name__)` over two sources derives no tag column while the runtime
holds two groups that differ only by metric name. Both are then reported under
one label set, as two rows a reader cannot tell apart. That is the failure the
metric-logical-table test already documents; it could not catch this path because
its tables make `multi_input` true.

The check is scoped to a tree that reads more than one source. One source folds
one metric, so its groups carry the same name and naming that name or dropping it
cannot merge two of them.

Signed-off-by: discord9 <discord9@163.com>

* test(query): cut a range function on each side of a division

The two-source cases so far cut an `Instant` prefix: one sample per grid point.
A window function's prefix computes its window inside the region over that
source's own samples instead, so the segment carries a `Window` rather than an
`Instant` and nothing tested that such a prefix is still a cut candidate.

`sum(increase(a[2m])) / sum(increase(b[2m]))` over two tables: at the first step
the increases are 60 and 30, at the second 120 and 90, so the ratio is 2 and then
4/3. Two different step values are what let a shared window bound change an
answer, which is asserted before the result is compared.

The shape is read from `MergeScanExec`'s `Debug` for the same reason as the offset
case: an in-process region hands the plan observer no separately wrapped region
tree, so the per-partition evidence lives in `native_aggregate_test.rs`.

This is the shape of the throttling query in the benchmark corpus: a window under
a fold, on each side of a division.

Signed-off-by: discord9 <discord9@163.com>

* test(query): plan the benchmark's throttling and namespace-ratio shapes

Every two-source case so far divides two ungrouped scalars. The corpus queries
that motivate this work group by a label on each side, so each segment reports
several groups and the parent matches the two sides by that label. Nothing
tested that path.

`container_cpu_cfs_throttled_periods_total` over `container_cpu_cfs_periods_total`
is the corpus query verbatim: two metric logical tables on one physical table,
`sum(increase(...)) by (container)` on each side. It plans as one frontend
`FusedSeries` over two `MergeScan` children, each holding a segment bound to its
own metric, and reports the same labelled rows as the existing plan.

The namespace-CPU-usage-over-requests shape differs from the corpus text only in
the metric names, because the corpus names carry recording-rule colons that a
table name cannot: it exercises the nested `sum(max by (...) (...)) by (...)` form
and is asserted as that shape rather than as the query verbatim.

Both tests assert the native route answered, so a decline fails here instead of
passing on the legacy rows. The value column still differs from the existing
plan's - the same known gap recorded for the ungrouped case - and both names are
pinned.

The fixture helper records the three setup traps the hard way: `INSERT` needs
explicit column lists because the physical table's column order differs from the
logical table's, SQL keywords used as tags need quoting, and no sample may sit at
timestamp zero, which both the scan's lower bound and PromQL staleness drop.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record which native route every corpus query takes

The corpus is 29 queries and nothing said which of them the native route
answers, or by which of its plans. One parameterised test now runs all of them
over generated fixtures and records, per query, the route and the numbers.

Four routes, and the frontend tree alone cannot tell them apart:

- `segmented` - the frontend runs a `FusedSeries` over `MergeScan` children, one
  per source; the prefix work happens in the regions.
- `aggregate-split` - the frontend runs `AggregateFinalizeExec` over a region
  `FusedSeries` that reports `partial state:`, so what crosses the boundary is
  aggregate state.
- `region-local` - the frontend holds no native node at all: a per-series tree is
  commutative, so the whole `FusedSeries` is pushed into the region. A classifier
  that only looks at the frontend calls these legacy, which is what happened
  before this test existed.
- `legacy` - no native node on either side.

All 29 take a native route: sixteen `aggregate-split`, nine `segmented`, two
`region-local`, none legacy. The two `region-local` ones are the corpus queries
that fold nothing - a bare selector and a `rate` - which is also why pushing the
whole tree into the region costs nothing.

The test asserts only that the existing plan answers each query with rows, and
that a native route's labelled rows agree to within `1e-10` relative. It records
the route rather than asserting it, because which route a shape takes is a
property of the planner that this test exists to report, not to legislate.

Ten queries differ from the existing plan in the last bits; the largest relative
difference is 2.9e-16, which is summation order, not arithmetic. The table prints
that figure per query so a future equality failure can be told apart from
rounding without re-measuring.

Every fixture here is generated: explicit `INSERT` column lists, quoted SQL
keywords, and no sample at timestamp zero. They establish the plans and the
values; they say nothing about speed, memory, or how many partitions actually
produced rows.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record how many partitions each corpus query's nodes used

The route matrix said which plan answers each query. It did not say whether that
plan produced anything, so a node that ran and emitted nothing looked the same as
one that worked.

`output_values_per_partition` reads `native.root.output_values` off a
`FusedSeriesExec` and returns the counts per partition, and the matrix prints them
as `partitions used`. `assert_four_native_partitions` keeps its own assertions and
now builds on the same helper.

What the column does not establish, and says so where it is printed: these
fixtures hold a handful of series each, so a scan puts them all in one partition -
every vector here is `[n,0,0,0]`. That is a property of the fixture, not of the
plan. The one place a fixture is built to fill the scan's partitions is
`native_aggregate_test.rs::check_two_source_segregation`, which asserts all four
produce values for both segments of a two-source statement. This matrix records a
single assertion instead: at least one partition produced values, so a node that
ran and emitted nothing fails.

Four cases report `no metric` - L05, L06, L08 and R03, the same-table pointwise
divisions. Their region side has no `FusedSeriesExec` at all, because both scans
are matched inside one region: that is the route, not a missing measurement, and
the table says `no metric` rather than implying zero.

Signed-off-by: discord9 <discord9@163.com>

* test(query): record the new rule in the explain stages

`TQL EXPLAIN` lists the plan after every physical optimizer rule, so registering
`SeriesFrameOutput` adds one line to that listing whether or not the rule changes
the plan. The three `TQL EXPLAIN VERBOSE` statements in this case use ordinary
tables, whose schemas carry no series frame declaration, so the rule returns the
plan untouched - the new line says `SAME TEXT AS ABOVE`, which is what the
existing lines say for the same reason.

Regenerated by the sqlness runner; the diff is those three lines and nothing else.

Signed-off-by: discord9 <discord9@163.com>

* test(query): make the corpus matrix prove what it claims

Three gaps, all in the test the "every corpus query answers natively" claim rests
on.

The partition helper returned early on a plan that reported no
`native.root.output_values`, so a producer that registered nothing passed on the
declared partition count alone. It now expects the metric: the point of the
assertion is that every partition executed, and an absent counter says nothing
about that.

`segmented` was decided by the frontend tree containing a `MergeScan`, which also
matches a frontend native node consuming raw frames from its regions - a
different placement that says nothing about a prefix running in a region. The
merge scan's `Debug` carries the plan its region runs, so the check now reads
that for a segment; the raw case gets its own `frontend-raw` route. Four queries
move there: L05, L06, L08 and R03.

The matrix held 23 of the 29 queries: Q05, Q08, Q09 and Q10 had only ever been
probed in a throwaway run. They are entries now, with the join labels Q08 and Q09
match on and real `le` buckets for Q10's histogram. A fallback is no longer a
route to record: a case that reaches the existing plan fails, because that is the
regression this table exists to catch.

Twenty-seven cases here plus the separate Q04 and Q07 tests cover all 29.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): let the compiler decide which calls this route answers

A second list of function names stood beside the compiler's own capability
check, and the two disagreed. `admitted_selector` named five calls; the compiler
folds `ceil`, `floor`, `sqrt`, `irate`, `avg_over_time`, `sort`, `absent` and
more. So `ceil(m)` was refused while `sum(ceil(m))` was admitted, and the same
call meant two different things depending on what wrapped it.

The compiler already answers this question: `compile_call` refuses a name it has
no kernel for with `NotImplemented`, and the planner turns that into a decline.
With the list gone there is one place that decides, and a call the compiler folds
reaches the route whether or not it is wrapped in an aggregate.

`sort` and `absent` read across a vector rather than one series at a time, which
is why the old list named them out. They compile, so they are planned; the tests
that asserted the refusal now assert the admission. Only the shapes the compiler
genuinely cannot express keep the existing plan - a subquery evaluates its body on
a grid of its own, which this route does not offer.

Signed-off-by: discord9 <discord9@163.com>

* refactor(query): admit an operand by its shape, not by call name

An operand of a hybrid binary was admitted only if it was a selector under one of
five named calls. The compiler folds many more than those five, so `ceil(m)`,
`irate(m[5m])`, `sort(m)` and `absent(m)` were refused while the same call under
an aggregate was admitted - the name list disagreed with the compiler about what
this route can answer.

What the list was really protecting is a layout assumption, and the shape states
it directly: `try_plan_native_operand` hands back a selector's planner context,
and the hybrid caller reads field columns and tags off that context, so an
operand whose output layout is not its selector's cannot be one. `sum(A) / B` is
refused for that reason, not because `sum` is a name.

`single_selector_spine` states it on the expression: a selector, parentheses,
unary, a call carrying exactly one vector-or-matrix argument, or a binary whose
other side is a compile-time constant. Calls are followed by the type of their
arguments rather than by `constant_value`, because a call's string parameters are
constants that `constant_value` does not fold and would otherwise look like data.

Which calls this route answers is now the compiler's answer alone: a name it has
no kernel for comes back as `NotImplemented`, which the planner turns into a
decline.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): mix raw and evaluated inputs in one fused tree

`build` refused a tree whose children were not all raw or all evaluated,
but nothing below that point assumed uniformity: `execute` picks the
decoder from each source's own schema, `FusedFeed` carries the source
index, and `collect_source_schemas` already validates a repeated source
against its own grid. A frame stream has one format per stream, not one
per consumer, so one segment can decode the raw samples of a rate and
compare the result against a vector a previous stage already evaluated.

Remove the refusal and make the remaining per-input decisions per source:
`evaluated_global` now asks about the source it orders, and
`window_sources` keeps only the evaluated sources, since a raw source is
answered by the raw path it always used.

Cover it by executing `rate(x[5m]) + y` where `x` arrives raw and `y`
arrives evaluated on the statement grid. The test asserts all ten
(host, timestamp) rows and fails on the removed refusal if the refusal is
restored.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): bind a segment to every source its subtree reads

`cut_as_segment` accepts any exportable subtree that is not
`SinglePartition`, so a matching subtree over two sources - `m / n` - is
a legal cut point. `build_inner_segment` then required exactly one source
and returned `None` for it, and `segment_tree` propagates that `None` as
"the statement keeps the plan it had", so `sum(topk(1, m / n))` fell back
to the legacy path even though its cut was legal.

Build the segment from every source the subtree reads. Each child gets the
`RawSeriesSource` boundary already used for separate metric tables, which
carries that source's own metric, tags and fields across the boundary and
makes the child placeable; a segment over two tables cannot sit inside
either region, so it runs above them. The single-source case keeps its
existing direct-scan path unchanged, which is what lets a one-table
segment still be pushed into its region.

Field names come from `tree_source_fields` on the rewritten tree, whose
source indexes are already dense in the order the children are built. That
helper also validates a repeated source against its own schema and refuses
a non-dense index set, which a second traversal would have to re-derive.
It becomes `pub` and is re-exported for the query crate to call.

Take the segment's time column from its own first source rather than from
the statement's first: a cut whose sources are `[1, 2]` would otherwise
name the time column of a table it does not read.

Cover the construction with a planner test and with a real-Mito
integration test that asserts both the exact values and the route: one
segment over the two tables, each child reading its own table through its
own raw series source. Without the route assertion a silent fallback would
pass on equal values.

Signed-off-by: discord9 <discord9@163.com>

* feat(promql): run a matching tree at the configured width

A matching tree's width came from its inputs: `matching_partitions` took
the selected source's count, or the maximum of every input's. In
standalone a cross-table statement arrives as one `MergeScanExec` per
table with one partition each, so `m / n` matched on a single worker while
the scans under it ran in parallel. The engine's
`execution.target_partitions` was reachable - `plan_fused_series` already
uses it for `series_scan` - but never reached this decision.

Add `to_execution_plan_with_parallelism(inputs, target_partitions)`, which
resolves a matching tree's width as `target_partitions.max(1)` for both
the same-source and the multi-input case. `to_execution_plan` keeps its
signature and its input-derived width, so a caller that has no configured
parallelism is unchanged. `PartitionLocal` and `SinglePartition` trees do
not consult the value at all: a per-series tree runs on whatever
partitions it was scanned with, and asking for a router there would only
add a shuffle.

Forward the value through `FusedSeriesExec` because `with_new_children`
rebuilds from the node's own children, which are the ones below the
router; rebuilding without it would resolve the width again from the
pre-router count and silently fall back to one.

Verified on a real standalone: the two-table segment now reports four
partitions with a `SeriesRepartitionExec` under each one-partition region
scan, where it reported one before, and the rows are identical.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): declare a keyed test frame's ordering, and order its keys

`plan_keyed` built its schema with `datatypes::series::frame::series_frame_schema`,
which hardcodes the frame metadata `ORDERING` to `contiguous`. The stream it
returns packs what `RawScript::ordering` reports, which is now the key-then-time
order this helper models. The node reads the ordering off the *plan* schema, so the
two disagreeing meant the router scheduled these fixtures as `Contiguous` while
unpacking validated them under the stricter ordered contract.

Build the schema through `frame_schema_with` with the ordering the stream actually
produces, so the plan and the runtime frames state the same contract.

That mismatch was invisible until `RawScript` stopped hardcoding `Contiguous`: with
both sides claiming `contiguous`, the ordered validation never ran. Running it
showed the matching fixture feeding key `b` before `a` within one partition, which
its declared ordering forbids. Reorder the partition; the two pieces of `host="a"`
still arrive in different partitions, which is what the test is about.

Signed-off-by: discord9 <discord9@163.com>

* fix(mito2): declare the key order a two-phase raw scan actually keeps

`SeriesScan::raw_ordering` reported `SeriesOrdering::Contiguous` for a
`TwoPhase` scan that is not reading columnar artifacts. The data that path
produces is ordered by primary key:

- candidates are merged and sorted ascending by the encoded primary-key
  column, then decoded to ids by walking that ordered stream without
  reordering;
- partition assignment is a deterministic TSID-to-partition map that
  appends in arrival order, so one series never spans two partitions;
- each assignment is drained completely before the next is taken, rather
  than emitted in completion order;
- the raw adapter's predicate buffering preserves input order and cannot
  let a later key overtake a held one, and two-phase records always report
  `ends_package: false`, so it never engages the package-tail machinery
  that legacy relies on.

Declaring the weaker order costs two things that are available: the frame
codec's `SequenceValidator` never ran its increasing-key check, and a
repartition owner could not use a key watermark to release an identity
before another input ended. It also contradicts the scanner's own
documented contract, which states the output is always ordered by
(primary key, time index) within a partition.

Report the order the path actually keeps, and reword the two ordering
errors that said "legacy" now that both modes can reach them.

Cover it with a test that forces two-phase mode and the non-prepared
fallback, asserts which of the two it ran, and then checks per-partition
key monotonicity, per-series timestamp monotonicity, single-partition
ownership, and the ordering the scan reports. The fixture writes three
rounds, uses two output partitions and a residual predicate, and keeps
one series above a batch so the multi-batch path is exercised.

The three existing two-phase raw tests asserted the old declaration. They
check that a series stays in one run and never recurs non-adjacently,
which the stronger order still guarantees; only their expected declaration
and their names said "contiguous", so those change and the substantive
assertions stay.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): route a raw match only by a key the computation keeps

`same_source` answers "this subtree reads one source", and a `MatchingLocal`
classification was taking that answer as licence to insert the router below
the fused computations, hashing the labels as they are at the raw stage.
Those are different facts. A math map drops the metric name and the
`__type__`/`__unit__` metadata, `timestamp` does the same, and every range
function drops them except `last_over_time` and `first_over_time` — so
`on(__name__)` over `abs(...)` routes by a key the very next node deletes.

Concretely, `abs(x) + on(__name__) abs(y)` with one series on each side
routes `x` and `y` to different owners; both become the empty key after
`abs`, nothing re-partitions between the map and the match, and the pair is
silently lost.

Decide preservation separately from source identity. A node keeps the key
when it cannot remove one of the labels the key is built from; a `Window`
consults `RangeFunction::drops_metric_name` rather than assuming, so the
two functions that keep the name still route early. Anything not proven to
keep the key — a label rewrite, a histogram map, an unmodelled node —
does not establish preservation, and the tree is classified `SinglePartition`.

That is not a fallback that trades parallelism for correctness. When the
computation removes the labels the key is built from, every series on one
side reduces to the same empty key, so every left series matches every
right one: the match is whole-vector by construction and has no partitioned
form that preserves its result. One partition is the only placement that
answers it. Where the key does survive, the match stays partitioned.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): describe an admitted operand by what it produces

`try_plan_native_operand` handed the binary matcher the context saved
during scanning, which describes the operand's *input* layout. An admitted
call can change that layout: `histogram_quantile` folds a classic
histogram and drops `le`, so the scan context still names a tag column the
operand no longer produces, and default matching then builds a join key
over a column that is not there.

Concretely, `histogram_quantile(0.9, m) / histogram_quantile(0.5, m)`
declines whole-tree planning (two selectors over one table) and is then
admitted operand by operand; each operand drops `le`, and planning fails
building the join with `FieldNotFound { field: "le" }` against a schema of
`timestamp, field_0, host`.

Publish only the tag and field columns that the operand's own plan schema
actually contains, and drop `use_tsid`, which the produced output does not
carry. The check is against the schema the node returns rather than
against a list of functions that happen to shrink it, so a call added
later cannot reintroduce the gap by not being on a list.

Covered by a classic-histogram fixture with `host` and `le` and a float
bucket field, asserting each operand's context reports `["host"]` and that
the planned join key omits `le`. Reverting the change fails that test with
the `FieldNotFound` above.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): pin the partitioned duplicate-group answer, with the gap named

`native_hybrid_binary_05_06_08` asserted that the right side's two
`job="other"` series are refused as a duplicate match group. On the
partitioned route they are not: the check is gated on `lhs.presence &&
rhs.presence` computed from the partition's own slices, so the partition
holding `other` sees no left-hand data and filters every point out.

Pin what the route actually answers, and state the three things that
constrain the fix, because each has already been tried and rejected here:

- the gate cannot be dropped. `prometheus_cases.txt:1065-1075` is a
  Prometheus-generated case where the left is absent at one point and the
  right carries three samples of one group there, and the answer is a
  value with no error;
- the scan cannot run unconditionally, for the same reason;
- the match cannot be collapsed to one partition to make both operands
  whole, because that gives up the parallelism this route exists for.

What remains is to let the verdict see whole-operand presence while the
partitions keep answering — a deferred check rather than a different
placement. This test now makes any change to the current answer visible
instead of silent.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): segment a nested match instead of gathering it

A nested one-to-one match — `x + y + z`, or `(x - y) / on(host) z` —
passes the logical admission check, because `independent_scan_pair`
recurses through `scan_operand`. The physical locality classification does
not go that far: it recognizes a match only when both operands read one
source each, so `same_source` fails on an operand that is itself a match
and the whole tree is classified `SinglePartition`. The fused computation
then ran on one partition while the scans under it stayed parallel, which
the existing planner does not do: it builds ordinary equijoins that
DataFusion can execute as partitioned hash joins.

Admission and placement are different questions, so they now get different
predicates. The tree is segmented when it is cross-table, not multi-input,
classified `SinglePartition`, and its root is a vector match with a nested
vector match directly under it. The inner match becomes a segment and the
outer one repartitions the evaluated operands by its own key, which is
what the segmenter already does for any exportable evaluated prefix; a
nested match whose two levels use different keys is routed twice, once per
key, rather than pretending one routing serves both.

The predicate stays narrow on purpose. Three-selector trees over distinct
metric logical ids are `multi_input` and keep their existing construction,
including the parent binding that construction produces. A same-table
nested match does not enter this branch either, so its prior admission is
unchanged.

The plan test's collector walks each fused node's native tree in
expression-child order and resolves each input's `source_index` against
that node's inputs, so a segmented plan is checked for the same property
as before — every selector occurrence bound exactly once, in expression
order — rather than being checked more weakly. It panics on a shape it
does not model instead of returning an empty list.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): prove a raw match key survives the computation

`preserves_matching_key` decides whether a subtree keeps the labels a
vector match routes by. Two arms answered yes without looking at the
field that decides it.

A binary with a scalar operand recursed into the vector side alone, but
arithmetic and a `bool` comparison drop the metric name while a filtering
comparison keeps it — the distinction execution already makes. And an
`Instant` recursed unconditionally, while `timestamp(x)` compiles to
`InstantValue::Timestamp`, which drops the name too; the check written for
the separate `Timestamp` node never covered that shape.

Both now read the field that decides: the operator together with
`return_bool`, and `InstantValue`. The labels checked are the whole
computed set — name, type and unit — because `drop_metric_name` removes
all three and default matching does not ignore the last two by itself.

Without this, `(x * 2) + on(__name__) (y * 2)` was classified
matching-local, the router hashed the raw names before the maps dropped
them, and a valid pair could land in different partitions with nothing
re-partitioning afterwards. The regression asserts the value is 10 and
that the tree is `SinglePartition`, computing the destinations the router
would have used (`x0` to partition 0, `y0` to partition 1 of two). With
the old arm restored the value assertion fails with `left: [10.0],
right: []`.

Signed-off-by: discord9 <discord9@163.com>

* feat(query): find a nested match through the wrappers around it

Segmentation was gated on the root itself being a vector match with a
nested match beneath it, so `abs(x + y + z)` and `2 * (x + y + z)` never
entered: the wrapper or the scalar broadcast sat where the match was
expected. Meanwhile the segmenter already walks children and leaves a
wrapper in the parent while cutting what it can carry, so no per-wrapper
case is needed — the search just has to look through them.

Replacing the root check with `carries_prefix` was too wide: almost any
multi-source tree contains something a boundary could carry, so ordinary
shapes like `some_metric unless on(tag_0) other_metric` were cut for no
gain and their inputs stopped being raw sources.

So keep requiring a nested vector match, and search for it anywhere in the
tree rather than at the root. Wrappers and scalar broadcasts are
transparent because the walk visits every child; the nested match itself
is still required, which is what keeps the plain multi-source shapes on
the plan they had.

`cross_table`, `!multi_input` and the `SinglePartition` locality
requirement stay, as before: they are what keeps distinct metric logical
ids and the same-table shapes out of this branch.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): let a router publish an identity whose chain is complete

`SeriesRepartitionExec` published an identity only once every input had
contributed it or ended, because an identity may be split across inputs. A
per-series region scan assigns each identity to one partition, so on that
shape the rule holds every identity until the other partitions reach EOF: a
region's whole raw selection is retained before the first series leaves.

The node now carries an `IdentityCompletion`, defaulting to the existing
merge. `CompleteDisjoint` publishes an identity as soon as its chain closes
and keeps the published identities charged in the pool, so a second chain
for the same identity fails the execution instead of emitting a partial
series as final. The declaration is the producer's, not something a consumer
infers from a node's type: publishing early against a producer that does not
own its identities would be wrong, so the conservative mode stays the
default and every existing constructor keeps it.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): choose a series-frame root's router from declared provenance

`SeriesFrameOutput` normalized every multi-partition frame root through a
router that merges identities, which is wrong for a root whose identities are
already disjoint: it waits for every scan partition to end before the first
series leaves.

The rule now selects `CompleteDisjoint` only for a root whose producer
declared that property. A raw root qualifies when `RawSeriesSourceExec` was
built through `to_per_series_execution_plan` - which also checks the child is
a `SeriesScanExec`, so the type alone is not taken as proof - and a direct
`SeriesRepartitionExec` root qualifies because it merged the identities
itself. Everything else, including a fused frame root, keeps the merging
router: a native label transformation can map two source series onto one
output label set, so its output identities are not exclusive.

The rejection of a key-only raw frame is unchanged and its reason is
restated: coalescing cannot produce such a frame, and a router needs explicit
labels to check that one key has one binding.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): report the labels `absent` names, not the table's columns

`fused_output_tags` answered `None` for `absent`, so the caller fell back to
the existing planner's column names. Those are the table's tag columns, which
is not what `absent` reports: `absent(m{host="x"})` carries the labels its
matchers name, and the compiled tree already knows them in
`BoundVectorOp::Absent { labels }`.

The label set is now read from that field: the named labels are kept in the
input's column order, so `absent(native_shapes{host="missing"})` over a table
with `host` and `le` reports `host` rather than `ts, val, host, le`. A label
the matchers name that is not an input column still answers `None`, because a
frame has no column to carry it; `count_values` and `histogram` are unchanged.

Caught by the native-versus-legacy value comparison: the extra column made
`absent` differ from the existing plan while every value matched.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): describe a frame root's emission from what it does

`FusedSeriesExec` declared `EmissionType::Final` for a frame output, but
`poll_output` hands a frame's evaluated chunk over while its input is still
arriving, so the declaration overstated the node to the optimizer. Only a
partial aggregate's state is emitted after EOF; every other output keeps its
input's emission type.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): compare the shapes the native route now admits with the existing plan

Removing the shapes a name list used to refuse put single-selector `sort` and
`sort_desc`, `absent`, and a bare `histogram_quantile` on the segmented route.
Nothing compared their answers with the existing plan, so the widening was
unverified.

The new case runs each shape both ways over one metric table. `sort`,
`sort_desc` and both `absent` cases compare the whole printed result, which
covers column names and values together. `histogram_quantile` compares values
row by row and asserts the one known difference instead of hiding it: a
segmented statement names its value column from the compiled tree (`val`)
where the existing plan names it after the expression that produced it. That
case is given a window long enough to produce a row, so it cannot pass by
comparing two empty results.

The `absent` cases caught a real difference: the native route reported the
table's tag columns rather than the labels the matchers name, which is fixed
in the preceding commit.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): publish a fused root's identities early when the tree keeps them

The root rule publishes an identity as soon as its chain closes only for a
root whose producer declared that each identity lives in one partition, and
it knew two such producers: a per-series raw scan, and a router that had
already merged its identities. A fused root was always treated as needing a
merge, so a region whose segment is a plain instant selection over one
per-series scan still held every identity until the other scan partitions
reached EOF - `sum(topk(1, m))` is that shape, and its segment is already
multi-partition with no label rewrite.

`FusedSeriesExec` now answers whether its output identities are disjoint, and
the rule asks it. The answer is narrow on purpose: one input, and a tree of
nothing but inputs and sample instant selections, over a per-series scan or
another fused node that already answered yes. Anything that computes labels
is excluded, because a timestamp or a pointwise map can map two source series
onto one output identity, and publishing early would then hand over one
identity's pieces as if they were different ones. A vector-to-vector binary
reads across series and is excluded too.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): segment a computed matching key's operands instead of gathering them

A vector match whose operands compute away one of the matched labels was
classified `SinglePartition` and executed on one partition. That is correct
for the bare tree - the raw names cannot route a match whose operands drop
them - but it also gave up the parallelism the operands still have:
`(x * 2) + on(__name__, host) (y * 2)` loses `__name__` and keeps `host`, so
the computed sides can be routed by what they actually carry.

The planner now cuts those operands into segments of their own and lets the
parent route their evaluated outputs by the unchanged match key. The key
itself is not rewritten: `group_key` reads the labels present at that point,
so `On(__name__, host)` degrades to the `host` a computed operand still has.

Which shapes take this route is decided by one predicate shared with
`input_locality`, so the admission rule cannot drift from the locality
judgement: a matching site whose two operands are each proven single-source
computations, at least one of which does not preserve the key. A failed
preservation proof on its own is not enough - it can also mean the proof gave
up, as with a parameter that is a compound expression or an operand that is
itself a vector match - and the shared doc says so.

The same table read twice is admitted for this shape only. The rule that
declines several selectors on one table exists to keep folds such as
`sum(m) / m` off this route, not because one table cannot be read twice; the
compiler already gives each occurrence its own source index, and each cut
gets its own input. Distinct partitions are what the two operands need, and
they get them.

The regression executes both the same-table and the cross-table shape over a
four-host fixture and asserts that at least two output partitions produce
rows, so the result is not silently produced by one of them.

Signed-off-by: discord9 <discord9@163.com>

* fix(query): take the early-publication guarantee from the scan's binding

The root rule publishes an identity as soon as its chain closes only when the
producer can promise that each identity lives in one partition, and the fused
answer to that question asked whether its input was a `SeriesScanExec`. The
node's type is not that promise: a scan binds a set of logical tags and decodes
those into the series' labels, and `try_to_named_series_scan` requires the
caller to pass complete tag names but does not check it. A binding that names
only some primary-key columns maps two stored series onto one output identity,
and early publication would hand one identity's pieces over as different ones.

The answer now reads the binding: the label source has to be `PrimaryKey`, and
every primary-key column has to be among the bound tags. A source that carries
its labels explicitly is not this shape and answers no, as before.

Covered by a new `bound_tags_cover_primary_key` unit test - partial coverage is
false, full coverage is true, an explicit source is false - and by tests for the
predicate's two other parts: a tree that computes labels and a tree that is not
one instant selection over one input both answer no, and a gated execution
shows a completed identity leaving a fused node while the other input is still
held.

Signed-off-by: discord9 <discord9@163.com>

* refactor(promql): drop the checks the types and decoders already guarantee

Two re-checks ran inside the repartition node over facts its inputs had
already established, and both cost real work on every query.

A raw merge scanned each chunk's timestamps for strict increase, but
`RawTimestamps` cannot hold an unordered buffer: its field is private, its only
constructor validates, and the accessors are read-only or order-preserving, so
no safe caller can build the value this check defends against. The check is
gone; the identity, label and shape checks around it stay, because those
compare different producers' pieces rather than one chunk's own timestamps.

An owner rebuilt each chunk's identity to prove a chain does not change it, but
the decoder that hands the chain over already refuses that: `SequenceValidator`
holds one open key and rejects a key or metric change before the chain closes,
`EvaluatedSequenceValidator` does the same for labels, and the input task that
assembles chains only reads from those decoders. What stays is the identity the
owner needs anyway to pick a destination, now built by the named functions
rather than inline.

Where a schema was needed, the node built an empty stream of its own, wrapped it
in a packer and read the packer's schema back. The frame module already answers
that question from the declaration, so the fake stream is deleted along with its
half of the roundtrip.

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): recursively segment native matching plans

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): derive native schemas without legacy probes

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): recognize complete metric scan identities

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 8f752a02dbb9dc7858618c1d5cc0c41e2d4efb23)

* fix(promql): release completed disjoint identity history

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit 9975fa6c5c68786e18af230f62041d2aa5e71f6c)

* test(promql): assert native aggregate schemas independently

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): identify internal metric keys by reserved id

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep parallel sources and derive group match labels

Signed-off-by: discord9 <discord9@163.com>

* test(promql): observe the frontend fused root in scan-parallelism checks

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): release duplicate evidence immediately on abort

Signed-off-by: discord9 <discord9@163.com>

* test(promql): preserve operand identity across matching repartition

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): charge the final duplicate verdict until it is reported

Signed-off-by: discord9 <discord9@163.com>

* test(promql): observe native scans across frontend and region plans

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): drop dead test helper and clippy lints

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): select limitk by canonical label order

Signed-off-by: discord9 <discord9@163.com>

* fix(promql): keep data-free subtrees in the parent segment

Signed-off-by: discord9 <discord9@163.com>

* test(promql): prove the duplicate error crosses matching partitions

Signed-off-by: discord9 <discord9@163.com>

* test(promql): drop debug prints from the segmentation test

Signed-off-by: discord9 <discord9@163.com>

* revert(promql): keep limitk's Prometheus input-order semantics

Signed-off-by: discord9 <discord9@163.com>

* chore(promql): satisfy clippy in fused root schema

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-04 14:12:56 +08:00
shuiyisong 5fe9bdc814 feat(prometheus): store Remote Write v1 and v2 exemplars (#9922)
* feat(prometheus): persist Remote Write exemplars

Prometheus Remote Write v1 and v2 now persist valid exemplars in the source schema's metrics_exemplars table. A shared row contract keeps inherited series labels separate from exemplar labels, creates or validates an ordinary sparse append table, and preserves compatible pre-created timestamp precision.

Malformed observations are dropped and counted without discarding valid samples or sibling exemplars. Permissions and admission precede writes, and v2 written-count headers use confirmed storage progress. Includes decoder, HTTP orchestration and physical-table routing coverage. The JSON2 row-schema helper becomes fallible; its three existing trace callers propagate that error in this layer.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* feat(exemplar): persist source metric identities

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): bound exemplar write memory

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): correct batching with exemplars

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(exemplar): accept mixed-case sparse encoding

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(exemplar): reduce write preparation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(prom): reuse exemplar source targets

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: clippy

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(prometheus): defer exemplar series expansion

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-11-03 11:43:33 +00:00
discord9 01b0076f3e fix(promql): collapse duplicate samples at one timestamp before range functions (#10069)
* fix(promql): collapse duplicate samples at one timestamp before range functions

A read path that does not deduplicate, such as an append-mode region, can
hand PromQL two rows for one series at one timestamp. Every range function
reads its samples from RangeManipulate, so the duplicate entries were
treated as independent samples: count_over_time counted both, idelta and
irate returned NULL because the last two entries shared a timestamp, and
delta/increase/rate deflated because the average sampling interval divides
the observed span by sample_count - 1.

Prometheus sample identity is one sample per series per timestamp, so its
range functions can never see this input. Restore that identity once, at
the operator every range function shares, instead of special-casing each
function's arithmetic.

The collapse compares native ticks, never the millisecond payload, so
distinct sub-millisecond samples that share a millisecond stay distinct as
calculate_range documents and tests. Batches without duplicate timestamps
take a single adjacency scan and are returned untouched; only when
duplicates exist is a take built, keeping tag and value columns aligned.
Keeping the last row of a group matches MergeMode::LastRow.

The normalize test that fed three rows at one timestamp now uses distinct
timestamps: it exercises histogram start-timestamp offsetting, and its
repeated timestamp was incidental, not asserted behavior.

Signed-off-by: discord9 <discord9@163.com>

* test(promql): cover duplicate samples end to end and keep the newer value

The operator now keeps the first row of a tied timestamp group, which is the row a
deduplicating read keeps: storage orders equal primary key and timestamp by sequence
descending, and FlatDedupReader takes the first row of each group. The end-to-end case
pins that choice by writing 100 and then 101 at the same timestamp: the window reports
increase 6, while keeping the stale 100 would report 8.

The sqlness case writes duplicate rows into an append_mode region, which is the read
path that does not deduplicate, and asserts every range function plus the unaffected
ordinary table in both the standalone and distributed environments.

Signed-off-by: discord9 <discord9@163.com>

* docs(promql): qualify duplicate-timestamp conflict selection

Signed-off-by: discord9 <discord9@163.com>

* perf(promql): avoid copying tags when collapsing duplicate samples

Signed-off-by: discord9 <discord9@163.com>

---------

Signed-off-by: discord9 <discord9@163.com>
2026-11-02 04:15:19 +00:00
Weny Xu ec59131d39 feat(ci): support optional Docker targets for query regression (#10028)
* feat(ci): support optional Docker targets for query regression

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): enable Docker when provisioning image regression runners

Signed-off-by: WenyXu <wenymedia@gmail.com>

* feat(ci): publish precompiled query regression helpers with pin update PRs

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): retry helper image publication on registry timeouts

Signed-off-by: WenyXu <wenymedia@gmail.com>

* fix(ci): publish helper artifacts with hosted Skopeo job

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor(ci): update helper image through repository variable

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-30 00:23:47 +00:00
fys d33c664183 feat(json2): support skipping indexes (#9967)
* feat: support skipping indexes on JSON2 type hints

Signed-off-by: fys <fengys1996@gmail.com>

* chore: remove unused code files

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): pass build type to bloom filter indexer

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito): share bloom filter field and JSON update loop

Signed-off-by: fys <fengys1996@gmail.com>

* refactor: better name

* refactor(mito2): simplify JSON path bloom index handling

Signed-off-by: fys <fengys1996@gmail.com>

* refactor(mito2): expose index targets from creators

Signed-off-by: fys <fengys1996@gmail.com>

* docs(mito2): note JSON path index rebuild risk

Signed-off-by: fys <fengys1996@gmail.com>

* chore: update greptime-proto

* chore: update greptime-proto

---------

Signed-off-by: fys <fengys1996@gmail.com>
2026-10-29 14:29:00 +00:00
7d42feb023 chore(ci): update compatibility versions (#10051)
* chore(ci): update compatibility versions

Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>

* test(compat): bound the fulltext bloom case below v1.3.0

Signed-off-by: discord9 <discord9@163.com>
(cherry picked from commit ea570c510e)
Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
Signed-off-by: discord9 <discord9@163.com>
Signed-off-by: WenyXu <wenymedia@gmail.com>
Co-authored-by: greptimedb-ci <greptimedb-ci@users.noreply.github.com>
Co-authored-by: discord9 <discord9@163.com>
2026-10-29 01:24:50 -07:00
shuiyisong ec240ad7e6 feat(mito): preserve ordinary sparse keys across writes and schema changes (#9920)
* feat(mito): support ordinary sparse append tables

Ordinary append tables need sparse keys for optional series tags, including JSON2 values written through row and bulk paths. This change validates sparse tag schemas and defaults, keeps key identity stable when nullable tags are added, selects flat SST storage, and preserves append-mode semantics when reopening regions.

Includes storage, bulk-write, schema-evolution and repartition tests, plus SQL and mixed-version fixtures. Existing metric-engine sparse key handling remains supported. This is the storage prerequisite for exemplar ingestion; it introduces no exemplar protocol endpoint.

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): preserve reserved sparse key semantics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix(mito): clarify sparse errors and avoid panics

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito): reduce sparse key validation overhead

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: sqlness

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* perf(mito-codec): reduce sparse validation cost

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* refactor(mito): remove ordinary sparse support

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-29 05:45:07 +00:00
Weny Xu eb8e4316ec ci: backport scoped heavy CI to narthex (#10053)
* ci: backport scoped heavy CI to narthex

Backport #9927 and the CI-only changes from #10043. Adapt script references to narthex, preserve manual platform checks, and document the default-branch schedule limitation.

Upstream-Commit: ebb7e22566
Upstream-Commit: e87766d7ef
Signed-off-by: WenyXu <wenymedia@gmail.com>

* ci: refresh narthex compatibility release window

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-29 13:06:14 +08:00
shuiyisong efed847ca6 fix!: enforce sparse primary key schema (#10020)
* fix: preserve required sorting for ordinary sparse scans

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: enforce sparse primary key schema

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* test: fix

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-28 17:09:33 +00:00
Dragon Roar b21cd16f1d feat(datanode): probe object storage at startup (#9954)
* feat(datanode): self-test object storage at startup

Signed-off-by: lczllx <2181719471@qq.com>

* fix(datanode): harden the startup object storage self-test

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): cover the File store skip in the object store self-test

Signed-off-by: lczllx <2181719471@qq.com>

* refactor(datanode): rename the object storage self-test to probe

Signed-off-by: lczllx <2181719471@qq.com>

* test(datanode): disable object store self-test in sqlness and integration tests

Signed-off-by: lczllx <2181719471@qq.com>

* docs(datanode): correct what the object storage probe claims to cover

Signed-off-by: lczllx <2181719471@qq.com>

---------

Signed-off-by: lczllx <2181719471@qq.com>
2026-10-28 17:09:29 +00:00
Lei, HUANG 045cf74c3f perf(mito2): pipeline compaction and parallelize decoding and encoding (#9945)
* perf(mito2): merge compaction inputs on a separate task

Compaction read, merged and encoded each output on a single task, so
parquet encoding waited for the merge and the merge waited for
encoding. The merge now runs on its own compaction runtime task and
hands merged batches to the SST writer through a bounded channel.

The channel also closes when that task panics, so the reading side
awaits the task before reporting the end of the input. A panic fails
the compaction instead of producing a truncated SST.

One output of 4.8M rows took 1.28s instead of 1.65s.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* perf(mito2): parallelize decoding and encoding of compaction outputs

Parquet encoding took about two thirds of a compaction output's time
and ran on a single thread. When `experimental_compaction_output_parallelism`
is above 1, the SST writer encodes the columns of each row group on up
to that many blocking threads of the compaction runtime while the next
row group is buffered, and the merge decodes up to that many input
files ahead of it on compaction runtime tasks.

The parallel writer feeds every column writer the same input as
`ArrowWriter`, so the SSTs are byte-identical. The bytes of a row group
reach the object store writer one row group later, so `max_file_size`
may split a file one row group later.

The option defaults to 1, which keeps today's CPU usage. One output of
4.8M rows took 0.44s with 4 and 0.38s with 8, against 1.28s with 1.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): cancel compaction producers before consumer polling

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* feat(mito2): support per-table compaction output parallelism

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoding tasks before returning failures

Abort remaining queued encoding tasks and await running tasks before propagating the first error, keeping row-group work inside the caller's compaction memory budget lifetime.

Add deterministic error and panic tests for queued cancellation, running-task cleanup, budget retention, and original error preservation.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain encoders on SST write errors

Add asynchronous abort cleanup to both SST write entry points so source errors cannot detach in-flight encoding work after its compaction memory budget is released. Share encoder task cleanup with the encoding-failure path and preserve the original write error.

Cover source-read failures in flat and primary-key writers with deterministic running and queued tasks.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

* fix(mito2): drain compaction outputs on cancellation

Stop scheduling new output batches on cancellation instead of aborting active writers. Await started outputs and retain successful SSTs, draining sibling tasks before propagating join failures.

Add deterministic cancellation regressions covering blocking encoder lifetime, budget retention, output ownership, join failures, and cancellation before work starts.

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>

---------

Signed-off-by: Lei, HUANG <ratuthomm@gmail.com>
2026-10-28 17:09:26 +00:00
discord9 4135e65a76 fix(query): keep requested sort when filtering on a lossy cast (#10026)
fix(query): pin DataFusion cast constraint propagation fix

Update the pinned GreptimeTeam/datafusion revision from e3221b8 to
cc55344 (the 55.1 fork carrying apache/datafusion#25531 plus a stronger
regression test).

CastExpr::propagate_constraints used to cast a filter's output interval
straight back to the input type. For CAST(k AS BIGINT) = 1 the singleton
output [1, 1] was inverted to [1.0, 1.0] although k = 1.5 matches too, so
FilterExec advertised k as constant, the equivalence classes dropped the
ordering required by ORDER BY k, and rows returned in scan order.

The sqlness case keeps every #9600 assertion and the explicit ORDER BY,
and adds ascending and descending checks for the partitioned table, the
control table and a table whose matching rows are stored in neither
ascending nor descending k order.

Cargo.lock only rewrites the 35 datafusion source lines.

Signed-off-by: discord9 <discord9@163.com>
2026-10-28 10:40:43 +00:00
LFC b369d35d8b feat: enable JSON2 columns on non-append-only tables (#9985)
feat(mito2): support JSON2 in non-append-only tables

Signed-off-by: luofucong <luofc@foxmail.com>
2026-10-28 02:43:29 +00:00
discord9 984f80e277 fix(query): stop windowed sort on precision-lowering time casts (#9524)
* test: cover windowed sort cast unit mismatch

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit- or timezone-changing time casts

A cast on the time index column keeps the windowed-sort optimization
eligible only when it preserves the underlying timestamp values. Casts
that change the time unit (e.g. ms -> s) or that add a timezone to a
naive timestamp re-interpret the values, so the working ranges derived
from the time index no longer describe the sort column and the scan
fails with a working-range error or silently wrong order. Reject both
cast shapes so such queries fall back to the normal SortExec path.

The integration fixture uses two rows 1000 seconds apart so the cast
output stays distinct and the ordering assertions are not vacuous; the
descending arm gets its own expectation.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): stop windowed sort on unit-changing timestamp wrappers

to_timestamp_seconds/millis/... cast their argument to their own unit
inside the function, so the hidden cast bypassed the explicit-cast guard
and kept the windowed-sort optimization eligible on a sort column whose
values no longer match the working ranges. The wrapper branch now applies
the same unit- and value-preserving predicate as the cast branch, with
the predicate extracted into preserves_timestamp_values and shared by
both.

The integration regression gains to_timestamp_seconds ASC/DESC arms and
a to_timestamp_millis value-preserving control; unit tests cover
unit-changing, tz-adding, and non-timestamp wrapper inputs.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test: regen windowed sort sqlness after wrapper guard

The windowed_sort_nanos and windowed_sort_advance cases sort by
to_timestamp_nanos over a millisecond index and to_timestamp_millis over
a nanosecond index. Both wrappers change the time unit inside the
function, so the windowed-sort rule no longer applies and the plans
render a plain SortExec. Query results are unchanged; only the EXPLAIN
plan line differs. Regenerated by the sqlness runner.

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): simplify windowed sort cast regressions

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-27 11:51:43 +00:00
dennis zhuang 2e84833ac8 feat(partition): support iceberg_hash in partition rules (#9693)
* feat(partition): support iceberg_hash in partition rules

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): reject partition column type changes and quote rule identifiers

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(table): keep time index widening for partition columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* refactor(partition): replace PartitionFunction with IcebergHash

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* docs(partition): describe iceberg_hash in the function module

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(meta): validate iceberg_hash partition bounds on ALTER and trim comments

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test: quote partition rule identifiers in SHOW CREATE TABLE expectations

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* test(partition): check hash rules in information_schema and repartition target columns

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): coerce hash partition rules against the evaluation schema

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

* fix(partition): keep quotes in partition column names when building filters

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>

---------

Signed-off-by: Dennis Zhuang <killme2008@gmail.com>
2026-10-27 09:23:00 +00:00
Chenglong Zheng 2117f8b059 feat: Support procedure_state() using the last procedure ID (#9876)
* feat: Support procedure_state() using the last procedure ID

Signed-off-by: hahaha <a310608318@gmail.com>

* feat: Fix and add tests according to review comments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: the zero-argument signature in ADMIN swallows explicit arguments

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: fix the CI failures caused by SQL version issues

Signed-off-by: hahaha <a310608318@gmail.com>

* fix: apply the changes based on the review

Signed-off-by: hahaha <a310608318@gmail.com>

---------

Signed-off-by: hahaha <a310608318@gmail.com>
2026-10-27 07:27:55 +00:00
shuiyisong 5e9bb3c689 feat: support native histograms in sampled remote read (#9971)
* feat: support native histograms in sampled remote read

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

* fix: align native histogram ingestion with remote read validation

Signed-off-by: shuiyisong <xixing.sys@gmail.com>

---------

Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-27 03:09:13 +00:00
discord9 0dab3aef21 fix(query): preserve IGNORE NULLS semantics in distributed queries (#9601)
* fix(query): skip LastRow pushdown for IGNORE NULLS

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): keep IGNORE NULLS aggregates above remote scans

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): record IGNORE NULLS results before and after flush

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): share IGNORE NULLS coverage across deployments

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh last-value plans after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 10:25:25 +00:00
discord9 df8f6848cc fix(query): avoid lossy casts in partition pruning (#9600)
* fix(query): keep lossy casts out of partition pruning

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): align cast pruning expectations with conservative fallback

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): record lossy cast partition pruning regression

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* fix(query): retain safe pruning conjuncts beside unsupported casts

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

* test(query): refresh cast pruning plans after rebase

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>

---------

Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 09:53:32 +00:00
Weny Xu 8170d0458a fix(ci): scope Aliyun janitor sweeps to the owning repository (#9966)
Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 09:36:49 +00:00
jeremyhi 6f5c064f54 fix(meta): fence datanode processes replaced by a newer process (#9878)
* fix(meta): fence datanode processes replaced by a newer process

Two datanode processes can run with the same node id, and metasrv
grants leases and delivers instructions by node id only. Every datanode
heartbeat already carries node_epoch, the time the process started.

The metasrv leader records the highest accepted epoch per node id under
__datanode_epoch/{node_id}, a prefix of the leader cache, and raises it
with compare-and-put. The heartbeat session checks each datanode request
with a non-zero epoch before the handler chain: a lower epoch gets a
reply without running the chain, so no lease, keep-alive, stats or
mailbox reply. An epoch more than 5 minutes ahead of the leader's clock
is not recorded and is fenced too.

Datanode pushers carry the epoch of their process: a lower epoch cannot
replace a higher one, and a stream removes only its own pusher.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache load and pusher updates coherent

A write made while the leader cache is disabled, as during the load at
leader start, went to the store without the cache noticing, so the load
could install a value read before the write. Such writes now create new
cache versions around the store call, and the load drops what it loaded
for a prefix if a write happened meanwhile. This also covers node
addresses, which heartbeats write during the load.

A stream's pusher that takes its first epoch is updated in place instead
of being replaced, so messages waiting on it are not cancelled, and the
connection gauge counts only a pusher that fills an empty slot.

The epoch check moves into the heartbeat service.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): keep the leader cache coherent with writes while it is disabled

A write that started while the cache was disabled could commit after the
load had read the old value and the cache had been enabled, leaving the
old value cached. Writes now maintain the cache whether or not it is
enabled, as enabled writes already did: new versions around the store
call, then the key is updated or invalidated.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* test(meta): pin the epoch 0 exemption and pusher order, and update docs

The datanode test without an epoch now runs after another process has
recorded one, and the pusher test checks that a stream's first accepted
request receives what its handlers broadcast. The leader cache load no
longer claims that writes must not run while it loads, and the meta-srv
guide states that epoch 0 is not checked.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): register a datanode pusher only after its first request passes

A datanode stream registered an epoch-0 pusher on its handshake, so an
older process's reconnecting stream could take an empty slot, or a newer
process's pusher before its first heartbeat, for about a round trip. A
datanode stream now registers its pusher on its first request that
passes the epoch check (epoch-0 requests after the handshake included),
still before the handlers run.

Pusher registration is logged only when the insert happens. Tests cover
a fenced stream removing its own pusher, a handshake not taking an empty
slot, and a KV error failing the request; two steps that could not fail
are removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): check a datanode epoch again before replying

A request could pass the epoch check, wait in the handler chain while a
newer process of the node was recorded, and still return a region lease.
After the handlers run, the session checks the epoch again against the
cached record and, if a newer process has been recorded, replies as to a
fenced request and removes the stream's own pusher. This costs one more
read of the cached record per checked datanode request.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

* fix(meta): handle a datanode's requests one at a time

A request of an older process could pass the epoch check, wait in the
handlers while a newer process was admitted, and then keep the node
alive with its own address. Checking the epoch again after the handlers
only dropped the reply.

The heartbeat session now takes an in-memory lock for the node id
before checking a request that carries an epoch, and releases it after
the handlers, before the reply is sent. A newer process is therefore
admitted only after the handlers of an older request finish. The check
after the handlers is removed.

Signed-off-by: jeremyhi <fengjiachun@gmail.com>

---------

Signed-off-by: jeremyhi <fengjiachun@gmail.com>
2026-10-26 09:02:40 +00:00
Weny Xu c96cdd0026 perf: stream and prune SST manifest scans (#9931)
* refactor: add reusable metadata pruning and projected SST batches

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(mito2): stream and prune manifest SST metadata

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(datanode): stream inspection queries through reusable providers

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf(catalog): push manifest filters through inspection queries

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: align manifest column constants with module conventions

Signed-off-by: WenyXu <wenymedia@gmail.com>

* perf: reduce manifest inspection allocations

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: narrow manifest inspection API visibility

Signed-off-by: WenyXu <wenymedia@gmail.com>

* refactor: retain residual filters for manifest inspection

Signed-off-by: WenyXu <wenymedia@gmail.com>

---------

Signed-off-by: WenyXu <wenymedia@gmail.com>
2026-10-26 08:54:02 +00:00
shuiyisong 513b9cee58 feat(catalog): add public-global information schema scope (#9957)
Signed-off-by: shuiyisong <xixing.sys@gmail.com>
2026-10-26 07:23:30 +00:00
B. Naut e87573e286 test(sqlness): cover query options in merge-scan regressions (#9956)
* test(sqlness): cover query options on persistent SQL sessions

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>

* test(query): cover merge-scan joins with query option controls

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>

---------

Signed-off-by: Byte-Naut <Byte-Naut@proton.me>
2026-10-26 07:22:04 +00:00
discord9 2b67d0d60c test(series): pin literal ordering declarations and decoder validation
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 13:57:51 +08:00
discord9 62652ec62e fix(series): validate v2 ordering and repair scan binding fixtures
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 13:38:23 +08:00
discord9 b20e22b426 refactor(series): unify raw and evaluated SeriesFrame codec
SeriesFrame is the internal transport of the Narthex native PromQL path, and
it could only carry raw samples: an evaluated result had to fall back to a
plain row table. This makes one self-describing Arrow format carry both, so a
receiver decodes a frame from the frame alone, without the scanner, region, or
planning context its producer had.

What is unified is the physical representation and its codec, not the runtime
data structures. RawSeriesChunk, Series, and EvaluatedChunk stay as they are;
only the adapter boundaries branch on the sample shape.

Three combinations are legal and every other one is refused in both
directions:

  raw + key + timestamps          a storage raw-series scan
  raw + explicit + timestamps     a raw input whose labels are decoded
  evaluated + explicit + grid_spans   a compute node's output

An evaluated frame cannot claim a storage key, because the computation may
rewrite its labels; raw samples have no grid to be addressed on. The schema
builder and the declaration parser both enforce that, so a frame cannot be
written or read in a shape no producer could mean.

An evaluated frame declares its GridSpec once in schema metadata and carries
absolute grid positions per row as spans, so a sparse grid costs its runs and
not its holes: nothing is expanded, and no position a chunk does not hold is
filled in. A chunk with no position is legal exactly when it closes its chain,
and it is kept as one frame row, because dropping it would drop the finality
it exists to carry.

Layout is v2 and the writer and reader of the reserved greptime.series.*
metadata live in one place each, so a producer and a consumer cannot spell one
frame two ways. The unreleased v1 format is not read: the in-tree producers,
consumers, tests, and the two frozen wire fixtures all move to v2 together.

Signed-off-by: discord9 <discord9@users.noreply.github.com>
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
(cherry picked from commit 176f6f6479)
2026-10-26 12:57:11 +08:00
discord9 6eff9f5804 test(promql): verify scalar routes and four-partition broadcasts
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-26 00:13:13 +08:00
discord9 78be1b1138 test(promql): regenerate independent matcher and range results
Signed-off-by: discord9 <55937128+discord9@users.noreply.github.com>
2026-10-25 20:27:32 +08:00